'ascii' codec can't encode character u'\u017c' in position 1: ordinal not in range(128)

dskrobacki · February 24, 2022, 12:21pm

Hello all.
I have a problem with Rockstor connected to Windows AD. After configuring the domain data in the Rockstor panel, when entering user groups, it throws the error “‘ascii’ codec can’t encode character u ‘\ u017c’ in position 1: ordinal not in range (128)”
From what I found out, it concerns a coding error (Rockstor stubbornly sees ASCII, and my domain controller works on UTF-8). Is there any way to solve this problem?

Error :

        Traceback (most recent call last):

File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/group.py”, line 40, in get_queryset
return combined_groups()
File “/opt/rockstor/src/rockstor/storageadmin/views/ug_helpers.py”, line 87, in combined_groups
sys_groups = get_groups()
File “/opt/rockstor/src/rockstor/system/users.py”, line 110, in get_groups
ifp_groups = ifp_get_groups()
File “/opt/rockstor/src/rockstor/system/users.py”, line 312, in ifp_get_groups
ifp_groups[str(gpname)] = int(gpid)
UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u017c’ in position 1: ordinal not in range(128)

phillxnet · February 25, 2022, 1:34pm

@dskrobacki Welcome to the Rockstor community.

You are correct that this very much looks like character encoding issue. We some time ago force converted all commands to a set local of en_US.UTF-8 to address some international issues we were having in differing number formats for instance:

github.com

rockstor/rockstor-core/blob/master/src/rockstor/system/osi.py#L197-L214


try:
    # We force run_command to always use en_US
    # to avoid issues on date and number formats
    # on not Anglo-Saxon systems (ex. it, es, fr, de, etc)
    fake_env = dict(os.environ)
    fake_env["LANG"] = "en_US.UTF-8"
    cmd = map(str, cmd)
    if log:
        logger.debug("Running command: {}".format(" ".join(cmd)))
    p = subprocess.Popen(
        cmd, shell=shell, stdout=stdout, stderr=stderr, stdin=stdin, env=fake_env
    )
    out, err = p.communicate(input=input)
    out = out.split("\n")
    err = err.split("\n")
    rc = p.returncode
except Exception as e:
    raise Exception("Exception while running command({}): {}".format(cmd, e))

So we are using UTF-8 in places but this does look like a related issue re:

@Flox any chance you could take a peek at this bit when you get time?

@dskrobacki Thanks for the report. I does very much look like we have an issue with a group name or the like containing the u’\u017c’ lower case character of ż. (html &zdot; ż) are are failing to deal with it correctly. Could you confirm if this is the case? We can then more easily develop a reproducer for this issue and prove any fix we come up with. In the mid term we are migrating to Python 3 which uses UTF-8 by default for all strings and so will likely deal with these sorts of issues across the board as we move forward. But for now this does look like a thing we can fix once we have a reproducer.

dskrobacki · February 27, 2022, 12:03am

Yes, it is possible. There are no such characters in the groups created by me, but they appear in the default system groups - the AD server machine works in the PL language.