rockstor-pre.service failing to start after update to 5.5

Troubleshooting
21

On two of my Leap 15.6 VMs where I performed the upgrade using zypper (from 5.1.0-0) I ran into the same python3-rpm problem as above. It announced itself during the failed rockstor-pre service via this [formatted so it’s not one single really loooong line]:

~: zypper ref && zypper up --no-recommend
[list of packages and confirmation for install]
...
[installation proceeds and rockstor is also updated successfully]
...
[reboot as recommended at the end of the zypper install run]
...
[After boot rockstor shows as dead, and further investigation shows the failure in rockstor-pre service. Checking the rockstor.log file, the message below shows up]
...
[07/May/2026 16:50:02] ERROR [system.osi:292] non-zero code(1) returned by command: ['/opt/rockstor/.venv/bin/django-admin', 'migrate', '--noinput', '--fake-initial', '--database=default', 'contenttypes']. output: [''] error: ["/opt/rockstor/src/rockstor/system/pkg_mgmt.py:134: SyntaxWarning: invalid escape sequence '\\-'", '  tags = "%{VERSION}\\-%{RELEASE}\\\\\\n%{BUILDTIME:day}"', 
'Traceback (most recent call last):', '  File "/opt/rockstor/.venv/lib/python3.13/site-packages/rpm/__init__.py", line 221, in <module>', '    _shim_module_initializing_', "NameError: name '_shim_module_initializing_' is not defined", '', 'During handling of the above exception, another exception occurred:', '', 
'Traceback (most recent call last):', '  File "/opt/rockstor/.venv/bin/django-admin", line 8, in <module>', '    sys.exit(execute_from_command_line())', '             ~~~~~~~~~~~~~~~~~~~~~~~~~^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line', '    utility.execute()', '    ~~~~~~~~~~~~~~~^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute', '    self.fetch_command(subcommand).run_from_argv(self.argv)', '    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv', '    self.execute(*args, **cmd_options)', '    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 461, in execute', '    self.check(**check_kwargs)', '    ~~~~~~~~~~^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 496, in check', '    all_issues = checks.run_checks(', '        app_configs=app_configs,', '    ...<2 lines>...', '        databases=databases,', '    )', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/checks/registry.py", line 89, in run_checks', '    new_errors = check(app_configs=app_configs, databases=databases)', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/checks/urls.py", line 16, in check_url_config', '    return check_resolver(resolver)', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/core/checks/urls.py", line 26, in check_resolver', '    return check_method()', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/urls/resolvers.py", line 531, in check', '    for pattern in self.url_patterns:', '                   ^^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/utils/functional.py", line 47, in __get__', '    res = instance.__dict__[self.name] = self.func(instance)', '                                         ~~~~~~~~~^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/urls/resolvers.py", line 718, in url_patterns', '    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)', '                       ^^^^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/utils/functional.py", line 47, in __get__', '    res = instance.__dict__[self.name] = self.func(instance)', '                                         ~~~~~~~~~^^^^^^^^^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/django/urls/resolvers.py", line 711, in urlconf_module', '    return import_module(self.urlconf_name)', '
  File "/root/.local/share/pypoetry/python/cpython@3.13.11/lib/python3.13/importlib/__init__.py", line 88, in import_module', '    return _bootstrap._gcd_import(name[level:], package, level)', '           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import', '  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load', '
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked', '
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked', '
  File "<frozen importlib._bootstrap_external>", line 1023, in exec_module', '
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed', '
  File "/opt/rockstor/src/rockstor/urls.py", line 22, in <module>', '    from smart_manager.views import (', '    ...<4 lines>...', '    )', '
  File "/opt/rockstor/src/rockstor/smart_manager/views/__init__.py", line 17, in <module>', '    from smart_manager.views.sprobes import SProbeView  # noqa E501', '    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/src/rockstor/smart_manager/views/sprobes.py", line 19, in <module>', '    import rest_framework_custom as rfc', '
  File "/opt/rockstor/src/rockstor/rest_framework_custom/__init__.py", line 18, in <module>', '    from rest_framework_custom.generic_view import GenericView  # noqa F401', '    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^', '
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 25, in <module>', '    from storageadmin.util import handle_exception', '
  File "/opt/rockstor/src/rockstor/storageadmin/util.py", line 18, in <module>', '    from system.pkg_mgmt import current_version', '
  File "/opt/rockstor/src/rockstor/system/pkg_mgmt.py", line 41, in <module>', '    from zypper_changelog_lib import get_zypper_changelog', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/zypper_changelog_lib/__init__.py", line 5, in <module>', '    from .zyppchangelog import get_zypper_changelog, get_zypper_repo_dict, zypp_info_codes, \\', '        zypp_err_codes', '  File "/opt/rockstor/.venv/lib/python3.13/site-packages/zypper_changelog_lib/zyppchangelog.py", line 44, in <module>', '    import rpm', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/rpm/__init__.py", line 224, in <module>', '    initialize()', '    ~~~~~~~~~~^^', '
  File "/opt/rockstor/.venv/lib/python3.13/site-packages/rpm/__init__.py", line 213, in initialize', '    raise ImportError(', '    ...<2 lines>...', '    )',
 'ImportError: Failed to import system RPM module. Make sure RPM Python bindings are installed on your system.', '']

Installing zypper in python3-rpm and a subsequent reboot seemingly fixed the failure and rockstor UI, etc. worked once again.

2 Likes
22

Thank you so much for that, @Hooverdan !
Now we know it’s related to our zypper-changelog-lib. We do specify Py3.11 there so I wonder if that’s causing issues with a Py3.13 venv for Rockstor. That, or it’s the standalone build (I hope not).

Could you check what you have installed in the venv, please?

cd /opt/rockstor
poetry show --tree

The PyPI package rpm v0.4.0 should be there, I think.

1 Like
23

Also, do you see anything out of the ordinary in your poetry-install.txt file?

I’m looking for any information on the success or problems during the installation of all of Rockstor’s dependencies there.

24

Here it is. It shows only rpm* at the very end if I didn’t miss it anywhere else. But that makes sense based on @phillxnet’s comment above on the zypper-changelog-lib dependency:

dbus-python 1.4.0 Python bindings for libdbus
distro 1.9.0 Distro - an OS platform information API
django 5.2.13 A high-level Python web framework that encourages rapid development and clean, pragmatic design.
├── asgiref >=3.8.1
├── sqlparse >=0.3.1
└── tzdata *
django-debug-toolbar 6.0.0 A configurable set of panels that display various debug information about the current request/response.
├── django >=4.2.9
│   ├── asgiref >=3.8.1
│   ├── sqlparse >=0.3.1
│   └── tzdata *
└── sqlparse >=0.2
django-oauth-toolkit 2.4.0 OAuth2 Provider for Django
├── django >=3.2,<4.0.0 || >4.0.0
│   ├── asgiref >=3.8.1
│   ├── sqlparse >=0.3.1
│   └── tzdata *
├── jwcrypto >=0.8.0
│   ├── cryptography >=3.4
│   │   └── cffi >=2.0.0
│   │       └── pycparser *
│   └── typing-extensions >=4.5.0
├── oauthlib >=3.1.0
├── pytz >=2024.1
└── requests >=2.13.0
    ├── certifi >=2023.5.7
    ├── charset-normalizer >=2,<4
    ├── idna >=2.5,<4
    └── urllib3 >=1.26,<3
django-pipeline 4.1.0 Pipeline is an asset packaging library for Django.
├── setuptools *
└── wheel *
    └── packaging >=24.0
djangorestframework 3.17.1 Web APIs for Django, made easy.
└── django >=4.2
    ├── asgiref >=3.8.1
    ├── sqlparse >=0.3.1
    └── tzdata *
docutils 0.21.2 Docutils -- Python Documentation Utilities
gevent 26.4.0 Coroutine-based network library
├── cffi >=1.17.1
│   └── pycparser *
├── greenlet >=3.2.2
├── zope-event *
└── zope-interface *
gunicorn 23.0.0 WSGI HTTP Server for UNIX
└── packaging *
huey 2.6.0 a little task queue
keyring-pass 0.9.3 https://www.passwordstore.org/ backend for https://pypi.org/project/keyring/
├── jaraco-classes >=3.2.3
│   └── more-itertools *
└── keyring >=23.9.3
    ├── jaraco-classes *
    │   └── more-itertools *
    ├── jaraco-context *
    ├── jaraco-functools *
    │   └── more-itertools * (circular dependency aborted here)
    ├── jeepney >=0.4.2
    ├── pywin32-ctypes >=0.2.0
    └── secretstorage >=3.2
        ├── cryptography >=2.0
        │   └── cffi >=2.0.0
        │       └── pycparser *
        └── jeepney >=0.6 (circular dependency aborted here)
packaging 25.0 Core utilities for Python packages
psutil 7.2.2 Cross-platform lib for process and system monitoring.
psycogreen 1.0 psycopg2 integration with coroutine libraries
psycopg 3.3.3 PostgreSQL database adapter for Python
└── tzdata *
python-engineio 4.8.0 Engine.IO server and client for Python
└── simple-websocket >=0.10.0
    └── wsproto *
        └── h11 >=0.16.0,<1
python-socketio 5.9.0 Socket.IO server and client for Python
├── bidict >=0.21.0
└── python-engineio >=4.7.0
    └── simple-websocket >=0.10.0
        └── wsproto *
            └── h11 >=0.16.0,<1
pyzmq 27.1.0 Python bindings for 0MQ
└── cffi *
    └── pycparser *
six 1.16.0 Python 2 and 3 compatibility utilities
zypper-changelog-lib 0.7.9 Changelogs for installable pending updates, or available/uninstalled packages
├── keyring-pass *
│   ├── jaraco-classes >=3.2.3
│   │   └── more-itertools *
│   └── keyring >=23.9.3
│       ├── jaraco-classes * (circular dependency aborted here)
│       ├── jaraco-context *
│       ├── jaraco-functools *
│       │   └── more-itertools * (circular dependency aborted here)
│       ├── jeepney >=0.4.2
│       ├── pywin32-ctypes >=0.2.0
│       └── secretstorage >=3.2
│           ├── cryptography >=2.0
│           │   └── cffi >=2.0.0
│           │       └── pycparser *
│           └── jeepney >=0.6 (circular dependency aborted here)
├── requests *
│   ├── certifi >=2023.5.7
│   ├── charset-normalizer >=2,<4
│   ├── idna >=2.5,<4
│   └── urllib3 >=1.26,<3
└── rpm *

I can post the entire poetry-install.txt file, but that’s a lot of lines :smiley:

I do find the entry for the rpm 0.4.0 install:

  - Installing rpm (0.4.0)
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /pypi/rpm/0.4.0/json HTTP/1.1" 304 0

Saw something else, but not sure whether this presents an issue or not. I picked that because there seemed to have been challenges around the gpg/keyring stuff. The last message block before installing dependencies from lock file message shows this:

...
Using virtualenv: /opt/rockstor/.venv
Loading environment variables: /opt/rockstor/.env
Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Backend 'fail Keyring' is not suitable
No valid keyring backend was found
Unavailable
Removing the project's plugin cache because it is outdated
Ensuring that the Poetry plugins required by the project are available...
All required plugins have already been installed in Poetry's environment.

Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 52 installs, 0 updates, 0 removals

...

Let me know if you want me to post the entire file, I can easily do that. or DM it instead.

2 Likes
25

:thinking: the gpg thing at the end is intriguing, I can’t remember whether it’s one of these expected outputs… that will require more investigation.

With regards to the rpm issue, I think I see what is happening. The rpm python module is actually just a shim to allow one to import the system rpm package in a virtual environment. It is thus looking for a system python interpreter and then it looks for the rpm package in that interpreter’s site-packages.
With our standalone on Leap15.6, we run on a Py3.13 interpreter in the venv so it is looking for a system Py3.13 rpm package. With Rockstor’s current dependencies, it is thus failing as it cannot find a system Py3.13 and thus no corresponding rpm package.
The reason it works after installing python3-rpm is that the rpm-shim module falls back to the major version-only interpreter, which is the python3 interpreter, which is itself just a symlink to 3.6 (if I remember correctly) on Leap 15.6 as explained by @phillxnet. Fortunately, the rpm-shim module is compatible with Py3.6, so this is why things work then.

@phillxnet, @Hooverdan, if my understanding above is correct, that has some repercussion on our ability to run isolated from the system packages but that’s a topic for another day, I’m afraid.

1 Like
26

Not sure whether that gives you more insight, but I just upgraded a SlowRoll VM from 5.5.1-0 to 5.5.2-0 (so some of the “bigger transitions” have already occurred, hence might not be of as much value).

Even though it’s only a minor version change, shows the same messages about the 'keyringin thepoetry-install.txt` file.

The previous file (I checked briefly before upgrading) that installed the 5.5.1-0 version, did not have any of those messages but went directly from some venv and other installs (I assume, belonging to the poetry install itself) into writing lock file and then installing the dependencies from the lock file. The writing lock file message I couldn’t find in any of the 5.5.2-0 updates.

2 Likes
27

OK, so my reading thus far, informed by @Flox & @Hooverdan analysis since my last comment here, is that we definitely require a work-around of having a fall-back install for our Leap 15.6 rpm only (Py3.13 via Poetry but no matching OS version) of python-rpm. Which although of a different Python version looks to be acceptable to the PyPi shim of rpm.

That’s quite the discovery and a most welcome find. I propose that we simply add this as an rpm dependency ready for our next Leap 15.6 rpm. I was reluctant to leave Leap 15.6 behind just yet, even given it is now EOL, as we still have a lot of users on it. And it would be good to give folks more time and continue our development towards the next stable with that target still in play. But unfortunately this has lead to an overreach of sorts and an oversight on my part. Apologies folks.

So if we add this older python-rpm package as a dependency to our 15.6 target only we should be able to contiue serving that install base, at least until the end of the current testing phase was my hope here. We will not, as is normal, publish another stable for this target - but the last testing published for 15.6 will be a final stable candidate anyway.

I’ve opened the following issue to add this workaround requirement now we know why:

But we still have more than one cause of failure here. However I’m chuffed we now have the answer to at least one failure mode, but there remains a likely gpg DB lock issue afoot.

Thanks folks.

2 Likes
28

Just a note that we have now, in the edge repo only currently, and only for the affected target OS of Leap 15.6, an x86_64 rpm with the discovered work-around from this thread: as 5.5.2-3124.x86_64.rpm as the rpm changes were tested along with our also pre-merged changes in rockstor-core to remove the now dated NIS service.

The plan is for our next testing repo rpm to contain this additional dependency when deployed to a Leap 15.6 OS target.

Hope that helps for further ‘edge’ experimentation before we get our next testing release out, which will be announced separately and will contain the NIS changelog item within the rpm itself.

3 Likes
29

To honour the reporting efforts of @meresda here, we have:

with more relevant context:

which ultimately results in a command line call (from within our venv) failure:

Ultimately failing the rockstor-build.servcie. So in @meresda case this failure looks to be related to one of our know but oustanding, or as yet elusive gpg lock issue cause.

2 Likes