After updating to 5.0.6-0 web interface is not accessible, and network shares are also not accessible

phillxnet · January 29, 2024, 3:52pm

Also we now have a much improved, debug wise, contents of:

cat /opt/rockstor/poetry-install.txt

to help pin down if the problem was the new venv creation or the DB update. The two failures from updating an older system. But again, we have successfully tested 5.0.7-0 to update from our last stable of 4.6.1-0 (when derived as per yours, from a 15.3 installer). But it would be good to diagnose you issue if it doesn’t just relate to an interim state due to active testing on this install. But more info is good and it would be good to get this old install back in order: if possible.

KarstenV · January 29, 2024, 4:19pm

OK a lot of things to digest.

I must admit my scripting capabilities may not be up to understanding all of it, I’m sorry.

I ran “zypper se -s postgresql10”

Which out put this.

Loading repository data…
Reading installed packages…

If I understand correctly, I have the required postgresql packages.

“ls -la /var/lib/pgsql/”

total 24
drwxr-x— 1 postgres postgres 204 Jan 29 16:08 .
drwxr-xr-x 1 root root 496 Jun 10 2023 …
-rw-r----- 1 postgres postgres 192 Oct 20 2021 .bash_profile
drwx------ 1 postgres postgres 550 Jan 29 16:33 data
drwx------ 1 postgres postgres 522 Jan 18 22:19 data10
drwx------ 1 postgres postgres 454 Jan 29 16:08 data13
-rw-r–r-- 1 postgres postgres 908 Jan 29 16:08 initlog
-rw------- 1 postgres postgres 260 Jan 29 16:08 pg_upgrade_internal.log
-rw------- 1 postgres postgres 179 Jan 29 16:08 pg_upgrade_server.log
-rw------- 1 postgres postgres 179 Jan 29 16:08 pg_upgrade_utility.log

Mine does not contain the data → data13 line if that is of consequence.

“cat /opt/rockstor/poetry-install.txt”
LC_ALL=C
LANG=da_DK.UTF-8
SUDO_GID=0
OLDPWD=/
COLORTERM=1
BOOST_TEST_CATCH_SYSTEM_ERRORS=no
RPM_IgnoreFailedSymlinks=1
SUDO_COMMAND=/usr/bin/zypper in --force rockstor
USER=root
PWD=/opt/rockstor
PIPX_HOME=/opt/pipx
LINES=24
HOME=/root
LC_CTYPE=da_DK.UTF-8
SUDO_USER=root
PIPX_MAN_DIR=/usr/local/share/man
ZYPP_IS_RUNNING=6271
PIPX_BIN_DIR=/usr/local/bin
SUDO_UID=0
COLUMNS=80
MAIL=/var/mail/root
TERM=xterm
SHELL=/bin/bash
SHLVL=2
LOGNAME=root
PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin
_=/usr/bin/env
Poetry (version 1.7.1)
Loading configuration file /opt/rockstor/poetry.toml
Creating virtualenv rockstor in /opt/rockstor/.venv
Using virtualenv: /opt/rockstor/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 50 installs, 0 updates, 0 removals, 1 skipped

• Installing pycparser (2.21)
• Installing cffi (1.16.0)
• Installing h11 (0.14.0)
• Installing cryptography (41.0.7)
• Installing jeepney (0.8.0)
• Installing more-itertools (10.1.0)
• Installing wsproto (1.2.0)
• Installing zipp (3.17.0)
• Installing wrapt (1.16.0)
• Installing asgiref (3.7.2)
• Installing certifi (2023.11.17)
• Installing charset-normalizer (3.3.2)
• Installing idna (3.6)
• Installing importlib-metadata (6.8.0)
• Installing jaraco-classes (3.3.0)
• Installing secretstorage (3.3.3)
• Installing simple-websocket (1.0.0)
• Installing deprecated (1.2.14)
• Installing sqlparse (0.4.4)
• Installing urllib3 (2.1.0)
• Installing bidict (0.22.1)
• Installing django (4.2.7)
• Installing greenlet (3.0.1)
• Installing jwcrypto (1.5.0)
• Installing keyring (23.13.1)
• Installing oauthlib (3.2.2)
• Installing packaging (23.2)
• Installing python-engineio (4.8.0)
• Installing pytz (2023.3.post1)
• Installing typing-extensions (4.8.0)
• Installing zope-event (5.0)
• Installing requests (2.31.0)
• Installing zope-interface (6.1)
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
• Installing dbus-python (1.3.2)
• Installing distro (1.8.0)
• Installing django-oauth-toolkit (2.3.0)
• Installing djangorestframework (3.14.0)
• Installing keyring-pass (0.8.1)
• Installing psycopg (3.1.13)
• Installing gevent (23.9.1)
• Installing psutil (5.9.4)
• Installing huey (2.5.0)
• Installing gunicorn (21.2.0)
• Installing python-socketio (5.9.0)
• Installing psycogreen (1.0)
• Installing setuptools (69.0.2): Skipped for the following reason: Already installed
• Installing six (1.16.0)
• Installing pyzmq (25.1.1)
• Installing supervisor (4.2.4)
• Installing django-pipeline (2.1.0)
• Installing urlobject (2.1.1)
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10

Installing the current project: rockstor (5.0.7)

Building package rockstor in editable mode
Adding rockstor.pth to /opt/rockstor/.venv/lib/python3.11/site-packages for /opt/rockstor
Adding the backup-config script to /opt/rockstor/.venv/bin
Adding the bootstrap script to /opt/rockstor/.venv/bin
Adding the data-collector script to /opt/rockstor/.venv/bin
Adding the debug-mode script to /opt/rockstor/.venv/bin
Adding the delete-api-key script to /opt/rockstor/.venv/bin
Adding the delete-rockon script to /opt/rockstor/.venv/bin
Adding the flash-optimize script to /opt/rockstor/.venv/bin
Adding the initrock script to /opt/rockstor/.venv/bin
Adding the mnt-share script to /opt/rockstor/.venv/bin
Adding the ovpn-client-gen script to /opt/rockstor/.venv/bin
Adding the ovpn-client-print script to /opt/rockstor/.venv/bin
Adding the ovpn-initpki script to /opt/rockstor/.venv/bin
Adding the prep_db script to /opt/rockstor/.venv/bin
Adding the pwreset script to /opt/rockstor/.venv/bin
Adding the qgroup-clean script to /opt/rockstor/.venv/bin
Adding the qgroup-maxout-limit script to /opt/rockstor/.venv/bin
Adding the replicad script to /opt/rockstor/.venv/bin
Adding the send-replica script to /opt/rockstor/.venv/bin
Adding the st-pool-scrub script to /opt/rockstor/.venv/bin
Adding the st-snapshot script to /opt/rockstor/.venv/bin
Adding the st-system-power script to /opt/rockstor/.venv/bin
Adding the rockstor-5.0.7.dist-info directory to /opt/rockstor/.venv/lib/python3.11/site-packages

I dont know if these things tell you anything.

I would of course still like to solve the problem, but I’m in a little deep here

phillxnet · January 29, 2024, 4:33pm

@KarstenV
Re:

but I don’t see the server package for 10 installed. I.e. it doesn’t start with i (installed) or i+ installed and updated.

I.e. they are not installed, when you uninstalled the postgres it likely too them with it. But note we still need the alternatives at 13, openSUSE systemd script (as per @Flox exposition) auto selects what is requried.

And yes: as per:

you “data” is a directory so our script likely failed and re-starting postgres service auto-created data.
Take care here, you do not want to loose data10 dir. I’ve got to step aside for a bit but ‘data’ not being a link means it is likely a default postgresql13 start enacted “I don’t see data, so I will create a blank one”.

I’ll be back in a bit for a safe re-run of this script, I did it multiple times when developing. Should have done a forum wiki on this but maybe in time I will :).

That’s the tell here. All good and recoverable, just install that missing server and unsure you still have 13 as the postgres alternatives as a re-install of rockstor will ensure also.

Looks good. Problem is down to dependencies removed re postgresql10-server is the initial though here and I’ll be back soon with the procedure to reset and not loose the old DB.

Yes, we have progress. Just give us a bit as I have to step away.

Me also and I think we have this. Hang in there, likely down to initial investigation and removal of dependency of script.

zypper history log as I’ve edited in a prior post would also be usefull. Should say where the migration script failed.

Almost there on restoring your testing instance .

KarstenV · January 29, 2024, 4:40pm

I’ll await your instructions.

I did run a zypper log, but output was enormous. I didnt want to spam the thread even more.

phillxnet · January 29, 2024, 6:17pm

@KarstenV Hello again.
Re:

N.B. these instructions are not intend for general consumption, but it’s not yet certain if @KarstenV failed DB format update is down to a short-fall in accounting for prior state (from a 15.3 DB), or just missing dependencies as a result of their testing orientated system and some package removals involved in diagnosing the original, now fixed (hopefully) issues observed in 5.0.6-0 testing rpm.

@KarstenV Assuming you have reinstalled postgresql10-server we can try running the db_update.sh script by-hand to get more feedback on the potential failure, if it wasn’t just an uninstalled package during development contributions above. Note that there is likely already some debug info in that long zypper history log and within postres users home (/var/lib/pgsql/) but give your system is now atypical as it was involved in some diagnostic alterations we are just after a re-do to get you up-and-running again.

re-do a 10 to 13 migration

Not normally required: intended for recovery for @KarstenV’s test system. But may be interesting for folks who would like to know how this script works.

Assumed start state, as per @KarstenV development instance reported state.

dir /var/lib/pgsql/data10 exists (db_upgrade.sh (5.0.7-0) created this via mv data data10)
/var/lib/pgsql/data is a directory (db_upgrade.sh creates data -> data13 symbolic link, not a directory) so this was not created by db_upgrade.sh and likely by default postgresql systemd service start after a failed db migration.

N.B. We are to loose all settings changed after initial migration: none for @KarstenV as no rockstor-service yet*

Note again we are assuming here that the data dir (not link) is created by postgresql13 upon start-up and is thus a default DB (essentially empty). If db_upgrade.sh created it, it would be a link. This is because we are now moving to versioned DB dirs: i.e. the data10 data13 ones.

# The following will also stop rockstor services.
systemctl stop postgresql.service
# 5.0.7-0's db_upgrade.sh creates data -> data13 but @KarstenV system has a dir not a link
# So we play safe (if space allows) and mv reminants out to different names.
mv /var/lib/pgsql/data /var/lib/pgsql/data-temp
mv /var/lib/pgsql/data13 /var/lib/pgsql/data13-temp
# We can now re-instate the V10 /data directory db_upgrade.sh expects:
mv /var/lib/pgsql/data10 /var/lib/pgsql/data
systemctl start postgresql.service
# confirm start-up of the old V10 format DB by what executable was chosen to read it:
ps aufx | grep '^postgres.* -D'
# likely output from above contains: /usr/lib/postgresql10/bin/postgres -D /var/lib/pgsql/data
#
# now we re-run, as root user, our 10 to 13 migration to see if we missed anything.
# `/opt/rockstor/src/rockstor/scripts/db_upgrade.sh` provides instructions.
# Expected execution time, up to a few minutes, likely only 1 or 2.
# V10 format to V13
/opt/rockstor/src/rockstor/scripts/db_upgrade.sh 10 13
#
# Note all output from above bash script.
# Confirm new Postgresql format via binary selected by it's systemd start script:
ps aufx | grep '^postgres.* -D'
# Likely output from above contains: /usr/lib/postgresql13/bin/postgres -D /var/lib/pgsql/data
# as instructed by the script output we can now start our rockstor services via their cascade:
systemctl start rockstor-bootstrap.service

@KarstenV let us know what you find out. Again db_upgrade.sh has been tested a lot of times on a generic install originating from our last Stable rpm and our earliest installer based on 15.3 (duped to 15.4), as per likely your instance. But field testing is always useful.

Plus given you helped find the cause of our prior upgrade limitations in 5.0.6-0 now fixed in 5.0.7-0, I’d rather not leave you in the lurch as it were . Plus we may learn some things here, and folks reading along will see what we do in that script, and the more folks aware, who are interested, the better. Pretty sure I’ve followed good practice in that db_upgrade.sh script by the way. Also pretty easy to read for anyone interested in improving what we have there. It was written to address all future DB format upgrades also, via differing parameters, so as always pull requests welcome.

Hope that helps.

KarstenV · January 29, 2024, 6:37pm

I installed postgreql-server.

i+ | postgresql10 | package | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
i+ | postgresql10-server | package | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15

So I should be ready to go.

Output from first "ps aufx | grep ‘^postgres.* -D’ ":
postgres 27155 0.2 0.3 204108 25216 ? S 19:26 0:00 /usr/lib/postgresql10/bin/postgres -D /var/lib/pgsql/data

Looks fine

Output after "/opt/rockstor/src/rockstor/scripts/db_upgrade.sh 10 13 ", showed no errors.

Output from "ps aufx | grep ‘^postgres.* -D’ ":

postgres 28232 0.0 0.3 199324 25728 ? Ss 19:29 0:00 /usr/lib/postgresql13/bin/postgres -D /var/lib/pgsql/data

Seems OK

Running “systemctl start rockstor-bootstrap.service” and waiting quite some time, resultet in no errors and in this:

So that is progress.

Network drives are not accessible yet, but I’ll work on that

KarstenV · January 29, 2024, 6:43pm

Perhaps I’m seeing the same problem others are seeing in this regard?

BTW this is stellar support. Thank you so much for your efforts

I work in a support position myself and apreciate that its hard giving good and well documented support as you do.

Other than missing SMB network access my system seems to be running as before, at least from what I can see right now.

KarstenV · January 29, 2024, 7:08pm

A little more testing.

Plex and Netdata Agent are working again.

NFS access to shares is working as well.

phillxnet · January 30, 2024, 10:12am

@KarstenV Glad your almost sorted again, and thanks for letting us know.

You feedback on those 5.0.6-0 update issues was invaluable. Hopefully they are now behind us.

OK, so that’s a goodly few steps forward but a big one backwards.
Yes, and investigations / info / logs on that front would be great.
We likely have a Samba update in the interim that is now not happy with some config we use. We haven’t actually changed anything on that front ourselves however. So a bit puzzling currently.

Should be something in your logs around Samba though.
@Flox & @Hooverdan are more knowledgeable on that front.

[EDIT] I also meant to ask: how long, roughly, did our new script take to complete for your database upgrade?

less than 2 minutes? No worries if you didn’t notice however - I should have asked before hand. On a fresh (but old versioned) DB it takes around 30 seconds, so I’ve estimated less than 2 mins for DB’s with some history.

Cheers.

KarstenV · January 30, 2024, 11:34am

I will off course try to be helpfull regarding Samba.

Its not critical for me to get access right now, but eventually it would be good to have access

You asked how long it took to run the upgrade script.

Without timing it, I would say less than a minute. I didn’t wait long.

DrHolzer · February 1, 2024, 1:45am

I am experiencing the same issues regarding Samba. I am not experiencing any issues with the web interface or ssh, just Samba.

[2024/01/31 20:38:42.406002,  3] ../../lib/util/access.c:372(allow_access)
  Allowed connection from 172.16.33.13 (172.16.33.13)
[2024/01/31 20:38:42.406240,  3] ../../source3/smbd/service.c:611(make_connection_snum)
  make_connection_snum: Connect path is '/mnt2/shares' for service [shares]
[2024/01/31 20:38:42.406353,  3] ../../source3/smbd/vfs.c:115(vfs_init_default)
  Initialising default vfs hooks
[2024/01/31 20:38:42.406456,  3] ../../source3/smbd/vfs.c:141(vfs_init_custom)
  Initialising custom vfs hooks from [/[Default VFS]/]
Error: password store is empty. Try "pass init".
Traceback (most recent call last):
  File "/opt/rockstor/.venv/bin/mnt-share", line 3, in <module>
    from scripts.mount_share import mount_share
  File "/opt/rockstor/src/rockstor/scripts/__init__.py", line 6, in <module>
    django.setup()
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/settings.py", line 120, in <module>
    SECRET_KEY = keyring.get_password("rockstor", "SECRET_KEY")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/keyring/core.py", line 55, in get_password
    return get_keyring().get_password(service_name, username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/.venv/lib/python3.11/site-packages/keyring/backends/fail.py", line 25, in get_password
    raise NoKeyringError(msg)
keyring.errors.NoKeyringError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
[2024/01/31 20:38:42.769936,  1] ../../source3/smbd/service.c:721(make_connection_snum)
  root preexec gave 1 - failing connection
[2024/01/31 20:38:42.770249,  3] ../../source3/smbd/smb2_server.c:3956(smbd_smb2_request_error_ex)
  smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../../source3/smbd/smb2_tcon.c:151
[2024/01/31 20:38:42.774527,  3] ../../source3/smbd/server_exit.c:240(exit_server_common)
  Server exit (NT_STATUS_CONNECTION_RESET)

I even tried resetting the samba password because of the error indicating the password store being empty. I still received the same errors after doing so.

Flox · February 1, 2024, 2:02am

Hi @DrHolzer and welcome!

Thank you for the documented report! It looks like the issue we are currently focusing on and described/followed in the following thread:

@Hooverdan created an issue in our tracker. I think we are getting there on this one.

github.com/rockstor/rockstor-core

(t) Samba shares not accessible after testing channel upgrade (affecting upgrades to 5.0.6-0 & 5.0.7-0)

opened 11:34PM - 31 Jan 24 UTC

Hooverdan96

Thanks to forum user [Mark93](https://forum.rockstor.com/u/Mark93), it appears t…hat in the latest testing channels (5.0.6 & 5.07) samba shares are not accessible and fail in the area related to the recently revamped secrets management. For details of symptoms and troubleshooting refer to the thread: [SMB Shares inacessible after update to 5.0.6-0](https://forum.rockstor.com/t/smb-shares-inacessible-after-update-to-5-0-6-0/9229) @FroggyFlox has identified a possible fix for this situation here: [Comment 18](https://forum.rockstor.com/t/smb-shares-inacessible-after-update-to-5-0-6-0/9229/18). If that is indeed the solve, two questions (and they can be handled in separate issues): - will other services be affected in a similar manner? - since that environment variable is used in multiple places now, can the keyring directory path be parameterized, so if in the future design decisions affect its location, it does not have to be adjusted in multiple places?

Hope this helps,