After updating to 5.0.6-0 web interface is not accessible, and network shares are also not accessible

KarstenV · January 19, 2024, 2:33pm

It responded 10 as you expected.

I would off course like to not have to reinstall, as many things would then have to be setup again.

But if the install is allready broken to a point where it would be hard to repair, I guess I’ll have to do just that.

I’m just having a hard time having to reinstall a setup that boots and works, but just needs to have the Rockstor install repaired

Flox · January 19, 2024, 4:10pm

Thanks a lot for the confirmation!

@phillxnet created a dedicated issue to explore our options to address this, thanks to your willingness to test and debug above. Thank you so much, @phillxnet, for already looking into this despite everything else you already had ongoing.

github.com/rockstor/rockstor-core

Establish Postgres database format upgrade

opened 12:23PM - 19 Jan 24 UTC

phillxnet

Thanks to @FroggyFlox & KarstenV on the forum for identifying this need. From t…esting channel/branch v5.0.6.0 onwards we have fully updated our Django to 4.2 LTS, and the related python interface library of psycopg to V3: this has surfaced a Django/db-format incompatibility requiring an in-place db update capability. From @FroggyFlox's forum comment here: https://forum.rockstor.com/t/after-updating-to-5-0-6-0-web-interface-is-not-accesible-and-network-shares-are-also-not-accesible/9226/29 We see that although we have previously update our Postgres dependency to 13, and enforce that binaries use via the alternatives system when doing an rpm update, this does not in-itself adjust the underlying DB format, and the newer start up scripts, if they find an older database, will in-fact invoke the relevant older binary of Postgres, regardless of the alternatives system. New, or more recent installs are not affected assuming the use of our official installer as this always wipes the entire system disk, and so always establish a fresh and alternatives guided postgresql version. But Rockstor installs that have been updated from older Leap instances will inherit DB formats that were default at that time: to date we have reports of Postgres version 10 databases. It is proposed that for the next Stable release we develop a script/mechanism to identify and then migrate, in-place, older db versions to the newer 13 version that we currently favour. From the reports in our ongoing testing phase we have the following from KarstenV's testing efforts with 5.0.6-0 (RC1) rpm. Where we significantly updated our Django (twice), as well as adopting psycopg3 which are both very significant update. ``` NotSupportedError(', ‘django.db.utils.NotSupportedError: PostgreSQL 12 or later is required (found 10.23).’, ‘’] ``` Due to the Postgresql startup scripts failing over to the older Postgres to accomodate the older DB format found: ``` Jan 18 22:28:33 RockstorNAS postgresql-script[7819]: Your database files were created by PostgreSQL version 10. ``` The latter resulted when removing the co-resident postgres v10 as openSUSE can carry multiple postgres package installs and auto-selects the newest able to read the database in question. I.e. form @FroggyFlox's exposition of this issue on the linked form thread we have: > This happens when the postgresql.service systemd service tries to start. If we look at that service (systemctl cat postgresql.service), we can see it tries to run the following: > > ExecStart=/usr/share/postgresql/postgresql-script start > That file is provided by the system postgresql-server package and > That script is just a bash script that does some checks, such as compatibility between postgresql versions. The part that is failing on your machine is: ```bash if test -r $DATADIR/PG_VERSION ; then DATA_VERSION=$(cat $DATADIR/PG_VERSION) POSTGRES=/usr/lib/postgresql$(echo -n $DATA_VERSION | tr -d .)/bin/postgres fi if test -x /usr/bin/postgres; then ACTIVE=$(readlink -q -f /usr/bin/postgres) test -z "$POSTGRES" && POSTGRES="$ACTIVE" fi if test -n "$DATA_VERSION"; then if test -z "$ACTIVE" -o "$ACTIVE" != "$POSTGRES"; then echo " Your database files were created by PostgreSQL version $DATA_VERSION." if test -x "$POSTGRES"; then echo " Using the executables in $(dirname $POSTGRES)." else echo " Could not find executables for this version." echo " Please install the PostgreSQL server package for version $DATA_VERSION." fi fi ``` Similarly we might establish our own script to identify older 10 db instances: established by old official installers, and provide an in-place update mechanism via presumably: (pg_upgrade)[https://www.postgresql.org/docs/13/pgupgrade.html] which must be run under the newer/target Postgres version involved in the migration. Note that a prior name for pg_upgrade was pg_migrator.

It’s not really broken per se. It’s just that postgresql refuses to start a database that was created by a different version. Rockstor itself simply cannot start because of that. That means that your database is not really broken or damaged, but it does need to be upgraded to a newer version of PostgreSQL. If you do delete /var/lib/pgsql/data/, then yes, your database is gone, but that is not the case here. I’m hopeful there is a way out of this for you without having to re-install Rockstor, but of course, we cannot know for sure yet as that has not been tested yet. I understand time may be of the essence here for you so it depends on whether you can/are willing to wait for this potential database upgrade option.

Sorry I’m not able to provide you with a better answer just now.

KarstenV · January 19, 2024, 4:30pm

Thats fine.

I’ll leave the system as it is right now, and will be happy to test the fix that they develop.

It seems I have triggered a specific set of events to make this happen, but I cant be the only one that has been upgrading from old installs, trying to stay updated, so others will probably see these problems.

I’ll await what happens.

phemmy22 · January 23, 2024, 2:29pm

I had the same issue and tried the below and it works for me.

Through SSH

sudo zypper in --force rockstor

Followed by:

sudo systemctl start rockstor-bootstrap

KarstenV · January 29, 2024, 3:17pm

I saw that a new version was available so I went ahead and did:

sudo zypper in --force rockstor

Which took some time, but seemingly produced no errors.

Followed by:

sudo systemctl start rockstor-bootstrap

Which gave this output:

A dependency job for rockstor-bootstrap.service failed. See ‘journalctl -xe’ for details.

“journalctl -xe” gives this:

░░ The unit run-user-26.mount has successfully entered the ‘dead’ state.
Jan 29 16:08:53 RockstorNAS systemd[1]: user-runtime-dir@26.service: Deactivate>
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit user-runtime-dir@26.service has successfully entered the ‘dead’ sta>
Jan 29 16:08:53 RockstorNAS systemd[1]: Stopped User Runtime Directory /run/use>
░░ Subject: A stop job for unit user-runtime-dir@26.service has finished
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A stop job for unit user-runtime-dir@26.service has finished.
░░
░░ The job identifier is 71789 and the job result is done.
Jan 29 16:08:53 RockstorNAS systemd[1]: Removed slice User Slice of UID 26.
░░ Subject: A stop job for unit user-26.slice has finished
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A stop job for unit user-26.slice has finished.
░░
░░ The job identifier is 71791 and the job result is done.
lines 1507-1529/1529 (END)
░░ The unit run-user-26.mount has successfully entered the ‘dead’ state.
Jan 29 16:08:53 RockstorNAS systemd[1]: user-runtime-dir@26.service: Deactivate>
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit user-runtime-dir@26.service has successfully entered the ‘dead’ sta>
Jan 29 16:08:53 RockstorNAS systemd[1]: Stopped User Runtime Directory /run/use>
░░ Subject: A stop job for unit user-runtime-dir@26.service has finished
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A stop job for unit user-runtime-dir@26.service has finished.
░░
░░ The job identifier is 71789 and the job result is done.
Jan 29 16:08:53 RockstorNAS systemd[1]: Removed slice User Slice of UID 26.
░░ Subject: A stop job for unit user-26.slice has finished
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A stop job for unit user-26.slice has finished.
░░
░░ The job identifier is 71791 and the job result is done.

“cat /var/lib/pgsql/data/PG_VERSION” gives 13, so the update of the database was successfull.

Any ideas as how to bring it back to life?

phillxnet · January 29, 2024, 3:17pm

@KarstenV Hello again.

Thanks again for helping to tease out the multi-faceted issues around this last testing update.

Re:

Just wanted to note that we now have 5.0.7-0 out in testing. It should address both the DB format update requirement issue from systems dating back to Leap 15.3, and the failure to remove our legacy Poetry version, both uncovered predominantly in this thread.

See:

A regular command line update of the rockstor rpm should invoke the required DB update and legacy poetry removal. But you system may be in an interim state, so do keep that in mind and report as you go here, just in case.

N.B. you will need to re-install your by-hand removed postgres10 dependencies. That helped to diagnose the problem: but they are a definite required to be installed to perform the db update. The script we added to do this DB update can be run by-hand as root at the command line, and will help to inform you of these requirements. Existing systems will already have these installed as that is what the systemd required before to access the older DB formats.

Hope that helps.

KarstenV · January 29, 2024, 3:20pm

@phillxnet

Seems I answered before your post.

As you’ll see it still does not work.

Any input to what I could try is welcome

phillxnet · January 29, 2024, 3:23pm

@KarstenV Hello again.

We will need a little more log entries on this front. Note that this update has been tested to update from a 4.6.1-0 install with your reported DB, but again we need to know if you did this rpm update before re-installing he by-hand removed postgres10 dependencies of yaw. If you did then this is the problem.

I.e. what postgres packages do you have installed ?

Folks running old DB formats will still have all the old dependencies to read them. This is assumed.

KarstenV · January 29, 2024, 3:27pm

I remember reinstalling postgres 10 again after removing it to restore my system to the previous state.
Running “update-alternatives --list postgresql”, gives this:

/usr/lib/postgresql10
/usr/lib/postgresql13
/usr/lib/postgresql14

So postgres 10, 13 & 14 is installed.

KarstenV · January 29, 2024, 3:31pm

After a reboot I tried “journalctl -xe”

This time it was much more talkative:

Jan 29 16:33:06 RockstorNAS su[19427]: pam_unix(su-l:session): session opened for user postgres by (uid=0)
Jan 29 16:33:06 RockstorNAS su[19427]: pam_unix(su-l:session): session closed for user postgres
Jan 29 16:33:06 RockstorNAS systemd[1]: session-c7.scope: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit session-c7.scope has successfully entered the ‘dead’ state.
Jan 29 16:33:06 RockstorNAS su[19466]: (to postgres) root on none
Jan 29 16:33:06 RockstorNAS systemd[1]: Started Session c8 of User postgres.
░░ Subject: A start job for unit session-c8.scope has finished successfully
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A start job for unit session-c8.scope has finished successfully.
░░
░░ The job identifier is 2969.
Jan 29 16:33:06 RockstorNAS su[19466]: pam_unix(su-l:session): session opened for user postgres by (uid=0)
Jan 29 16:33:06 RockstorNAS su[19466]: pam_unix(su-l:session): session closed for user postgres
Jan 29 16:33:06 RockstorNAS systemd[1]: session-c8.scope: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit session-c8.scope has successfully entered the ‘dead’ state.
Jan 29 16:33:06 RockstorNAS su[19499]: (to postgres) root on none
Jan 29 16:33:06 RockstorNAS systemd[1]: Started Session c9 of User postgres.
░░ Subject: A start job for unit session-c9.scope has finished successfully
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A start job for unit session-c9.scope has finished successfully.
░░
░░ The job identifier is 3089.
Jan 29 16:33:06 RockstorNAS su[19499]: pam_unix(su-l:session): session opened for user postgres by (uid=0)
Jan 29 16:33:07 RockstorNAS su[19499]: pam_unix(su-l:session): session closed for user postgres
Jan 29 16:33:07 RockstorNAS systemd[1]: session-c9.scope: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit session-c9.scope has successfully entered the ‘dead’ state.
Jan 29 16:33:09 RockstorNAS systemd[1]: rockstor-pre.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ An ExecStart= process belonging to unit rockstor-pre.service has exited.
░░
░░ The process’ exit code is ‘exited’ and its exit status is 1.
Jan 29 16:33:09 RockstorNAS systemd[1]: rockstor-pre.service: Failed with result ‘exit-code’.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ The unit rockstor-pre.service has entered the ‘failed’ state with result ‘exit-code’.
Jan 29 16:33:09 RockstorNAS systemd[1]: Failed to start Tasks required prior to starting Rockstor.
░░ Subject: A start job for unit rockstor-pre.service has failed
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A start job for unit rockstor-pre.service has finished with a failure.
░░
░░ The job identifier is 2719 and the job result is failed.
Jan 29 16:33:09 RockstorNAS systemd[1]: Dependency failed for Rockstor startup script.
░░ Subject: A start job for unit rockstor.service has failed
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A start job for unit rockstor.service has finished with a failure.
░░
░░ The job identifier is 2603 and the job result is dependency.
Jan 29 16:33:09 RockstorNAS systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
░░ Subject: A start job for unit rockstor-bootstrap.service has failed
░░ Defined-By: systemd
░░ Support: systemd-devel Info Page
░░
░░ A start job for unit rockstor-bootstrap.service has finished with a failure.
░░
░░ The job identifier is 2602 and the job result is dependency.
Jan 29 16:33:09 RockstorNAS systemd[1]: rockstor-bootstrap.service: Job rockstor-bootstrap.service/start failed with result ‘dependency’.
Jan 29 16:33:09 RockstorNAS systemd[1]: rockstor.service: Job rockstor.service/start failed with result ‘dependency’.
Jan 29 16:33:09 RockstorNAS sudo[19083]: pam_unix(sudo:session): session closed for user root

phillxnet · January 29, 2024, 3:48pm

Take a look at the following PR:

github.com/rockstor/rockstor-core

Establish Postgres database format upgrade #2780

rockstor:testing ← phillxnet:2780-Establish-Postgres-database-format-upgrade

opened 07:02PM - 24 Jan 24 UTC

phillxnet

+145 -0

Adds a shell script, to be run by rpm during update, to ensure any Postgres 10 (…or later by hand) formated DBs are updated to our current Postgres target version: established by rpm dependencies and 'alternatives'. Includes: - Enforce Django required UTF8 encoding during initdb. - Establish LANG from install prior to invoking initdb. - Set initdb --locale to installs' LANG. Fixes #2780 # Testing This script was developed on, and has had initial testing with, an install originally derived from the following installer: > Rockstor-Leap15.3-generic.x86_64-4.1.0-0.install.iso Jan 14 2022 - The resulting install was then subscribed to the Stable updates channel. - An underlying OS update to a 15.4 base was then enacted via: **Distribution update from 15.3 to 15.4**: https://rockstor.com/docs/howtos/15-3_to_15-4.html Resulting also in an automatic update to our last stable release of 4.6.1-0. The above system then represents a reproducer for a pending stable update issue found in the current testing channel; where the far newer Django and psycopg3 there-in are no longer compatible with a Postgres 10 formatted DB: such as a system with the above, or older heritage, will have. --- Partnered with rockstor-rpmbuild PR: "Update DB format on install/upgrade if required ...": https://github.com/rockstor/rockstor-rpmbuild/pull/59

Your presumed shell scripting capability should enable you to see what we do, and why. We also do not delete the old DB. If you are careful you can also preserve that and re-instate it, if need be. But we need more info on what is failing on your system currently. I.e. your system is in an indeterminate state, and we need to know more about exactly what is failing.

I.e. if you re-installed all of Posgtres dependencies, it’s not just postgresql10 for example, but also the server related package that may have been removed when you uninstalled postgres10.
For example a system such as yours would have had at least the following installed by way of grand-fathering in form the Leap 15.3 base OS of the installer and original DB instantiation:

installer:~ # zypper se -s postgresql10
...
i+ | postgresql10               | package    | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
...
i+ | postgresql10-server        | package    | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
...

Also look to the zypper history, that will be far more informative (hopefully) on your install. Also I’m unsure if the --force in will have resulted in the same outcome as an update.
Also note the following from the proof of function system used here:

installer:~ # ls -la /var/lib/pgsql/
total 20
drwxr-x--- 1 postgres postgres 144 Jan 26 15:42 .
drwxr-xr-x 1 root     root     442 Jan 19 16:34 ..
-rwx------ 1 postgres postgres 769 Jan 26 15:42 analyze_new_cluster.sh
-rw-r----- 1 postgres postgres 192 Oct 20  2021 .bash_profile
lrwxrwxrwx 1 postgres postgres   6 Jan 26 15:42 data -> data13
drwx------ 1 postgres postgres 522 Jan 26 15:41 data10
drwx------ 1 postgres postgres 550 Jan 29 15:32 data13
-rwx------ 1 postgres postgres  42 Jan 26 15:42 delete_old_cluster.sh

Your’s should be very similar, but for others reading this, not necessarily so. @KarstenV system proved our need for DB update that is associated only with older installs zypper dup’ed to cutting edge testing.
Your V10 DB is saved in the data10 directory. Take great care not to loose it if you experiment. If you look to that script we create data13 from an original data dir (not link back then) moved to data10, then create the new data (link this time) to data13 after the migration used data10 & data13. The zypper history log:

less /var/log/zypp/history

should have some info as per the linked PR example. I.e. we given feedback there on the progress of our new script.

phillxnet · January 29, 2024, 3:52pm

Also we now have a much improved, debug wise, contents of:

cat /opt/rockstor/poetry-install.txt

to help pin down if the problem was the new venv creation or the DB update. The two failures from updating an older system. But again, we have successfully tested 5.0.7-0 to update from our last stable of 4.6.1-0 (when derived as per yours, from a 15.3 installer). But it would be good to diagnose you issue if it doesn’t just relate to an interim state due to active testing on this install. But more info is good and it would be good to get this old install back in order: if possible.

KarstenV · January 29, 2024, 4:19pm

OK a lot of things to digest.

I must admit my scripting capabilities may not be up to understanding all of it, I’m sorry.

I ran “zypper se -s postgresql10”

Which out put this.

Loading repository data…
Reading installed packages…

If I understand correctly, I have the required postgresql packages.

“ls -la /var/lib/pgsql/”

total 24
drwxr-x— 1 postgres postgres 204 Jan 29 16:08 .
drwxr-xr-x 1 root root 496 Jun 10 2023 …
-rw-r----- 1 postgres postgres 192 Oct 20 2021 .bash_profile
drwx------ 1 postgres postgres 550 Jan 29 16:33 data
drwx------ 1 postgres postgres 522 Jan 18 22:19 data10
drwx------ 1 postgres postgres 454 Jan 29 16:08 data13
-rw-r–r-- 1 postgres postgres 908 Jan 29 16:08 initlog
-rw------- 1 postgres postgres 260 Jan 29 16:08 pg_upgrade_internal.log
-rw------- 1 postgres postgres 179 Jan 29 16:08 pg_upgrade_server.log
-rw------- 1 postgres postgres 179 Jan 29 16:08 pg_upgrade_utility.log

Mine does not contain the data → data13 line if that is of consequence.

“cat /opt/rockstor/poetry-install.txt”
LC_ALL=C
LANG=da_DK.UTF-8
SUDO_GID=0
OLDPWD=/
COLORTERM=1
BOOST_TEST_CATCH_SYSTEM_ERRORS=no
RPM_IgnoreFailedSymlinks=1
SUDO_COMMAND=/usr/bin/zypper in --force rockstor
USER=root
PWD=/opt/rockstor
PIPX_HOME=/opt/pipx
LINES=24
HOME=/root
LC_CTYPE=da_DK.UTF-8
SUDO_USER=root
PIPX_MAN_DIR=/usr/local/share/man
ZYPP_IS_RUNNING=6271
PIPX_BIN_DIR=/usr/local/bin
SUDO_UID=0
COLUMNS=80
MAIL=/var/mail/root
TERM=xterm
SHELL=/bin/bash
SHLVL=2
LOGNAME=root
PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin
_=/usr/bin/env
Poetry (version 1.7.1)
Loading configuration file /opt/rockstor/poetry.toml
Creating virtualenv rockstor in /opt/rockstor/.venv
Using virtualenv: /opt/rockstor/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 50 installs, 0 updates, 0 removals, 1 skipped

• Installing pycparser (2.21)
• Installing cffi (1.16.0)
• Installing h11 (0.14.0)
• Installing cryptography (41.0.7)
• Installing jeepney (0.8.0)
• Installing more-itertools (10.1.0)
• Installing wsproto (1.2.0)
• Installing zipp (3.17.0)
• Installing wrapt (1.16.0)
• Installing asgiref (3.7.2)
• Installing certifi (2023.11.17)
• Installing charset-normalizer (3.3.2)
• Installing idna (3.6)
• Installing importlib-metadata (6.8.0)
• Installing jaraco-classes (3.3.0)
• Installing secretstorage (3.3.3)
• Installing simple-websocket (1.0.0)
• Installing deprecated (1.2.14)
• Installing sqlparse (0.4.4)
• Installing urllib3 (2.1.0)
• Installing bidict (0.22.1)
• Installing django (4.2.7)
• Installing greenlet (3.0.1)
• Installing jwcrypto (1.5.0)
• Installing keyring (23.13.1)
• Installing oauthlib (3.2.2)
• Installing packaging (23.2)
• Installing python-engineio (4.8.0)
• Installing pytz (2023.3.post1)
• Installing typing-extensions (4.8.0)
• Installing zope-event (5.0)
• Installing requests (2.31.0)
• Installing zope-interface (6.1)
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
• Installing dbus-python (1.3.2)
• Installing distro (1.8.0)
• Installing django-oauth-toolkit (2.3.0)
• Installing djangorestframework (3.14.0)
• Installing keyring-pass (0.8.1)
• Installing psycopg (3.1.13)
• Installing gevent (23.9.1)
• Installing psutil (5.9.4)
• Installing huey (2.5.0)
• Installing gunicorn (21.2.0)
• Installing python-socketio (5.9.0)
• Installing psycogreen (1.0)
• Installing setuptools (69.0.2): Skipped for the following reason: Already installed
• Installing six (1.16.0)
• Installing pyzmq (25.1.1)
• Installing supervisor (4.2.4)
• Installing django-pipeline (2.1.0)
• Installing urlobject (2.1.1)
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10
Connection pool is full, discarding connection: pypi.org. Connection pool size: 10

Installing the current project: rockstor (5.0.7)

Building package rockstor in editable mode
Adding rockstor.pth to /opt/rockstor/.venv/lib/python3.11/site-packages for /opt/rockstor
Adding the backup-config script to /opt/rockstor/.venv/bin
Adding the bootstrap script to /opt/rockstor/.venv/bin
Adding the data-collector script to /opt/rockstor/.venv/bin
Adding the debug-mode script to /opt/rockstor/.venv/bin
Adding the delete-api-key script to /opt/rockstor/.venv/bin
Adding the delete-rockon script to /opt/rockstor/.venv/bin
Adding the flash-optimize script to /opt/rockstor/.venv/bin
Adding the initrock script to /opt/rockstor/.venv/bin
Adding the mnt-share script to /opt/rockstor/.venv/bin
Adding the ovpn-client-gen script to /opt/rockstor/.venv/bin
Adding the ovpn-client-print script to /opt/rockstor/.venv/bin
Adding the ovpn-initpki script to /opt/rockstor/.venv/bin
Adding the prep_db script to /opt/rockstor/.venv/bin
Adding the pwreset script to /opt/rockstor/.venv/bin
Adding the qgroup-clean script to /opt/rockstor/.venv/bin
Adding the qgroup-maxout-limit script to /opt/rockstor/.venv/bin
Adding the replicad script to /opt/rockstor/.venv/bin
Adding the send-replica script to /opt/rockstor/.venv/bin
Adding the st-pool-scrub script to /opt/rockstor/.venv/bin
Adding the st-snapshot script to /opt/rockstor/.venv/bin
Adding the st-system-power script to /opt/rockstor/.venv/bin
Adding the rockstor-5.0.7.dist-info directory to /opt/rockstor/.venv/lib/python3.11/site-packages

I dont know if these things tell you anything.

I would of course still like to solve the problem, but I’m in a little deep here

phillxnet · January 29, 2024, 4:33pm

@KarstenV
Re:

but I don’t see the server package for 10 installed. I.e. it doesn’t start with i (installed) or i+ installed and updated.

I.e. they are not installed, when you uninstalled the postgres it likely too them with it. But note we still need the alternatives at 13, openSUSE systemd script (as per @Flox exposition) auto selects what is requried.

And yes: as per:

you “data” is a directory so our script likely failed and re-starting postgres service auto-created data.
Take care here, you do not want to loose data10 dir. I’ve got to step aside for a bit but ‘data’ not being a link means it is likely a default postgresql13 start enacted “I don’t see data, so I will create a blank one”.

I’ll be back in a bit for a safe re-run of this script, I did it multiple times when developing. Should have done a forum wiki on this but maybe in time I will :).

That’s the tell here. All good and recoverable, just install that missing server and unsure you still have 13 as the postgres alternatives as a re-install of rockstor will ensure also.

Looks good. Problem is down to dependencies removed re postgresql10-server is the initial though here and I’ll be back soon with the procedure to reset and not loose the old DB.

Yes, we have progress. Just give us a bit as I have to step away.

Me also and I think we have this. Hang in there, likely down to initial investigation and removal of dependency of script.

zypper history log as I’ve edited in a prior post would also be usefull. Should say where the migration script failed.

Almost there on restoring your testing instance .

KarstenV · January 29, 2024, 4:40pm

I’ll await your instructions.

I did run a zypper log, but output was enormous. I didnt want to spam the thread even more.

phillxnet · January 29, 2024, 6:17pm

@KarstenV Hello again.
Re:

N.B. these instructions are not intend for general consumption, but it’s not yet certain if @KarstenV failed DB format update is down to a short-fall in accounting for prior state (from a 15.3 DB), or just missing dependencies as a result of their testing orientated system and some package removals involved in diagnosing the original, now fixed (hopefully) issues observed in 5.0.6-0 testing rpm.

@KarstenV Assuming you have reinstalled postgresql10-server we can try running the db_update.sh script by-hand to get more feedback on the potential failure, if it wasn’t just an uninstalled package during development contributions above. Note that there is likely already some debug info in that long zypper history log and within postres users home (/var/lib/pgsql/) but give your system is now atypical as it was involved in some diagnostic alterations we are just after a re-do to get you up-and-running again.

re-do a 10 to 13 migration

Not normally required: intended for recovery for @KarstenV’s test system. But may be interesting for folks who would like to know how this script works.

Assumed start state, as per @KarstenV development instance reported state.

dir /var/lib/pgsql/data10 exists (db_upgrade.sh (5.0.7-0) created this via mv data data10)
/var/lib/pgsql/data is a directory (db_upgrade.sh creates data -> data13 symbolic link, not a directory) so this was not created by db_upgrade.sh and likely by default postgresql systemd service start after a failed db migration.

N.B. We are to loose all settings changed after initial migration: none for @KarstenV as no rockstor-service yet*

Note again we are assuming here that the data dir (not link) is created by postgresql13 upon start-up and is thus a default DB (essentially empty). If db_upgrade.sh created it, it would be a link. This is because we are now moving to versioned DB dirs: i.e. the data10 data13 ones.

# The following will also stop rockstor services.
systemctl stop postgresql.service
# 5.0.7-0's db_upgrade.sh creates data -> data13 but @KarstenV system has a dir not a link
# So we play safe (if space allows) and mv reminants out to different names.
mv /var/lib/pgsql/data /var/lib/pgsql/data-temp
mv /var/lib/pgsql/data13 /var/lib/pgsql/data13-temp
# We can now re-instate the V10 /data directory db_upgrade.sh expects:
mv /var/lib/pgsql/data10 /var/lib/pgsql/data
systemctl start postgresql.service
# confirm start-up of the old V10 format DB by what executable was chosen to read it:
ps aufx | grep '^postgres.* -D'
# likely output from above contains: /usr/lib/postgresql10/bin/postgres -D /var/lib/pgsql/data
#
# now we re-run, as root user, our 10 to 13 migration to see if we missed anything.
# `/opt/rockstor/src/rockstor/scripts/db_upgrade.sh` provides instructions.
# Expected execution time, up to a few minutes, likely only 1 or 2.
# V10 format to V13
/opt/rockstor/src/rockstor/scripts/db_upgrade.sh 10 13
#
# Note all output from above bash script.
# Confirm new Postgresql format via binary selected by it's systemd start script:
ps aufx | grep '^postgres.* -D'
# Likely output from above contains: /usr/lib/postgresql13/bin/postgres -D /var/lib/pgsql/data
# as instructed by the script output we can now start our rockstor services via their cascade:
systemctl start rockstor-bootstrap.service

@KarstenV let us know what you find out. Again db_upgrade.sh has been tested a lot of times on a generic install originating from our last Stable rpm and our earliest installer based on 15.3 (duped to 15.4), as per likely your instance. But field testing is always useful.

Plus given you helped find the cause of our prior upgrade limitations in 5.0.6-0 now fixed in 5.0.7-0, I’d rather not leave you in the lurch as it were . Plus we may learn some things here, and folks reading along will see what we do in that script, and the more folks aware, who are interested, the better. Pretty sure I’ve followed good practice in that db_upgrade.sh script by the way. Also pretty easy to read for anyone interested in improving what we have there. It was written to address all future DB format upgrades also, via differing parameters, so as always pull requests welcome.

Hope that helps.

KarstenV · January 29, 2024, 6:37pm

I installed postgreql-server.

i+ | postgresql10 | package | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
i+ | postgresql10-server | package | 10.23-150100.8.53.1 | x86_64 | Update repository with updates from SUSE Linux Enterprise 15

So I should be ready to go.

Output from first "ps aufx | grep ‘^postgres.* -D’ ":
postgres 27155 0.2 0.3 204108 25216 ? S 19:26 0:00 /usr/lib/postgresql10/bin/postgres -D /var/lib/pgsql/data

Looks fine

Output after "/opt/rockstor/src/rockstor/scripts/db_upgrade.sh 10 13 ", showed no errors.

Output from "ps aufx | grep ‘^postgres.* -D’ ":

postgres 28232 0.0 0.3 199324 25728 ? Ss 19:29 0:00 /usr/lib/postgresql13/bin/postgres -D /var/lib/pgsql/data

Seems OK

Running “systemctl start rockstor-bootstrap.service” and waiting quite some time, resultet in no errors and in this:

So that is progress.

Network drives are not accessible yet, but I’ll work on that

KarstenV · January 29, 2024, 6:43pm

Perhaps I’m seeing the same problem others are seeing in this regard?

BTW this is stellar support. Thank you so much for your efforts

I work in a support position myself and apreciate that its hard giving good and well documented support as you do.

Other than missing SMB network access my system seems to be running as before, at least from what I can see right now.

KarstenV · January 29, 2024, 7:08pm

A little more testing.

Plex and Netdata Agent are working again.

NFS access to shares is working as well.

phillxnet · January 30, 2024, 10:12am

@KarstenV Glad your almost sorted again, and thanks for letting us know.

You feedback on those 5.0.6-0 update issues was invaluable. Hopefully they are now behind us.

OK, so that’s a goodly few steps forward but a big one backwards.
Yes, and investigations / info / logs on that front would be great.
We likely have a Samba update in the interim that is now not happy with some config we use. We haven’t actually changed anything on that front ourselves however. So a bit puzzling currently.

Should be something in your logs around Samba though.
@Flox & @Hooverdan are more knowledgeable on that front.

[EDIT] I also meant to ask: how long, roughly, did our new script take to complete for your database upgrade?

less than 2 minutes? No worries if you didn’t notice however - I should have asked before hand. On a fresh (but old versioned) DB it takes around 30 seconds, so I’ve estimated less than 2 mins for DB’s with some history.

Cheers.