Rockstor not starting after upgrade to 4.0.4 to 4.0.6

I have found after this upgrade, the first reboot failed to start rockstor properly.

Investigations showed that the PostgreSQL service was not starting due to permissions…

$ systemctl status postgresql

FATAL: could not create lock file “/var/run/postgresql/.s.PGSQL.5432.lock”: Permission denied

On my installation, this directory didn’t exist. Therefore I ran the following:

$ mkdir -p /var/run/postgresql
$ chown -R postgres:postgres /var/run/postgresql
$ systemctl start postgresql
$ systemctl status postgresql

● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2021-04-19 13:07:23 BST; 3s ago
Process: 2475 ExecStart=/usr/share/postgresql/postgresql-script start (code=exited, status=0/SUCCESS)
Main PID: 2493 (postgres)

$ systemctl start rockstor
$ systemctl status rockstor

● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-04-19 13:08:28 BST; 2min 28s ago
Main PID: 2643 (supervisord)

This still left my shares unmounted so I rebooted. Now I am back to square 1…

$ systemctl status postgresql

● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-04-19 13:15:06 BST; 1min 51s ago

FATAL: could not create lock file “/var/run/postgresql/.s.PGSQL.5432.lock”: No such file or directory

Help anyone?

@nigel Thanks for the report and initial investigation. That’s a funny one.

Take a look at the following forum thread where @Tony_Cristiano also had what may have been a similar or related issue with postgresql not being available:

There @Flox went for the force re-install approach and after @Tony_Cristiano then re-established the enable default and started the service it seems to have sorted itself.

Your issue may not be related but take a look to see as we have yet to get to the bottom on what happened in @Tony_Cristiano reported incident.

Thanks.

1 Like

Yeah I tried all of that except the re-install @phillxnet, but it didn’t work. I managed to get PG working by changing it to use /tmp instead of /var/run/postgres, but then rockstor doesn’t work, as it’s coded to expect /var/run, not tmp.

I will attempt the re-install and report back.

@phillxnet Unfortunately this didn’t work. Looking further, it seems that the creation of the directory /run/postgresql from /usr/lib/tmpfiles.d/postgresql.conf is not being triggered for some reason…

$ cat /usr/lib/tmpfiles.d/postgresql.conf
# For the PostgreSQL server’s unix domain socket
d /run/postgresql 1775 postgres postgres - -

@phillxnet I think the problem is the tmpfiles service is failing, which is a pre-condition for PG…

$ systemctl list-units --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● dmraid-activation.service loaded failed failed Activation of DM RAID sets
● docker.service loaded failed failed Docker Application Container Engine
● postgresql.service loaded failed failed PostgreSQL database server
● rockstor-pre.service loaded failed failed Tasks required prior to starting Rockstor
● snapper-cleanup.service loaded failed failed Daily Cleanup of Snapper Snapshots
● snapper-timeline.service loaded failed failed Timeline of Snapper Snapshots
systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories
● twonky.service loaded failed failed Twonky Server Service

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

9 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use ‘systemctl list-unit-files’.
:/opt/rockstor/var/log # systemctl status systemd-tmpfiles-setup
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-04-19 16:27:35 BST; 1h 2min ago
Docs: man:tmpfiles.d(5)
man:systemd-tmpfiles(8)
Main PID: 568 (code=exited, status=1/FAILURE)

Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /var/log/wtmp, refusing.
Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /var/log/btmp, refusing.
Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /var/log/lastlog, refusing.
Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /run/nologin, refusing.
Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /run/user, refusing.
Apr 19 16:27:35 systemd-tmpfiles[568]: Unsafe symlinks encountered in /run/utmp, refusing.
Apr 19 16:27:35 systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Apr 19 16:27:35 systemd[1]: Failed to start Create Volatile Files and Directories.
Apr 19 16:27:35 systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Apr 19 16:27:35 systemd[1]: systemd-tmpfiles-setup.service: Failed with result ‘exit-code’.
:/opt/rockstor/var/log # systemctl status systemd-tmpfiles-setup-dev
● systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev
Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup-dev.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-04-19 16:27:34 BST; 1h 2min ago
Docs: man:tmpfiles.d(5)
man:systemd-tmpfiles(8)
Main PID: 464 (code=exited, status=1/FAILURE)

Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/btrfs-control, refusing.
Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/loop-control, refusing.
Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/net, refusing.
Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/net/tun, refusing.
Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/ppp, refusing.
Apr 19 16:27:34 systemd-tmpfiles[464]: Unsafe symlinks encountered in /dev/userio, refusing.
Apr 19 16:27:34 systemd[1]: systemd-tmpfiles-setup-dev.service: Main process exited, code=exited, status=1/FAILURE
Apr 19 16:27:34 systemd[1]: Failed to start Create Static Device Nodes in /dev.
Apr 19 16:27:34 systemd[1]: systemd-tmpfiles-setup-dev.service: Unit entered failed state.
Apr 19 16:27:34 systemd[1]: systemd-tmpfiles-setup-dev.service: Failed with result ‘exit-code’.

1 Like

Wow. Looks like / ownership got changed. Saw this…

chowned / to root, and rebooted. PG and Rockstor back up.

Now I have to figure out what did this, because it sure can’t have been Rockstor. It was owned by a share user, so must have been something happened internally.

2 Likes