Web UI Fails to Load After Crash

Hi everyone,

I have a newly built Rockstor system using the latest ISO. System was up and running for about 24 hours, basic config and everything was working well. I created a Raid 1 mirror and share on two secondary drives installed in the system, confirmed file transfers were working and started migrating data to the new system. I had a few hiccups/file transfer crashes that I attributed to my inexperience with linux or just network gremlins. I was running an Rsync transfer from my old linux server when the Rsync transfer crashed after about 15 minutes of transferring, so I resumed the file transfer and let the system sit. After a few more minutes, the Web UI started becoming non responsive, not letting me see different pages and finally fully crashed. The Web interface asked me to save the logs as a tar file and I did, but was unable to open it a 7zip said the CRC didn’t match. I rebooted the server and wasn’t able to get the web UI to reload. I do have SSH access (this is how I was running the Rsync transfers) and was able to get a few logs and systemctl statuses. Any help in sorting this out would be greatly appreciated. I’ll include the outputs of the systemctl status commands from a similar post to see if this will help clarify the issue and how to get it fixed. Thanks so much!

Sincerely,

Christopher

[root@blackhole ~]# systemctl status -l rockstor-pre rockstor rockstor-bootstrap
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2016-01-30 22:09:55 EST; 16min ago
Process: 9452 ExecStart=/opt/rockstor/bin/initrock (code=exited, status=1/FAILURE)
Main PID: 9452 (code=exited, status=1/FAILURE)

Jan 30 22:09:55 blackhole initrock[9452]: delete_old_kernels(logging)
Jan 30 22:09:55 blackhole initrock[9452]: File “/opt/rockstor/src/rockstor/scripts/initrock.py”, line 49, in delete_old_kernels
Jan 30 22:09:55 blackhole initrock[9452]: o, e, rc = run_command([RPM, ‘-q’, ‘kernel-ml’])
Jan 30 22:09:55 blackhole initrock[9452]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 89, in run_command
Jan 30 22:09:55 blackhole initrock[9452]: raise CommandException(cmd, out, err, rc)
Jan 30 22:09:55 blackhole initrock[9452]: system.exceptions.CommandException: Error running a command. cmd = [’/usr/bin/rpm’, ‘-q’, ‘kernel-ml’]. rc = 1. stdout = [‘package kernel-ml is not installed’, ‘’]. stderr = [‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages index using db5 - (-30973)’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘’]
Jan 30 22:09:55 blackhole systemd[1]: rockstor-pre.service: main process exited, code=exited, status=1/FAILURE
Jan 30 22:09:55 blackhole systemd[1]: Failed to start Tasks required prior to starting Rockstor.
Jan 30 22:09:55 blackhole systemd[1]: Unit rockstor-pre.service entered failed state.
Jan 30 22:09:55 blackhole systemd[1]: rockstor-pre.service failed.

● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: inactive (dead)

Jan 30 22:02:41 blackhole systemd[1]: Dependency failed for RockStor startup script.
Jan 30 22:02:41 blackhole systemd[1]: Job rockstor.service/start failed with result ‘dependency’.
Jan 30 22:09:55 blackhole systemd[1]: Dependency failed for RockStor startup script.
Jan 30 22:09:55 blackhole systemd[1]: Job rockstor.service/start failed with result ‘dependency’.

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: inactive (dead)

Jan 30 22:02:41 blackhole systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
Jan 30 22:02:41 blackhole systemd[1]: Job rockstor-bootstrap.service/start failed with result ‘dependency’.
Jan 30 22:09:55 blackhole systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
Jan 30 22:09:55 blackhole systemd[1]: Job rockstor-bootstrap.service/start failed with result ‘dependency’.
[root@blackhole ~]#

[root@blackhole ~]# systemctl restart rockstor-pre
Job for rockstor-pre.service failed because the control process exited with error code. See “systemctl status rockstor-pre.service” and “journalctl -xe” for details.
[root@blackhole ~]#

[root@blackhole ~]# systemctl status -l rockstor-pre
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2016-01-30 22:26:40 EST; 44s ago
Process: 9516 ExecStart=/opt/rockstor/bin/initrock (code=exited, status=1/FAILURE)
Main PID: 9516 (code=exited, status=1/FAILURE)

Jan 30 22:26:40 blackhole initrock[9516]: delete_old_kernels(logging)
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/scripts/initrock.py”, line 49, in delete_old_kernels
Jan 30 22:26:40 blackhole initrock[9516]: o, e, rc = run_command([RPM, ‘-q’, ‘kernel-ml’])
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 89, in run_command
Jan 30 22:26:40 blackhole initrock[9516]: raise CommandException(cmd, out, err, rc)
Jan 30 22:26:40 blackhole initrock[9516]: system.exceptions.CommandException: Error running a command. cmd = [’/usr/bin/rpm’, ‘-q’, ‘kernel-ml’]. rc = 1. stdout = [‘package kernel-ml is not installed’, ‘’]. stderr = [‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages index using db5 - (-30973)’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘’]
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service: main process exited, code=exited, status=1/FAILURE
Jan 30 22:26:40 blackhole systemd[1]: Failed to start Tasks required prior to starting Rockstor.
Jan 30 22:26:40 blackhole systemd[1]: Unit rockstor-pre.service entered failed state.
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service failed.

Journalctl -xe

● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2016-01-30 22:26:40 EST; 44s ago
Process: 9516 ExecStart=/opt/rockstor/bin/initrock (code=exited, status=1/FAILURE)
Main PID: 9516 (code=exited, status=1/FAILURE)

Jan 30 22:26:40 blackhole initrock[9516]: delete_old_kernels(logging)
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/scripts/initrock.py”, line 49, in delete_old_kernels
Jan 30 22:26:40 blackhole initrock[9516]: o, e, rc = run_command([RPM, ‘-q’, ‘kernel-ml’])
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 89, in run_command
Jan 30 22:26:40 blackhole initrock[9516]: raise CommandException(cmd, out, err, rc)
Jan 30 22:26:40 blackhole initrock[9516]: system.exceptions.CommandException: Error running a command. cmd = [’/usr/bin/rpm’, ‘-q’, ‘kernel-ml’]. rc = 1. stdout = [‘package kernel-ml is not installed’, ‘’]. stderr = [‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages index using db5 - (-30973)’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘error: rpmdb: BDB0113 Thread/process 6351/140330671642432 failed: BDB1507 Thread died in Berkeley DB library’, ‘error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery’, ‘error: cannot open Packages database in /var/lib/rpm’, ‘’]
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service: main process exited, code=exited, status=1/FAILURE
Jan 30 22:26:40 blackhole systemd[1]: Failed to start Tasks required prior to starting Rockstor.
Jan 30 22:26:40 blackhole systemd[1]: Unit rockstor-pre.service entered failed state.
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service failed.
[root@blackhole ~]# journalctl -xe
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/scripts/initrock.py”,
Jan 30 22:26:40 blackhole initrock[9516]: delete_old_kernels(logging)
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/scripts/initrock.py”,
Jan 30 22:26:40 blackhole initrock[9516]: o, e, rc = run_command([RPM, ‘-q’, ‘kernel-ml’])
Jan 30 22:26:40 blackhole initrock[9516]: File “/opt/rockstor/src/rockstor/system/osi.py”, line 8
Jan 30 22:26:40 blackhole initrock[9516]: raise CommandException(cmd, out, err, rc)
Jan 30 22:26:40 blackhole initrock[9516]: system.exceptions.CommandException: Error running a com
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service: main process exited, code=exited, sta
Jan 30 22:26:40 blackhole systemd[1]: Failed to start Tasks required prior to starting Rockstor.
– Subject: Unit rockstor-pre.service has failed
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit rockstor-pre.service has failed.

– The result is failed.
Jan 30 22:26:40 blackhole systemd[1]: Dependency failed for RockStor startup script.
– Subject: Unit rockstor.service has failed
– Defined-By: systemd

– Unit rockstor.service has failed.

– The result is dependency.
Jan 30 22:26:40 blackhole systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
– Subject: Unit rockstor-bootstrap.service has failed
– Defined-By: systemd

– Unit rockstor-bootstrap.service has failed.

– The result is dependency.
Jan 30 22:26:40 blackhole systemd[1]: Dependency failed for Docker Application Container Engine.
– Subject: Unit docker.service has failed
– Defined-By: systemd

– Unit docker.service has failed.

– The result is dependency.
Jan 30 22:26:40 blackhole systemd[1]: Job docker.service/start failed with result ‘dependency’.
Jan 30 22:26:40 blackhole systemd[1]: Dependency failed for Samba SMB Daemon.
– Subject: Unit smb.service has failed
– Defined-By: systemd

– Unit smb.service has failed.

– The result is dependency.
Jan 30 22:26:40 blackhole systemd[1]: Job smb.service/start failed with result ‘dependency’.
Jan 30 22:26:40 blackhole systemd[1]: Job rockstor-bootstrap.service/start failed with result 'de
Jan 30 22:26:40 blackhole systemd[1]: Job rockstor.service/start failed with result ‘dependency’.
Jan 30 22:26:40 blackhole systemd[1]: Unit rockstor-pre.service entered failed state.
Jan 30 22:26:40 blackhole systemd[1]: rockstor-pre.service failed.
Jan 30 22:26:40 blackhole polkitd[2330]: Unregistered Authentication Agent for unix-process:9510:
lines 2771-2825/2825 (END)

I think I found the cause of the problem, an errant Rsync command caused the root filesystem on my boot drive to completely fill, now I’m trying to sort out what damage I caused in the process.

I think I have it fixed. I rebooted after deleting all the extra data and now the UI loaded. Thanks and have a good evening :smiley:

1 Like