[Solved] Rockstor WebUI no longer starts (3.8.13)

grizzly · May 3, 2016, 8:15pm

Hi,
I’ve upgraded to 3.8.13 and rebooted. Now Rockstor WebUI won’t load (404 not found). NFS, SMB & SSH all work fine. /opt/rockstor/var/log/rockstor.log just says:

[03/May/2016 20:56:48] DEBUG [smart_manager.data_collector:403] Listening on port http://127.0.0.1:8080 and on port 10843 (flash policy server)
[03/May/2016 20:57:28] DEBUG [storageadmin.views.command:152] Bootstrap operations completed

“systemctl status -l rockstor-pre rockstor rockstor-bootstrap” says:

● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: active (exited) since Tue 2016-05-03 21:25:25 BST; 15min ago
Main PID: 3594 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-pre.service

May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,177: Updating the timezone from the system
May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,178: system timezone = Europe/London
May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,179: Updating sshd_config
May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,182: sshd_config already has the updates. Leaving it unchanged.
May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,183: Running prepdb…
May 03 21:25:23 nas03 initrock[3594]: 2016-05-03 21:25:23,967: stopping firewalld…
May 03 21:25:24 nas03 initrock[3594]: 2016-05-03 21:25:24,073: firewalld stopped and disabled
May 03 21:25:24 nas03 initrock[3594]: 2016-05-03 21:25:24,166: Registered task: fs.btrfs.start_balance
May 03 21:25:25 nas03 initrock[3594]: 2016-05-03 21:25:25,625: rockstor service looks correct. Not updating.
May 03 21:25:25 nas03 initrock[3594]: 2016-05-03 21:25:25,626: rockstor-bootstrap.service looks correct. Not updating.

● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2016-05-03 21:25:25 BST; 15min ago
Main PID: 5944 (supervisord)
CGroup: /system.slice/rockstor.service
├─5944 /usr/bin/python /opt/rockstor/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
├─6443 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
├─6444 /usr/bin/python /opt/rockstor/bin/data-collector
├─6445 /usr/bin/python2.7 /opt/rockstor/bin/django ztaskd --noreload --replayfailed -f /opt/rockstor/var/log/ztask.log
├─7072 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
└─7073 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application

May 03 21:25:35 nas03 supervisord[5944]: 2016-05-03 21:25:35,010 INFO gave up: nginx entered FATAL state, too many start retries too quickly
May 03 21:34:54 nas03 supervisord[5944]: 2016-05-03 21:34:54,926 INFO spawned: ‘nginx’ with pid 11755
May 03 21:34:54 nas03 supervisord[5944]: 2016-05-03 21:34:54,939 INFO exited: nginx (exit status 1; not expected)
May 03 21:34:55 nas03 supervisord[5944]: 2016-05-03 21:34:55,944 INFO spawned: ‘nginx’ with pid 11756
May 03 21:34:55 nas03 supervisord[5944]: 2016-05-03 21:34:55,957 INFO exited: nginx (exit status 1; not expected)
May 03 21:34:57 nas03 supervisord[5944]: 2016-05-03 21:34:57,965 INFO spawned: ‘nginx’ with pid 11778
May 03 21:34:57 nas03 supervisord[5944]: 2016-05-03 21:34:57,977 INFO exited: nginx (exit status 1; not expected)
May 03 21:35:00 nas03 supervisord[5944]: 2016-05-03 21:35:00,984 INFO spawned: ‘nginx’ with pid 11779
May 03 21:35:00 nas03 supervisord[5944]: 2016-05-03 21:35:00,996 INFO exited: nginx (exit status 1; not expected)
May 03 21:35:01 nas03 supervisord[5944]: 2016-05-03 21:35:01,998 INFO gave up: nginx entered FATAL state, too many start retries too quickly

● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: active (exited) since Tue 2016-05-03 21:26:11 BST; 14min ago
Main PID: 5946 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-bootstrap.service

May 03 21:26:11 nas03 bootstrap[5946]: BTRFS device scan complete
May 03 21:26:11 nas03 bootstrap[5946]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and
try again. Exception: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host=‘127.0.0.1’, port=8000): Max retries exceeded with url: /o/token/ (Caused by <class ‘socket.error’>: [Errno 111] Connection refused). content: None
May 03 21:26:11 nas03 bootstrap[5946]: Bootstrapping complete
May 03 21:26:11 nas03 bootstrap[5946]: Running qgroup cleanup. /opt/rockstor/bin/qgroup-clean
May 03 21:26:11 nas03 bootstrap[5946]: Running qgroup limit maxout. /opt/rockstor/bin/qgroup-maxout-limit

So nginx doesn’t look healthy. supervisord.log looks much the same as the rockstor service output.

supervisord_replication_stderr.log is full of these:

Process ReplicaScheduler-1:
Traceback (most recent call last):
File “/usr/lib64/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
self.run()
File “/opt/rockstor/src/rockstor/smart_manager/replication/listener_broker.py”, line 185, in run
frontend.bind(‘tcp://%s:%d’ % (self.listener_interface, self.listener_port))
File “zmq/backend/cython/socket.pyx”, line 487, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:5156)
File “zmq/backend/cython/checkrc.pxd”, line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:7535)
raise ZMQError(errno)
ZMQError: No such device

Finally, running: /opt/rockstor/bin/initrock shows:

2016-05-03 21:34:52,976: Supported kernel(/boot/vmlinuz-4.4.5-1.el7.elrepo.x86_64) is already the default
2016-05-03 21:34:53,110: /etc/rc.d/rc.local looks correct. Not updating.
2016-05-03 21:34:53,110: Checking for flash and Running flash optimizations if appropriate.
2016-05-03 21:34:53,723: Updating the timezone from the system
2016-05-03 21:34:53,724: system timezone = Europe/London
2016-05-03 21:34:53,725: Updating sshd_config
2016-05-03 21:34:53,726: sshd_config already has the updates. Leaving it unchanged.
2016-05-03 21:34:53,726: Running prepdb…
2016-05-03 21:34:54,438: stopping firewalld…
2016-05-03 21:34:54,504: firewalld stopped and disabled
2016-05-03 21:34:54,588: Registered task: fs.btrfs.start_balance
2016-05-03 21:34:57,503: rockstor service looks correct. Not updating.
2016-05-03 21:34:57,503: rockstor-bootstrap.service looks correct. Not updating.

Any ideas?

suman · May 4, 2016, 5:17am

nginx logs are in /var/log/nginx directory. There should be clues in there.

restarting rockstor service is also worth a try: systemctl restart rockstor

grizzly · May 4, 2016, 6:49am

No luck with systemctl restart rockstor.
/var/log/nginx/error.log contains:

2016/05/04 07:41:12 [emerg] 14606#0: no “events” section in configuration
2016/05/04 07:41:13 [emerg] 14636#0: no “events” section in configuration
2016/05/04 07:41:16 [emerg] 14794#0: no “events” section in configuration
2016/05/04 07:41:19 [emerg] 14861#0: no “events” section in configuration

Prime · May 4, 2016, 8:04pm

This release is broken as $%&* just revert to 3.8.12 until it’s fixed
I would also recommend you have a test machine for testing new updates so you don’t mess up your production server.

grizzly · May 4, 2016, 8:10pm

I have just fixed. For some reason /opt/rockstor/etc/nginx/nginx.conf was empty (zero kb). I copied this from another Rockstor, restarted the rockstor service, and now WebUI starts fine Not sure how it happened. I configured NIC Teaming (LoadBalance), perhaps this attempts to change this config file…

Prime · May 4, 2016, 8:11pm

Yes it does but you can just make your own config.

hms · May 5, 2016, 12:43pm

How can I revert to .13-04 (I think this was working ok, maybe 05 too) from terminal, this can’t run yum and won’t start rockstor anymore. It’s complaining about cache/ $releasever repomod.xml.old.tmp is readonly (this was listed on issues too #1303 ), most likely because I have(had) added repo’s for KVM, firmware jenkings and epel epel-testing.

systemctl restart rockstor-pre

Error getting authority: Error initializing authority: Error calling StartServiceByName for org.freedesktop.PolicyKit1: Timeout was reached (g-io-error-quark, 24)

yum install rockstor-3.8-13.07

[Errno 30] Writeprotected filesystem: ‘/var/cache/yum/x86_64/$releasever/Rockstor-Testing/repomd.xml.old.tmp’

I’m running it from USB-stick, whats proper way to check btrfs FS (rockstor os partition)?