@bbarnhill A very belated welcome to the Rockstor forum.
Thanks for reporting this. Yes, it appears we have an issue with the latest 3.8.16-7 testing channel release, the suggestion is to hold off on updating on the testing channel until a fix is released.
Cheers for reporting in a timely manner.
Hi @bbarnhill,
did you try a second reboot??
We all had some testing and on my side a second reboot solved it
M.
Oh yes I have rebooted several times. I can not get my UI back up and running. Im hoping its salvageable. Any ideas?
I cant hold off updating since Ive already updated. Is there a way to back out an update? Preferably from the CLI since thats all I have access to?
Hi @bbarnhill,
can you please report from console some Rockstor logs?
cat /opt/rockstor/var/log/rockstor.log or tail -n 100 /opt/rockstor/var/log/rockstor.log after a reboot
Mirko
I am not the thread opener, but I ran into the same problem.
There is no error information in rockstor.log for today, but the attached sysctl state could be of value for you.
There is a mismatch in the IP adresses, which is assigned to my NIC, while the web service is reaching out for localhost.
Shares and SSH are still working, though, so it is not a general network issue, rather than related to the web service itself.
Greetings from Pinky & TheBrain
[root@brainstor /]# systemctl status -l rockstor-pre rockstor rockstor-bootstrap
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: active (exited) since Di 2017-01-17 19:40:28 CET; 10min ago
Main PID: 3504 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-pre.service
Jan 17 19:40:20 brainstor initrock[3504]: 2017-01-17 19:40:20,957: system timezone = Europe/Berlin
Jan 17 19:40:20 brainstor initrock[3504]: 2017-01-17 19:40:20,960: Updating sshd_config
Jan 17 19:40:20 brainstor initrock[3504]: 2017-01-17 19:40:20,966: sshd_config already has the updates. Leaving it unchanged.
Jan 17 19:40:20 brainstor initrock[3504]: 2017-01-17 19:40:20,966: Running prepdb…
Jan 17 19:40:23 brainstor initrock[3504]: 2017-01-17 19:40:23,914: stopping firewalld…
Jan 17 19:40:24 brainstor initrock[3504]: 2017-01-17 19:40:24,070: firewalld stopped and disabled
Jan 17 19:40:28 brainstor initrock[3504]: 2017-01-17 19:40:28,204: rockstor service looks correct. Not updating.
Jan 17 19:40:28 brainstor initrock[3504]: 2017-01-17 19:40:28,205: rockstor-bootstrap.service looks correct. Not updating.
Jan 17 19:40:28 brainstor initrock[3504]: default_if = enp3s0
Jan 17 19:40:28 brainstor initrock[3504]: ipaddr = 192.168.178.29
● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: active (running) since Di 2017-01-17 19:40:28 CET; 10min ago
Main PID: 6541 (supervisord)
CGroup: /system.slice/rockstor.service
├─6541 /usr/bin/python /opt/rockstor/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
├─7150 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
├─7153 /usr/bin/python /opt/rockstor/bin/data-collector
├─7157 /usr/bin/python2.7 /opt/rockstor/bin/django ztaskd --noreload --replayfailed -f /opt/rockstor/var/log/ztask.log
├─7697 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
└─7698 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
Jan 17 19:40:31 brainstor supervisord[6541]: 2017-01-17 19:40:31,771 INFO spawned: ‘nginx’ with pid 7556
Jan 17 19:40:31 brainstor supervisord[6541]: 2017-01-17 19:40:31,845 INFO exited: nginx (exit status 1; not expected)
Jan 17 19:40:32 brainstor supervisord[6541]: 2017-01-17 19:40:32,848 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 17 19:40:32 brainstor supervisord[6541]: 2017-01-17 19:40:32,850 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 17 19:40:33 brainstor supervisord[6541]: 2017-01-17 19:40:33,860 INFO spawned: ‘nginx’ with pid 8127
Jan 17 19:40:33 brainstor supervisord[6541]: 2017-01-17 19:40:33,965 INFO exited: nginx (exit status 1; not expected)
Jan 17 19:40:35 brainstor supervisord[6541]: 2017-01-17 19:40:35,968 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Jan 17 19:40:36 brainstor supervisord[6541]: 2017-01-17 19:40:36,974 INFO spawned: ‘nginx’ with pid 9088
Jan 17 19:40:37 brainstor supervisord[6541]: 2017-01-17 19:40:37,014 INFO exited: nginx (exit status 1; not expected)
Jan 17 19:40:38 brainstor supervisord[6541]: 2017-01-17 19:40:38,016 INFO gave up: nginx entered FATAL state, too many start retries too quickly
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: active (exited) since Di 2017-01-17 19:40:47 CET; 10min ago
Main PID: 6548 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-bootstrap.service
Jan 17 19:40:47 brainstor bootstrap[6548]: BTRFS device scan complete
Jan 17 19:40:47 brainstor bootstrap[6548]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host=‘127.0.0.1’, port=8000): Max retries exceeded with url: /o/token/ (Caused by <class ‘socket.error’>: [Errno 111] Connection refused). content: None
Jan 17 19:40:47 brainstor bootstrap[6548]: Bootstrapping complete
Jan 17 19:40:47 brainstor bootstrap[6548]: Running qgroup cleanup. /opt/rockstor/bin/qgroup-clean
Jan 17 19:40:47 brainstor bootstrap[6548]: Running qgroup limit maxout. /opt/rockstor/bin/qgroup-maxout-limit
Hi @Christian_Rost,
thanks for your systemctl! 
rockstor-pre -> OK
rockstor.service -> here we come with nginx failing so WebUI not accessible & others
can you provide nginx log from /opt/rockstor/var/log/*everything_with_nginx?
Thanks
M.
P.S.: Had same issue, actually not able to reproduce it after some reboots
Update:
Rollback to Rockstor 3.8.16-1, update with yum to latest Rockstor, reboot and all working fine
There’s only a small error on nginx while services waking, but that’s ok
Are you updating from 3.8-16-1 or others?
M.
I do not know what the version before was. I never tried a rollback.
nginx stdout is of no value.
This is what stderr shows:
nginx: [emerg] host not found in “None:443” of the “listen” directive in /opt/rockstor/etc/nginx/nginx.conf:45
Doing rollback once again, but checking my /etc/nginx/ folder found a weird nginx.conf file, while nginx.conf.rpmsave is ok
@phillxnet & @suman?
I upgraded from 3.8.16-6 to 3.8.16-7
How are you rolling back to the older version via yum?
Proxmox under Rockstor, rollback with snapshots for tests (not Rockstor itself)
Good news: we found code related to issue
Bad news: not able to reproduce it now
M.
rolled back to 3.8.16-6 and now it wont boot … looks like Im reinstalling
I think you missed that after the rollback to 3.8.16-6 that my machine would no longer boot. I have run through the reinstall process 3 times though and only had the one SSD in the machine and it will not boot after it installs. It gets to where it should show the grub menu and it has a single period on the screen and a blinking cursor and stops.
Thanks @Flyer and @phillxnet for doing everything you can to help troubleshoot the problem. @bbarnhill, sorry to hear your box is unbootable at the moment. The bug though, is a small one that produced an unusable nginx config file. nginx is the webserver gateway and is crucial as it proxies the UI and backend services. Having said that, I am sure your issue is not really related to the Rockstor bug. It’s probably a compound issue that is triggered by upstream package updates or hardware failure or something else. Rockstor bug probably prompted you to reboot which exposed other issues? Sorry, I know that doesn’t help. Just some thoughts.
I had the same issue regarding a boot stop before grub should come up a while ago.
After a lot of research it turned out that the btrfs got corrupted pretty mad. After a full wipe of the disk it installed just fine.
I just did that by using dd and overwriting the whole disk with /dev/zero.
What you are describing makes me also think more of a hardware related problem. As Suman already wrote, I do not believe it is related to the Rockstor service itself.
Im going to replace the ssd and see if that helps as I was leaning that way after it locked up several times in the install.
That’s a very good point actually. On systems installed prior to 3.8.16 ISO, the rootfs goes out of balance over time due purely to accounting mistake by btrfs and leads to out of space errors. However, I have not see this problem become so bad that it won’t even drop you into dracut emergency shell. If you are dropped into it or if the system manages to boot but won’t let you do anything more than logging in, you can rectify the problem by running a balance job. Depending on severity, you may have to run it a few times with different filter arguments. BTRFS wiki post has more info.
I believe systems installed with 3.8.16 ISO do auto balance the rootfs as expected.
reinstallation worked just fine. Ran into a couple of issues installing but those were all due to hardware things such as leaving the top off the rack mount case and RAM getting hot or something like that. Once those were ironed out the box is now back up and running and running on a stable release.