I had this occur on my stable machine when it auto-updated to 3.9.0 this morning
Appears to be working after a yum downgrade rockstor (which put it to 3.8.16) and then update + reboot from the gui
2 Likes
Awesome! Fixed my problem! Thank you!
1 Like
Just want to chime in I had the same problem going from 3.8.15-9 to 3.8.16-16.
yum upgrade rockstor-3.8.16-16 ### upgrade from 3.8.15-9. Post-Install RPM Failure!!!
systemctl restart rockstor-pre rockstor rockstor-bootstrap ### FAILURE!!!
I downgraded to 3.8.16-12 (from 3.8.16-16) first, restarted Rockstor, then upgraded back to 3.8.16-16. That seemed to work this time.
yum downgrade rockstor-3.8.16-12
systemctl restart rockstor-pre rockstor rockstor-bootstrap
yum upgrade rockstor-3.8.16-16
systemctl restart rockstor-pre rockstor rockstor-bootstrap
Hi all, this is a copy&paste message on multiple forum threads:
checking over a Github ad hoc issue, any news on this after last Rockstor update?
On my side: some checks required with Rock-ons share usage (probably docker related), but no blocking errors like missing db fields, etc
Thanks
I have this problem too after auto upgrading to 3.9.0, trying to downgrade. But has any solution been found? I really don’t like being on old releases.
The root cause of this issue is a bug in our schema migration logic. I fixed it in https://github.com/rockstor/rockstor-core/pull/1694 which went into 3.9.0-4
1 Like
hi @suman - i am on 3.9.0-5 and still have the issue listed:
Houston, we’ve had a problem.
column storageadmin_share.pqgroup_rusage does not exist LINE 1: …n_share".“rusage”, “storageadmin_share”.“eusage”, "storagead… ^
I have been a Stable Update Channel user - but after experiencing this issue - switched to Testing Updates channel to access the 0-4 release - this has actually provided me the 0-5 release but the error persists.
Upgraded from 3.8 to 3.9 (stable) here and had this problem. Tried:
yum downgrade rockstor-3.8.16-12
but errored “No package rockstor-3.8.16-12 available”
Then:
yum downgrade rockstor
systemctl restart rockstor-pre rockstor rockstor-bootstrap
But errored:
Job for rockstor-bootstrap.service failed because the control process exited with error code. See “systemctl status rockstor-bootstrap.service” … for details
Bootstrap service won’t start:
[root@nas ~]# systemctl status rockstor-bootstrap.service
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-04-17 23:04:23 BST; 1min 12s ago
Process: 17144 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=1/FAILURE)
Main PID: 17144 (code=exited, status=1/FAILURE)
Then upgraded back to 3.9 using webui, which got rid of the GUI errors. However rockstor-bootstrap service still won’t start and my main NFS share couldn’t be found (samba was fine). NFS service was started (tried restarting it). Webui showed
Unknown internal error doing a POST to /api/sm/services/rockstor-bootstrap/start
Rebooted, same state (except clock became correct timezone). Tried to recreate NFS share I needed but pressing trash icon did nothing. So just created a new NFS share for the same share, but with * for host and nfs client instead of the FQDN of the host that needs it. Now can access this share. But not a smooth upgrade experience, and there are still glitches:
- Can’t delete NFS shares
- Bootstrap service is stopped - should this be started? If so, how?
- Should I temporarily change to testing updates to get to -05, then back onto stable?
Could you reply with the output of systemctl status -l rockstor-pre command?
Really sorry this nasty db migration bug go into stable. The easiest way to fix it is to update to -05, but also make sure rockstor-pre service has restarted. After update, please capture the output of systemctl status -l rockstor-pre and if it has not restarted/started recently, go ahead and restart it with systemctl restart rockstor-pre. If still having problems, reply with output of systemctl status -l rockstor-pre rockstor rockstor-bootstrap
Upgraded to -05. Can delete NFS shares now. systemctl status -l rockstor-pre says:
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: active (exited) since Tue 2017-04-18 09:24:45 BST; 1h 40min ago
Main PID: 3702 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-pre.service
Apr 18 09:24:35 nas initrock[3702]: 2017-04-18 03:24:35,888: Running app database migrations…
Apr 18 09:24:44 nas initrock[3702]: 2017-04-18 03:24:44,185: Done
Apr 18 09:24:44 nas initrock[3702]: 2017-04-18 03:24:44,186: Running prepdb…
Apr 18 09:24:45 nas initrock[3702]: 2017-04-18 03:24:45,211: Done
Apr 18 09:24:45 nas initrock[3702]: 2017-04-18 03:24:45,212: stopping firewalld…
Apr 18 09:24:45 nas initrock[3702]: 2017-04-18 03:24:45,306: firewalld stopped and disabled
Apr 18 09:24:45 nas initrock[3702]: 2017-04-18 03:24:45,829: rockstor service looks correct. Not updating.
Apr 18 09:24:45 nas initrock[3702]: 2017-04-18 03:24:45,829: rockstor-bootstrap.service looks correct. Not updating.
Apr 18 09:24:45 nas initrock[3702]: default_if = nas03_team
Apr 18 09:24:45 nas initrock[3702]: ipaddr = #.#.#.#
Does “Active (Exited)” mean it has started successfully? Which genius Linux kernel dev dreamed up such an unintuitive status!? My guess is it’s started because has green text, not red. In any case, the “bootstrap” service in webui still shows stopped:
Other statuses, rockstor-bootstrap:
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2017-04-18 11:04:48 BST; 12min ago
Process: 20213 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=1/FAILURE)
Main PID: 20213 (code=exited, status=1/FAILURE)
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 18 11:04:48 nas bootstrap[20213]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: [‘Internal Server Error: No JSON object could be decoded’]
and rockstor:
● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2017-04-18 09:24:45 BST; 1h 52min ago
Main PID: 9367 (supervisord)
CGroup: /system.slice/rockstor.service
├─ 9367 /usr/bin/python /opt/rockstor/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
├─10157 nginx: master process /usr/sbin/nginx -c /opt/rockstor/etc/nginx/nginx.con
├─10158 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log
–pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
├─10159 /usr/bin/python /opt/rockstor/bin/data-collector
├─10160 /usr/bin/python2.7 /opt/rockstor/bin/django ztaskd --noreload --replayfailed -f /opt/rockstor/var/log/ztask.log
├─10161 nginx: worker process
├─10162 nginx: worker process
├─10226 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log
–pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
└─10227 /usr/bin/python /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log
–pythonpath=/opt/rockstor/src/rockstor --settings=settings --timeout=120 --graceful-timeout=120 wsgi:application
Apr 18 09:24:46 nas supervisord[9367]: 2017-04-18 09:24:46,405 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
Apr 18 09:24:46 nas supervisord[9367]: 2017-04-18 09:24:46,405 INFO supervisord started with pid 9367
Apr 18 09:24:47 nas supervisord[9367]: 2017-04-18 09:24:47,408 INFO spawned: ‘nginx’ with pid 10157
Apr 18 09:24:47 nas supervisord[9367]: 2017-04-18 09:24:47,410 INFO spawned: ‘gunicorn’ with pid 10158
Apr 18 09:24:47 nas supervisord[9367]: 2017-04-18 09:24:47,414 INFO spawned: ‘data-collector’ with pid 10159
Apr 18 09:24:47 nas supervisord[9367]: 2017-04-18 09:24:47,424 INFO spawned: ‘ztask-daemon’ with pid 10160
Apr 18 09:24:49 nas supervisord[9367]: 2017-04-18 09:24:49,432 INFO success: data-collector entered RUNNING state, process has stayed up for > than
2 seconds (startsecs)
Apr 18 09:24:49 nas supervisord[9367]: 2017-04-18 09:24:49,432 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2
seconds (startsecs)
Apr 18 09:24:52 nas supervisord[9367]: 2017-04-18 09:24:52,436 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Apr 18 09:24:52 nas supervisord[9367]: 2017-04-18 09:24:52,436 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Maybe I can just ignore bootstrap and assume this issue is benign, no other problems now.
Hi @suman_chakravartula
It appears that is now working - however i was required to boot with a previous kernel version.
I now report:
You are running an unsupported kernel(4.8.7-1.el7.elrepo.x86_64). Some features may not work properly. Please reboot and the system will automatically boot using the supported kernel(4.10.6-1.el7.elrepo.x86_64)
Am running a HP N40L - i think i have seen an existing thread about this with the new kernel.
1 Like
I got the exact same error today after I completed the stable update to 3.9.
Upgrading to 3.9.0-5 solved the problem. At first I restarted the services (not the system). Of course I got the message “unsupported kernel”, but the “storageadmin_share…” error was already gone.
Rebooting the machine was the final step to have the recent kernel loaded and now everything’s fine (as far as I can tell).
2 Likes
Hi guys happy to read this seems solved
Can we mark it solved?
Mirko
[quote=“grizzly, post:31, topic:2847”]
Maybe I can just ignore bootstrap and assume this issue is benign, no other problems now.
[/quote]I’m seeing weird stuff, so I no longer think ignoring the failed bootstrap service is sensible. How can I troubleshoot rockstor-bootstrap.service failing to start? :
[root@nas mnt2]# systemctl status rockstor-bootstrap.service -l
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2017-04-25 12:48:30 BST; 4min 27s ago
Process: 15242 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=1/FAILURE)
Main PID: 15242 (code=exited, status=1/FAILURE)
Apr 25 12:48:30 nas bootstrap[15242]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait
…
2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Apr 25 12:48:30 nas bootstrap[15242]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: [‘Internal Server Error: No JSON object could be decoded’]
Apr 25 12:48:30 nas systemd[1]: rockstor-bootstrap.service: main process exited, code=exited, status=1/FAILURE
Apr 25 12:48:30 nas systemd[1]: Failed to start Rockstor bootstrapping tasks.
Apr 25 12:48:30 nas systemd[1]: Unit rockstor-bootstrap.service entered failed state.
Apr 25 12:48:30 nas systemd[1]: rockstor-bootstrap.service failed.
Followed suggestion in this forum topic for a similar error, to check the existence of “/opt/rockstor/src/rockstor/storageadmin/migrations/0042_auto__add_field_disk_smart_options__add_field_disk_role.py”, but I have no such file
Any suggestions? Pretty close to trashing this installation, reinstalling Rockstor and reimporting pools & shares.
That file doesn’t exist as of 3.8.0
What’s your current Rockstor version?
My Rockstor is v3.9.0-5
What’s the output of systemctl status -l rockstor-pre. Restart that service to see if you get the same error: systemctl restart rockstor-pre and finally, what’s the output of /opt/rockstor/bin/django showmigrations
That’s started fine:
[root@nas ~]# systemctl status -l rockstor-pre
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: active (exited) since Tue 2017-04-25 13:19:32 BST; 19h ago
Main PID: 3580 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rockstor-pre.service
Apr 25 13:19:30 nas initrock[3580]: 2017-04-25 13:19:30,627: Done
Apr 25 13:19:30 nas initrock[3580]: 2017-04-25 13:19:30,627: Running prepdb…
Apr 25 13:19:31 nas initrock[3580]: 2017-04-25 13:19:31,786: Done
Apr 25 13:19:31 nas initrock[3580]: 2017-04-25 13:19:31,786: stopping firewalld…
Apr 25 13:19:31 nas initrock[3580]: 2017-04-25 13:19:31,906: firewalld stopped and disabled
Apr 25 13:19:32 nas initrock[3580]: 2017-04-25 13:19:32,284: rockstor service looks correct. Not updating.
Apr 25 13:19:32 nas initrock[3580]: 2017-04-25 13:19:32,286: rockstor-bootstrap.service looks correct. Not updating.
Apr 25 13:19:32 nas initrock[3580]: default_if = nas
Apr 25 13:19:32 nas initrock[3580]: ipaddr = #.#.#.#
Apr 25 13:19:32 nas systemd[1]: Started Tasks required prior to starting Rockstor.
and:
[root@nas ~]# /opt/rockstor/bin/django showmigrations
admin
[ ] 0001_initial
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
django_ztask
(no migrations)
oauth2_provider
[X] 0001_initial
[ ] 0002_08_updates
sessions
[ ] 0001_initial
sites
[ ] 0001_initial
smart_manager
[ ] 0001_initial
[ ] 0002_auto_20170216_1212
storageadmin
[X] 0001_initial
[X] 0002_auto_20161125_0051
[X] 0003_auto_20170114_1332
[root@nas ~]#
How do you install 3.9.0-4(or 5)? My repo only shows 3.9.0