Bootstrap service disabled after v4.5.4-0 upgrade

Hello all,

Is it just me, or was the Bootstrap service disabled for you too, after the upgrade finished?
I’ve upgraded from v4.1.

1 Like

It’s running on mine:

rockstor4:/var/log # systemctl status -l rockstor rockstor-bootstrap
● rockstor.service - Rockstor startup script
Loaded: loaded (/usr/lib/systemd/system/rockstor.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2023-01-18 11:07:28 CET; 18min ago
Main PID: 15248 (supervisord)
Tasks: 11
CGroup: /system.slice/rockstor.service
├─15248 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
├─15286 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rocks>
├─15287 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/data-collector
├─15288 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/django-admin run_huey --workers 2 --worker-type thread --logfile /opt/rockstor/var/log/huey.log
├─15298 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rocks>
└─15303 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rocks>
Jan 18 11:07:28 rockstor4 systemd[1]: Started Rockstor startup script.
Jan 18 11:07:28 rockstor4 poetry[15248]: 2023-01-18 11:07:28,771 INFO Set uid to user 0 succeeded
Jan 18 11:07:28 rockstor4 poetry[15248]: 2023-01-18 11:07:28,777 INFO RPC interface ‘supervisor’ initialized
Jan 18 11:07:28 rockstor4 poetry[15248]: 2023-01-18 11:07:28,777 INFO supervisord started with pid 15248
Jan 18 11:07:29 rockstor4 poetry[15248]: 2023-01-18 11:07:29,783 INFO spawned: ‘gunicorn’ with pid 15286
Jan 18 11:07:29 rockstor4 poetry[15248]: 2023-01-18 11:07:29,786 INFO spawned: ‘data-collector’ with pid 15287
Jan 18 11:07:29 rockstor4 poetry[15248]: 2023-01-18 11:07:29,788 INFO spawned: ‘ztask-daemon’ with pid 15288
Jan 18 11:07:31 rockstor4 poetry[15248]: 2023-01-18 11:07:31,791 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 18 11:07:34 rockstor4 poetry[15248]: 2023-01-18 11:07:34,795 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Jan 18 11:07:39 rockstor4 poetry[15248]: 2023-01-18 11:07:39,801 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/usr/lib/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2023-01-18 11:07:45 CET; 17min ago
Process: 15249 ExecStart=/opt/rockstor/.venv/bin/bootstrap (code=exited, status=0/SUCCESS)
Main PID: 15249 (code=exited, status=0/SUCCESS)
Jan 18 11:07:28 rockstor4 systemd[1]: Starting Rockstor bootstrapping tasks…
Jan 18 11:07:45 rockstor4 bootstrap[15249]: BTRFS device scan complete
Jan 18 11:07:45 rockstor4 bootstrap[15249]: Bootstrapping complete
Jan 18 11:07:45 rockstor4 bootstrap[15249]: Running qgroup cleanup. /opt/rockstor/.venv/bin/qgroup-clean
Jan 18 11:07:45 rockstor4 bootstrap[15249]: Running qgroup limit maxout. /opt/rockstor/.venv/bin/qgroup-maxout-limit
Jan 18 11:07:45 rockstor4 systemd[1]: Finished Rockstor bootstrapping tasks.

1 Like

Hi @aremiaskfa,

Thanks for the report. Could you paste the output of the corresponding journal log?

journactl -b0 -u rockstor.bootstrap

It should tell us more about why it didn’t start in your case.

Also the output of the same command as @Mark93 would be helpful too.

systemctl status -l rockstor.bootstrap
1 Like

Sry for breviness, I’m on mobile.

First command said No entries.
Second command said Unit could not be found.

If I repeat the commands for rockstor-bootstrap.service instead:

Maybe it was only a UI glitch?

2 Likes

That’s entirely my fault… I apologize; this is what happens when I write in a rush :frowning:
It should indeed be rockstor-bootstrap as you found out.

That is curious indeed… You didn’t reboot or restart a service or anything like that?

Repro steps:
Upgrade via web ui
Reboot PC
Button was red in UI.

Unfortunately I didn’t check the status of the Bootstrap for their last 24hours. I only remember it was green then last time I checked it.

Anyway, it’s a moot point and it’s suggest closing this bug until I manage to get repro it again.

Thanks for your support.

3 Likes

Hello all.

I can reproduce this bug every update. It doesnt matter if in 15.3 or 15.4 version after a fresh install.
This time I have saved the logs before turning it on again.

UI:

systemctl:

● rockstor.service - Rockstor startup script
     Loaded: loaded (/usr/lib/systemd/system/rockstor.service; disabled; vendor preset: disabled)
     Active: active (running) since Thu 2023-01-26 16:38:45 CET; 19h ago
   Main PID: 10742 (supervisord)
      Tasks: 11
     CGroup: /system.slice/rockstor.service
             ├─ 10742 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
             ├─ 10758 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
             ├─ 10759 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/data-collector
             ├─ 10760 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/django-admin run_huey --workers 2 --worker-type thread --logfile /opt/rockstor/var/log/huey.log
             ├─ 10774 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
             └─ 10778 /opt/rockstor/.venv/bin/python /opt/rockstor/.venv/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application

Jan 26 16:38:45 rockstor systemd[1]: Started Rockstor startup script.
Jan 26 16:38:45 rockstor poetry[10742]: 2023-01-26 16:38:45,882 INFO Set uid to user 0 succeeded
Jan 26 16:38:45 rockstor poetry[10742]: 2023-01-26 16:38:45,896 INFO RPC interface 'supervisor' initialized
Jan 26 16:38:45 rockstor poetry[10742]: 2023-01-26 16:38:45,896 INFO supervisord started with pid 10742
Jan 26 16:38:46 rockstor poetry[10742]: 2023-01-26 16:38:46,899 INFO spawned: 'gunicorn' with pid 10758
Jan 26 16:38:46 rockstor poetry[10742]: 2023-01-26 16:38:46,901 INFO spawned: 'data-collector' with pid 10759
Jan 26 16:38:46 rockstor poetry[10742]: 2023-01-26 16:38:46,903 INFO spawned: 'ztask-daemon' with pid 10760
Jan 26 16:38:48 rockstor poetry[10742]: 2023-01-26 16:38:48,907 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 26 16:38:51 rockstor poetry[10742]: 2023-01-26 16:38:51,912 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Jan 26 16:38:56 rockstor poetry[10742]: 2023-01-26 16:38:56,919 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

○ rockstor-bootstrap.service - Rockstor bootstrapping tasks
     Loaded: loaded (/usr/lib/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2023-01-26 16:38:26 CET; 19h ago
   Main PID: 27010 (code=exited, status=0/SUCCESS)

Jan 26 14:23:24 rockstor systemd[1]: Starting Rockstor bootstrapping tasks...
Jan 26 14:23:30 rockstor bootstrap[27010]: BTRFS device scan complete
Jan 26 14:23:30 rockstor bootstrap[27010]: Bootstrapping complete
Jan 26 14:23:30 rockstor bootstrap[27010]: Running qgroup cleanup. /opt/rockstor/.venv/bin/qgroup-clean
Jan 26 14:23:30 rockstor bootstrap[27010]: Running qgroup limit maxout. /opt/rockstor/.venv/bin/qgroup-maxout-limit
Jan 26 14:23:30 rockstor systemd[1]: Finished Rockstor bootstrapping tasks.
Jan 26 16:38:26 rockstor systemd[1]: rockstor-bootstrap.service: Deactivated successfully.
Jan 26 16:38:26 rockstor systemd[1]: Stopped Rockstor bootstrapping tasks.

journalctl:

Jan 26 14:18:57 rockstor systemd[1]: rockstor-bootstrap.service: Deactivated successfully.
Jan 26 14:18:57 rockstor systemd[1]: Stopped Rockstor bootstrapping tasks.
Jan 26 14:19:11 rockstor systemd[1]: Starting Rockstor bootstrapping tasks...
Jan 26 14:19:17 rockstor bootstrap[21358]: BTRFS device scan complete
Jan 26 14:19:17 rockstor bootstrap[21358]: Bootstrapping complete
Jan 26 14:19:17 rockstor bootstrap[21358]: Running qgroup cleanup. /opt/rockstor/.venv/bin/qgroup-clean
Jan 26 14:19:17 rockstor bootstrap[21358]: Running qgroup limit maxout. /opt/rockstor/.venv/bin/qgroup-maxout-limit
Jan 26 14:19:17 rockstor systemd[1]: Finished Rockstor bootstrapping tasks.
Jan 26 14:20:42 rockstor systemd[1]: rockstor-bootstrap.service: Deactivated successfully.
Jan 26 14:20:42 rockstor systemd[1]: Stopped Rockstor bootstrapping tasks.
Jan 26 14:20:52 rockstor systemd[1]: Starting Rockstor bootstrapping tasks...
Jan 26 14:20:59 rockstor bootstrap[24265]: BTRFS device scan complete
Jan 26 14:20:59 rockstor bootstrap[24265]: Bootstrapping complete
Jan 26 14:20:59 rockstor bootstrap[24265]: Running qgroup cleanup. /opt/rockstor/.venv/bin/qgroup-clean
Jan 26 14:20:59 rockstor bootstrap[24265]: Running qgroup limit maxout. /opt/rockstor/.venv/bin/qgroup-maxout-limit
Jan 26 14:20:59 rockstor systemd[1]: Finished Rockstor bootstrapping tasks.
Jan 26 14:23:11 rockstor systemd[1]: rockstor-bootstrap.service: Deactivated successfully.
Jan 26 14:23:11 rockstor systemd[1]: Stopped Rockstor bootstrapping tasks.
Jan 26 14:23:24 rockstor systemd[1]: Starting Rockstor bootstrapping tasks...
Jan 26 14:23:30 rockstor bootstrap[27010]: BTRFS device scan complete
Jan 26 14:23:30 rockstor bootstrap[27010]: Bootstrapping complete
Jan 26 14:23:30 rockstor bootstrap[27010]: Running qgroup cleanup. /opt/rockstor/.venv/bin/qgroup-clean
Jan 26 14:23:30 rockstor bootstrap[27010]: Running qgroup limit maxout. /opt/rockstor/.venv/bin/qgroup-maxout-limit
Jan 26 14:23:30 rockstor systemd[1]: Finished Rockstor bootstrapping tasks.
Jan 26 16:38:26 rockstor systemd[1]: rockstor-bootstrap.service: Deactivated successfully.
Jan 26 16:38:26 rockstor systemd[1]: Stopped Rockstor bootstrapping tasks.

Going from zypper log, I started update at 2023-01-26 16:38:29

2023-01-26 16:38:29|command|root@rockstor|'/usr/bin/zypper' '--non-interactive' 'install' 'rockstor'|
2023-01-26 16:38:31|install|rockstor|4.5.6-0|x86_64|root@rockstor|Rockstor-Testing|429fba2721dce92f9d27b9eb4977ef24042e9ba258186721ff2e9892667bf58a|
# 2023-01-26 16:38:34 Output of rockstor-4.5.6-0.x86_64.rpm %posttrans script:
# Retrieving Poetry metadata

@aremiaskfa Hello there.
Re:

Is this from a Rockstor installer instance, or a custom rpm install.

This looks to be correct, i.e it’s enabled by default via our rockstor presets app from OBS.

Actually I think I know where we are going wrong here.

I recently made a change to our OBS app here:

to move from enabling the rockstor.service (olden days we didn’t have a rockstor-bootstrap)
to enabling instead the rockstor-bootstrap service:

Because in more recent tidy-ups this has sufficient dependencies to start all other services. Hence the top reference there. When I made that change and later did some work on the packaging I found this:

And meant to issue it. But obviously failed with all the distractions. Fancy stepping in and creating the appropriate issue (pr maybe) as it is now the case that rockstor-bootstrap.service invokes all the rest, in order. So I think it would be good to have that tally with our systemd-presets-branding-rockstor app.

If you look around the above bit of code you see that we create a bash script to enable us to update ourselves, then run that script (which cleans up after itself) via AT. The assists with what would otherwise be a tricky manoeuvre.

Cheers. And thanks for the prompt on that one. I had meant to get back to that and you debuggin here reminded me. Do steam in with that issue and pr is you fancy. Informed help always appreciated.

Hope that helps.

3 Likes

Hi,

The fresh install was from the RC2 iso image.

Issue created:

3 Likes

@aremiaskfa Just a quick follow-up on your:

This issue has now been addressed as part of our ongoing GitHub Milestone towards the next stable release.

See the release notes just published here on the forum:

Note however it will not be untill the next release that we will have full field testing given the nature of this change. But I think we do have this sorted now.

Thanks again for the excellent reporting, this has been an issue for a long time now and it just needed a ‘sponsor’ of sorts.

Hope that help, and remember to keep an eye on this when we have our next release (likely 4.5.8-0) as that is when we will have the proof-in-the-pudding as it were.

3 Likes