RockStor not starting

HarryHUK · January 22, 2021, 1:09am

Hi all,

(3.92-57 still) RockStor hasn’t started

boot.log said to run

[root@rocky ~]# systemctl status rockstor-hdparm.service
● rockstor-hdparm.service - Rockstor hdparm settings
   Loaded: loaded (/etc/systemd/system/rockstor-hdparm.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2021-01-22 00:39:27 GMT; 12min ago
 Main PID: 2973 (code=exited, status=2)

Jan 22 00:39:27 rocky systemd[1]: Starting Rockstor hdparm settings...
Jan 22 00:39:27 rocky hdparm[2973]: /dev/disk/by-id/ata-ST4000VN000-1H4168_Z302CFSV: No such file or directory
Jan 22 00:39:27 rocky systemd[1]: rockstor-hdparm.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 22 00:39:27 rocky systemd[1]: Failed to start Rockstor hdparm settings.
Jan 22 00:39:27 rocky systemd[1]: Unit rockstor-hdparm.service entered failed state.
Jan 22 00:39:27 rocky systemd[1]: rockstor-hdparm.service failed.

Everything was ok yesterday.
I think the drive in the error is one I swapped out before Christmas.
btrfs fi show seems ok

Any help much appreciated.

Matthew.

phillxnet · January 22, 2021, 10:12am

@HarryHUK Hello again.

You can safely just delete that hdparm service file, if that is the trouble all you will lose is your drive spin down settings which you can just re-enter for the remaining drives. We definitely have a clean up issue to address here.

Let us know if deleting that rockstor hdparm systemd service file does the trick.

Hope that helps.

HarryHUK · January 30, 2021, 6:03pm

Hi,

thanks for replying. It’s been taking me a while to get back up to speed since Christmas.

I deleted rockstor-hdparm.service and the rockstor service seems to start, but now nginx is failing on startup.

systemctl status -l rockstor.service

    Jan 30 17:13:26 rocky supervisord[3518]: 2021-01-30 17:13:26,002 INFO spawned: 'nginx' with pid 3687
    Jan 30 17:13:26 rocky supervisord[3518]: 2021-01-30 17:13:26,010 INFO exited: nginx (exit status 1; not expected)
    Jan 30 17:13:28 rocky supervisord[3518]: 2021-01-30 17:13:28,013 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
    Jan 30 17:13:29 rocky supervisord[3518]: 2021-01-30 17:13:29,016 INFO spawned: 'nginx' with pid 3688
    Jan 30 17:13:29 rocky supervisord[3518]: 2021-01-30 17:13:29,024 INFO exited: nginx (exit status 1; not expected)
    Jan 30 17:13:30 rocky supervisord[3518]: 2021-01-30 17:13:30,025 INFO gave up: nginx entered FATAL state, too many start retries too quickly

I look at the nginx error.log:

2021/01/30 17:13:22 [emerg] 3535#0: no "events" section in configuration
2021/01/30 17:13:23 [emerg] 3593#0: no "events" section in configuration
2021/01/30 17:13:26 [emerg] 3687#0: no "events" section in configuration
2021/01/30 17:13:29 [emerg] 3688#0: no "events" section in configuration

If the nginx config is in /etc/nginx/nginx.conf

...
events {
    worker_connections 1024;
}
...

So it looks ok. Is nginx being launched with a config located somewhere else?

There is a link to the hdparms.service file in /etc/systemd/system/basic.target.wants and suspends.target.wants, should I remove those too?

Many thanks again.

phillxnet · January 30, 2021, 6:11pm

@HarryHUK Re:

This is a known issue that we have yet to get to the bottom of, cause wise:

Yes.

Take a look at the following issue for a work around to re-create your nginx config:

github.com/rockstor/rockstor-core

nginx.conf empty - no "events" section in configuration

opened 10:13AM - 25 Sep 17 UTC

phillxnet

Thanks to forum members grizzly and petermc for highlighting this issue. On rare… occasions the /opt/rockstor/etc/nginx/nginx.conf file is empty. This leads to a Web-UI failure and the following error messages in /var/log/nginx/errors.log: ``` 2016/05/04 07:41:12 [emerg] 14606#0: no "events" section in configuration 2016/05/04 07:41:13 [emerg] 14636#0: no "events" section in configuration 2016/05/04 07:41:16 [emerg] 14794#0: no "events" section in configuration 2016/05/04 07:41:19 [emerg] 14861#0: no "events" section in configuration ``` The issue author has also seen this occurrence but it's cause has yet to be established: ``` ls -la /opt/rockstor/etc/nginx/nginx.conf -rw------- 1 root root 0 Nov 5 11:23 /opt/rockstor/etc/nginx/nginx.conf ``` Referenced forum threads (please update on this issues resolution): grizzly: https://forum.rockstor.com/t/solved-rockstor-webui-no-longer-starts-3-8-13/1444 petermc: https://forum.rockstor.com/t/opt-rockstor-etc-nginx-nginx-conf-empty/3812

Re:

Yes, you can delete those. They will irritate systemd but not adversely .

Hope that helps.

HarryHUK · January 30, 2021, 6:14pm

Cheers!
I had just found the empty rockstor nginx.conf, will look at the link

HarryHUK · January 30, 2021, 6:23pm

Done, restarting.

I take it the variables are not shell variables but are meaningful in the nginx config context?

Hurrah, web UI back up and running, will check through the rest a little later.

Many thanks!

phillxnet · January 30, 2021, 6:40pm

@HarryHUK Thanks for the update, and glad your back up and running.

And do add to that issue if you manage to track down why this happened in your case. It’s very strange. I suspect we have a race condition of some sort and with very specific timing we end up not updating that file and leaving it blank! Not ideal.

Cheers.

HarryHUK · January 30, 2021, 9:44pm

@phillxnet I can’t remember exactly when it broke, but we had a powercut overnight on 15th.

(I haven’t yet sorted out NUTS…)