[Solved] Power loss - reboot successful, but no WebUI access

After a rare power outage, Rockstor rebooted, and seems to function fine as a NAS as well as the RockOns working, etc. However, the WebUI access is suddenly not working anymore, on various browser variants I get the SSL Errors regarding the protocol or, in Mozilla’s case, the RECORD_TOO_LONG message.
As before, I have been using the https:// version of the IP address.

Chrome, New Edge:
image
Firefox:
image
I investigated whether I suddenly suffered from the elusive empty nginx.conf file issue


but my configuration file seems to be populated just fine
In November 2020, I had already checked whether I possibly had an issue with discrepancy between an nginx update and the dependency of openssl11-libs as mentioned here:

But I got both of versions in sync, and they still are.

However, if I check the nginx status right after a reboot, it shows this:
systemctl status nginx

> ● nginx.service - The nginx HTTP and reverse proxy server
>    Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
>    Active: inactive (dead)

And overall system status:
systemctl status -l rockstor-pre rockstor-bootstrap Rockstor

● rockstor-pre.service - Tasks required prior to starting Rockstor

Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)

Active: active (exited) since Fri 2021-01-29 09:43:59 PST; 1h 2min ago

 Main PID: 3923 (code=exited, status=0/SUCCESS)

CGroup: /system.slice/rockstor-pre.service

Jan 29 09:43:58 rockstorw initrock[3923]: 2021-01-29 09:43:58,908: Done

Jan 29 09:43:58 rockstorw initrock[3923]: 2021-01-29 09:43:58,908: Running prepdb...

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,287: Done

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,287: stopping firewalld...

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,332: firewalld stopped and disabled

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,463: Normalising on shellinaboxd service file

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,463: - shellinaboxd.service already exists

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,463: rockstor service looks correct. Not updating.

Jan 29 09:43:59 rockstorw initrock[3923]: 2021-01-29 09:43:59,463: rockstor-bootstrap.service looks correct. Not updating.

Jan 29 09:43:59 rockstorw systemd[1]: Started Tasks required prior to starting Rockstor.

● rockstor-bootstrap.service - Rockstor bootstrapping tasks

Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)

Active: active (exited) since Fri 2021-01-29 09:44:33 PST; 1h 2min ago

Process: 9196 ExecStart=/opt/rockstor/bin/bootstrap (code=exited, status=0/SUCCESS)

 Main PID: 9196 (code=exited, status=0/SUCCESS)

CGroup: /system.slice/rockstor-bootstrap.service

Jan 29 09:44:00 rockstorw bootstrap[9196]: If you have done those things and are still encountering

Jan 29 09:44:00 rockstorw bootstrap[9196]: this message, please follow up at

Jan 29 09:44:00 rockstorw bootstrap[9196]: https://bit.ly/setuptools-py2-warning.

Jan 29 09:44:00 rockstorw bootstrap[9196]: ************************************************************

Jan 29 09:44:00 rockstorw bootstrap[9196]: sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)

Jan 29 09:44:33 rockstorw bootstrap[9196]: BTRFS device scan complete

Jan 29 09:44:33 rockstorw bootstrap[9196]: Bootstrapping complete

Jan 29 09:44:33 rockstorw bootstrap[9196]: Running qgroup cleanup. /opt/rockstor/bin/qgroup-clean

Jan 29 09:44:33 rockstorw bootstrap[9196]: Running qgroup limit maxout. /opt/rockstor/bin/qgroup-maxout-limit

Jan 29 09:44:33 rockstorw systemd[1]: Started Rockstor bootstrapping tasks.
● rockstor.service - RockStor startup script
   Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-01-29 09:43:59 PST; 1h 2min ago
 Main PID: 9195 (supervisord)
   CGroup: /system.slice/rockstor.service
           ├─9195 /usr/bin/python2 /opt/rockstor/bin/supervisord -c /opt/rockstor/etc/supervisord.conf
           ├─9212 nginx: master process /usr/sbin/nginx -c /opt/rockstor/etc/nginx/nginx.conf
           ├─9213 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
           ├─9214 /usr/bin/python2 /opt/rockstor/bin/data-collector
           ├─9215 /usr/bin/python2 /opt/rockstor/bin/django ztaskd --noreload --replayfailed -f /opt/rockstor/var/log/ztask.log
           ├─9216 nginx: worker process
           ├─9217 nginx: worker process
           ├─9238 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
           └─9243 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application

Jan 29 09:44:00 rockstorw supervisord[9195]: 2021-01-29 09:44:00,187 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Jan 29 09:44:00 rockstorw supervisord[9195]: 2021-01-29 09:44:00,187 INFO supervisord started with pid 9195
Jan 29 09:44:01 rockstorw supervisord[9195]: 2021-01-29 09:44:01,189 INFO spawned: 'nginx' with pid 9212
Jan 29 09:44:01 rockstorw supervisord[9195]: 2021-01-29 09:44:01,190 INFO spawned: 'gunicorn' with pid 9213
Jan 29 09:44:01 rockstorw supervisord[9195]: 2021-01-29 09:44:01,191 INFO spawned: 'data-collector' with pid 9214
Jan 29 09:44:01 rockstorw supervisord[9195]: 2021-01-29 09:44:01,192 INFO spawned: 'ztask-daemon' with pid 9215
Jan 29 09:44:03 rockstorw supervisord[9195]: 2021-01-29 09:44:03,592 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 29 09:44:03 rockstorw supervisord[9195]: 2021-01-29 09:44:03,592 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
Jan 29 09:44:06 rockstorw supervisord[9195]: 2021-01-29 09:44:06,596 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
Jan 29 09:44:06 rockstorw supervisord[9195]: 2021-01-29 09:44:06,597 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

My nginx.conf file looks like this (/opt/rockstor/etc/nginx/nginx.conf) and appears normal to me:

daemon off;
worker_processes  2;

events {
        worker_connections  1024;
        use epoll;
}

http {
        include         /opt/rockstor/etc/nginx/mime.types;
        default_type    application/octet-stream;

    log_format main
            '$remote_addr - $remote_user [$time_local] '
            '"$request" $status $bytes_sent '
            '"$http_referer" "$http_user_agent" '
            '"$gzip_ratio"';

    client_header_timeout   10m;
    client_body_timeout             10m;
    send_timeout                    10m;

    connection_pool_size            256;
    client_header_buffer_size       1k;
    large_client_header_buffers     4 8k;
    request_pool_size                       4k;

    gzip on;
    gzip_min_length 1100;
    gzip_buffers    4 8k;
    gzip_types      text/plain;

    output_buffers  1 32k;
    postpone_output 1460;

    sendfile        on;
    tcp_nopush      on;
    tcp_nodelay     on;

    keepalive_timeout       75 20;
    ignore_invalid_headers  on;
    index index.html;

    server {
            listen 443 default_server;
            server_name "~^(?<myhost>.+)$";
            #ssl on; deprecated
            ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
            ssl_certificate /opt/rockstor/certs/rockstor.cert;
            ssl_certificate_key /opt/rockstor/certs/rockstor.key;

            location /site_media  {
                    root /media/; # Notice this is the /media folder that we create above
            }
            location ~* ^.+\.(zip|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|mov) {
                    access_log   off;
                    expires      30d;
            }
            location /static  {
                    root /opt/rockstor/;
            }
            location /logs {
                    root /opt/rockstor/src/rockstor/;
            }
            location / {
                    proxy_pass_header Server;
                    proxy_set_header Host $http_host;
                    proxy_set_header X-Forwarded-Proto https;
                    proxy_redirect off;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_set_header X-Scheme $scheme;
                    proxy_connect_timeout 75;
                    proxy_read_timeout 120;
                    proxy_pass http://127.0.0.1:8000/;
            }
            location /socket.io {
                    proxy_pass http://127.0.0.1:8001;
                    proxy_redirect off;
                    proxy_http_version 1.1;
                    proxy_set_header Upgrade $http_upgrade;
                    proxy_set_header Connection "upgrade";
            }
            location /shell/ {
                    valid_referers server_names;
                    if ($invalid_referer) { return 404; }
                    proxy_pass http://127.0.0.1:4200;
                    proxy_redirect off;
                    proxy_http_version 1.1;
                    proxy_set_header Upgrade $http_upgrade;
                    proxy_set_header Connection "upgrade";
            }
    }
 }

In the logs (/opt/rockstor/var/log/rockstor.log) I don’t seem to find anything suspicious either. They’re all about the “can’t list qgroups: quotas are not enabled” message.

Any suggestions what else I should look at? The good part is that I can still access the shares and their data, as well as the RockOns are operating normally…

2 Likes

Hi there, @Hooverdan,

Sorry to see you’re having issue… that’s a rather odd one.

Nginx used by Rockstor is actually controlled by supervisord and not systemd, hence your systemd output; I actually have the same (on a Rockstor system running fine):

ockdev:~ # systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

What you would do to check its status would thus be:

rockdev:~ # /opt/rockstor/bin/supervisorctl status nginx
nginx                            RUNNING    pid 2574, uptime 0:48:58

Accordingly, the nginx logs are to be found at:

/opt/build/var/log/supervisord_nginx_std*

Hopefully this helps get some more clues…

1 Like

@Flox,
that would explain a portion of the message I saw. So, if I do the same as you posted:
/opt/rockstor/bin/supervisorctl status nginx
nginx RUNNING pid 9212, uptime 3:49:45

so that’s the same as what you see.
I checked the other logs, but they don’t provide me any clues. All I see are multiple instance of the messages below:

For /opt/rockstor/var/log/supervisord_nginx_stdout.log:
Package upgrade
For /opt/rockstor/var/log/supervisord_nginx_stderr.log:
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /opt/rockstor/etc/nginx/nginx.conf:47

My last reboot, prior to these fixing attempts with multiple reboots was:
reboot system boot 5.8.7-1.el7.elre Mon Oct 12 12:45 - 13:00 (108+01:15)
And since October all of these packages were applied
12/18/2020

Packages Altered:
Updated bind-export-libs-32:9.11.4-26.P2.el7_9.2.x86_64 @updates
Update 32:9.11.4-26.P2.el7_9.3.x86_64 @updates
Updated bind-libs-lite-32:9.11.4-26.P2.el7_9.2.x86_64 @updates
Update 32:9.11.4-26.P2.el7_9.3.x86_64 @updates
Updated bind-license-32:9.11.4-26.P2.el7_9.2.noarch @updates
Update 32:9.11.4-26.P2.el7_9.3.noarch @updates
Updated device-mapper-7:1.02.170-6.el7.x86_64 @base
Update 7:1.02.170-6.el7_9.3.x86_64 @updates
Updated device-mapper-libs-7:1.02.170-6.el7.x86_64 @base
Update 7:1.02.170-6.el7_9.3.x86_64 @updates
Updated gd-2.0.35-26.el7.x86_64 @anaconda/3
Update 2.0.35-27.el7_9.x86_64 @updates
Updated libsmbclient-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated libwbclient-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated microcode_ctl-2:2.1-73.2.el7_9.x86_64 @updates
Update 2:2.1-73.4.el7_9.x86_64 @updates
Updated openssl-1:1.0.2k-19.el7.x86_64 @base
Update 1:1.0.2k-21.el7_9.x86_64 @updates
Updated openssl-libs-1:1.0.2k-19.el7.x86_64 @base
Update 1:1.0.2k-21.el7_9.x86_64 @updates
Updated realmd-0.16.1-12.el7.x86_64 @base
Update 0.16.1-12.el7_9.1.x86_64 @updates
Updated samba-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-client-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-client-libs-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-common-4.10.16-7.el7_9.noarch @updates
Update 4.10.16-9.el7_9.noarch @updates
Updated samba-common-libs-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-common-tools-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-libs-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-winbind-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-winbind-clients-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-winbind-krb5-locator-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated samba-winbind-modules-4.10.16-7.el7_9.x86_64 @updates
Update 4.10.16-9.el7_9.x86_64 @updates
Updated vim-filesystem-2:7.4.629-7.el7.x86_64 @base
Update 2:7.4.629-8.el7_9.x86_64 @updates
Updated vim-minimal-2:7.4.629-7.el7.x86_64 @base
Update 2:7.4.629-8.el7_9.x86_64 @updates

1/1/2021

Packages Altered:
Updated openssl11-libs-1:1.1.1g-1.el7.x86_64 @epel
Update 1:1.1.1g-2.el7.x86_64 @epel

1/3/2021

Packages Altered:
Updated libzstd-1.4.5-3.el7.x86_64 @epel
Update 1.4.7-1.el7.x86_64 @epel
Updated libzstd-devel-1.4.5-3.el7.x86_64 @epel
Update 1.4.7-1.el7.x86_64 @epel

1/26/2021

Packages Altered:
Updated dnsmasq-2.76-16.el7.x86_64 @base
Update 2.76-16.el7_9.1.x86_64 @updates
Updated net-snmp-1:5.7.2-49.el7.x86_64 @base
Update 1:5.7.2-49.el7_9.1.x86_64 @updates
Updated net-snmp-agent-libs-1:5.7.2-49.el7.x86_64 @base
Update 1:5.7.2-49.el7_9.1.x86_64 @updates
Updated net-snmp-libs-1:5.7.2-49.el7.x86_64 @base
Update 1:5.7.2-49.el7_9.1.x86_64 @updates
Updated tzdata-2020d-2.el7.noarch @updates
Update 2020f-1.el7.noarch @updates

1/27/2021

Packages Altered:
Updated sudo-1.8.23-10.el7.x86_64 @base
Update 1.8.23-10.el7_9.1.x86_64 @updates
Updated tzdata-2020f-1.el7.noarch @updates
Update 2021a-1.el7.noarch @updates

reboot system boot 5.8.7-1.el7.elre Thu Jan 28 12:13 - 13:00 (00:47) <-- that was after the power outage

1 Like

Mmmm… How about the other supervisord programs? Would you check the gunicorn one? You can find all the names of the programs from the names of the logs im the same folder. Sorry i can’t give more details as I’m on my phone at the moment and thus limited :-\

2 Likes

@Flox,
Nothing standing out at me … but who knows:

supervisord_gunicorn_stderr.log

Many of these:

/opt/rockstor/eggs/setuptools-46.1.3-py2.7.egg/pkg_resources/py2_warn.py:21: UserWarning: Setuptools will stop working on Python 2


You are running Setuptools on Python 2, which is no longer
supported and

SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).
Please ensure you are installing
Setuptools using pip 9.x or later or pin to setuptools<45
in your environment.
If you have done those things and are still encountering
this message, please follow up at
https://bit.ly/setuptools-py2-warning.

some of these:

  sys.version_info < (3,) and warnings.warn(pre + "*" * 60 + msg + "*" * 60)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/handlers.py", line 76, in emit
    if self.shouldRollover(record):
  File "/usr/lib64/python2.7/logging/handlers.py", line 154, in shouldRollover
    msg = "%s\n" % self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 324, in getMessage
    msg = str(self.msg)
TypeError: __str__ returned non-string (type list)
Logged from file rockon.py, line 114

supervisord_data-collector_stderr.log
some of these, but I can’t tell how old they are:

disconnect handler error
Traceback (most recent call last):
  File "/opt/rockstor/eggs/python_engineio-2.3.2-py2.7.egg/engineio/server.py", line 423, in _trigger_event
    return self.handlers[event](*args)
  File "/opt/rockstor/eggs/python_socketio-1.6.0-py2.7.egg/socketio/server.py", line 504, in _handle_eio_disconnect
    self._handle_disconnect(sid, '/')
  File "/opt/rockstor/eggs/python_socketio-1.6.0-py2.7.egg/socketio/server.py", line 413, in _handle_disconnect
    self._trigger_event('disconnect', n, sid)
  File "/opt/rockstor/eggs/python_socketio-1.6.0-py2.7.egg/socketio/server.py", line 467, in _trigger_event
    event, *args)
  File "/opt/rockstor/eggs/python_socketio-1.6.0-py2.7.egg/socketio/namespace.py", line 30, in trigger_event
    return getattr(self, handler_name)(*args)
  File "/opt/rockstor/src/rockstor/smart_manager/data_collector.py", line 870, in on_disconnect
    self.cleanup(sid)
  File "/opt/rockstor/src/rockstor/smart_manager/data_collector.py", line 70, in cleanup
    gevent.killall(self.threads[sid])
  File "/opt/rockstor/eggs/gevent-1.1.2-py2.7-linux-x86_64.egg/gevent/greenlet.py", line 699, in killall
    alive = waiter.get()
  File "/opt/rockstor/eggs/gevent-1.1.2-py2.7-linux-x86_64.egg/gevent/hub.py", line 878, in get
    return self.hub.switch()
  File "/opt/rockstor/eggs/gevent-1.1.2-py2.7-linux-x86_64.egg/gevent/hub.py", line 609, in switch
    return greenlet.switch(self)
GreenletExit

supervisord_ztask-daemon_stderr.log
error message about a zombie ztask for deleting a device that I had removed over a year ago (had a separate thread going on that when replacing all my drives).

the *stdout.log files essentially contained the Package update messages.

1 Like

How about gunicorn status? I suppose it is running fine but we should make sure.

1 Like

@Flox, I think it’s running:
ps aux | grep gunicorn

ps aux | grep gunicorn
root      9213  0.0  0.0 217428 20668 ?        S    09:44   0:05 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
root      9238  0.0  0.1 408712 43156 ?        Sl   09:44   0:00 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
root      9243  0.0  0.1 433056 57608 ?        Sl   09:44   0:03 /usr/bin/python2 /opt/rockstor/bin/gunicorn --bind=127.0.0.1:8000 --pid=/run/gunicorn.pid --workers=2 --log-file=/opt/rockstor/var/log/gunicorn.log --pythonpath=/opt/rockstor/src/rockstor --timeout=120 --graceful-timeout=120 wsgi:application
root     12987  0.0  0.0 112832  2312 pts/0    S+   17:26   0:00 grep --color=auto gunicorn

And in the gunicorn.log I see this:

[2021-01-29 09:42:52 +0000] [9242] [INFO] Worker exiting (pid: 9242)
[2021-01-29 09:42:52 +0000] [9217] [INFO] Handling signal: term
[2021-01-29 09:42:52 +0000] [9247] [INFO] Worker exiting (pid: 9247)
[2021-01-29 09:42:52 +0000] [9217] [INFO] Shutting down: Master
[2021-01-29 09:44:01 +0000] [9213] [INFO] Starting gunicorn 19.7.1
[2021-01-29 09:44:01 +0000] [9213] [INFO] Listening at: http://127.0.0.1:8000 (9213)
[2021-01-29 09:44:01 +0000] [9213] [INFO] Using worker: sync
[2021-01-29 09:44:01 +0000] [9238] [INFO] Booting worker with pid: 9238
[2021-01-29 09:44:01 +0000] [9243] [INFO] Booting worker with pid: 9243

OK, it seems like all services are running like they should… you could verify all of that as follows:

  1. in first terminal window, stop the rockstor service: systemctl stop rockstor
  2. in a second terminal window, watch the supervisord logs: tail -f /opt/rockstor/var/log/supervisord*
  3. in the first terminal window, start the rockstor service to verify it is starting up as it should: systemctl start rockstor
    On my system, here’s what that gives:
==> /opt/build/var/log/supervisord.log <==
2021-01-30 11:12:04,011 CRIT Supervisor running as root (no user in config file)
2021-01-30 11:12:04,021 INFO RPC interface 'supervisor' initialized
2021-01-30 11:12:04,021 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-01-30 11:12:04,021 INFO supervisord started with pid 3920
2021-01-30 11:12:05,025 INFO spawned: 'nginx' with pid 3926
2021-01-30 11:12:05,030 INFO spawned: 'gunicorn' with pid 3927
2021-01-30 11:12:05,033 INFO spawned: 'data-collector' with pid 3928
2021-01-30 11:12:05,035 INFO spawned: 'ztask-daemon' with pid 3929

==> /opt/build/var/log/supervisord_nginx_stderr.log <==
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /opt/build/etc/nginx/nginx.conf:47

==> /opt/build/var/log/supervisord.log <==
2021-01-30 11:12:07,046 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2021-01-30 11:12:07,047 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2021-01-30 11:12:10,052 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2021-01-30 11:12:10,052 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

I don’t expect yours to look different based on what you reported, but better be safe and make sure of it.

Given all your services seem to work as intended, it might be worth digging deeper into the SSL-related errors displayed by your browsers, then. Also, I’ve seen a few times issues of the sort due to some caching problems so it might be worth trying with a new browser/clearing cache/private window and see what happens; it’s always better to rule something that simple out (if you haven’t already done so).

You might also verify your certificates are where they should:

/opt/rockstor/certs/rockstor.*

These SSL-errors unfortunately are not very helpful as they seem to have many different causes possible so I’m not sure what it could be in your case.

2 Likes

@Flox, thanks for the additional suggestions. I did the two terminal thing and I am getting the same messages that you have listed. I also have the certificates in the folder:

-rw-r--r-- 1 root root 1326 Nov 14  2016 rockstor.cert
-rw-r--r-- 1 root root 1066 Nov 14  2016 rockstor.csr
-rw-r--r-- 1 root root 1679 Nov 14  2016 rockstor.key

In general, I would be surprised if some of these long standing items were deleted or changed due to the sudden loss of power. When I had a failing power supply a few years ago it would suddenly restart many times and that never caused any problems …

Yes, that was my first thought, actually. So cache/history clearing, incognito windows, different machines, Windows vs. Linux, browsers, the works :). All to no avail unfortunately.

I continue to be stumped. Would it make sense to force reinstallation of the nginx and/or SSL packages that were updated over the last few months? I assume, glancing over my nginx.conf file nothing stood out for you as being strangely different than it should be?

1 Like

Absolutely… I don’t see a risk in reinstalling them.

Indeed I checked and it looks exactly as it should.

1 Like

@Flox,
alright. I did the reinstallation of nginx and openssl. restarted the nginx service. Rebooted the entire box, actually. No difference.

Getting desperate I tried one more thing … just one more thing…

I went into the nginx.conf file, and looked at these lines that I have in there (line ~43 or so):

...
server {
        listen 443 default_server;
        server_name "~^(?<myhost>.+)$";
        #ssl on; deprecated
        ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
        ssl_certificate /opt/rockstor/certs/rockstor.cert;
        ssl_certificate_key /opt/rockstor/certs/rockstor.key;
...

I figured, since I keep getting that deprecated warning (see above), I might as well put in the “new” directive. So I changed the line to:

listen 443 ssl default_server;

executed the stop and restart of nginx:
/opt/rockstor/bin/supervisorctl stop nginx
/opt/rockstor/bin/supervisorctl start nginx

and … Voilà!!! I got UI access again.

Why this seems to have worked is beyond me, as I have not touched the conf file (I think ever). And I can’t imagine that a power outage would “change” this, either.

image

Thanks for following my lengthy postings through all of this. But I think for now I can call this solved.

3 Likes

Strange that conf got touched/altered. Good spot getting that fixed though :+1:

1 Like

@Flox, here is an addendum to my saga. I decided to reboot the server once again.
Interestingly enough, I ran into the same exact issue, no WebUI. I went into terminal, and it looks like my nginx.conf file was altered again during the reboot.
My addition of ‘ssl’ was removed and the commented out line for ssl with the comment deprecated was still there:

    server {
            listen 443 default_server;
            server_name "~^(?<myhost>.+)$";
            #ssl on; deprecated
            ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
            ssl_certificate /opt/rockstor/certs/rockstor.cert;
            ssl_certificate_key /opt/rockstor/certs/rockstor.key;

I suspect, nginx itself updates the ssl on and adds the deprecated comment to it? And then, the ssl switch missing in the listen line causes the WebUI being unreachable, unless manually re-added.

Does the /opt/rockstor/etc/nginx/nginx.conf get “rebuilt” upon boot-up?

And if so, where from? I assume, the files in the /opt/rockstor/conf directory like nginx.conf, nginx.conf.in or nginx-prod.conf.in have nothing to do with that? I actually made changes to each of them and rebooted, but that had no impact, so I am pretty certain they don’t.

Alternatively, maybe @phillxnet can shed some light on that?

That makes sense, yes, even though I’m still puzzled this is what is causing issues on your system. But let’s answer your question first:

I have not looked at that in details or tested it, but I would say yes… see below:

They actually do, I believe. First, let’s look at a previous post by @phillxnet describing Rockstor “boot” procedure:

If we look at the first systemd service of interest, we have rockstor-pre, which simply triggers the initrock script. In this script, you will find the following:


… which triggers the following:

… which itself is a wrapper for the following:

As you can see in the latter, the file in /opt/rockstor/etc/nginx/nginx.conf is used as a template and updated with port:ip settings, for instance. Let’s look at one part of that in particular:

As you can see there, it searches for the listen 443 default_server line and make sure the port (and ip if specified) fits what it should be by replacing the line with listen (ip:)port default_server. I believe this is where your manual addition of ssl gets overwritten at every boot.
You could thus try to manually alter this (note the two lines with the inclusion of ssl:

                substr = "listen {} ssl default_server".format(port)
                if ip is not None:
                    substr = "listen {}:{} ssl default_server".format(ip, port)
                lines[i] = re.sub(r"listen.* default_server", substr, lines[i])

That being said, I’m really confused as to why you’re experiencing this, honestly… I cannot reproduce this problem on my systems here; they’re all Rockstor 4 (so built on openSUSE), but I’m unsure where it’s coming from.

Let us know how it goes, though, as we might need to implement this as permanent solution if that is the proper way to configure our Nginx conf. I’m not well versed in that myself, but I know we have quite a few experts in our community here so I will let these experts chime in.

Either way, thanks a lot for your finding and reporting back as it might very well help everybody in the end!

2 Likes

@Flox,
that did the trick! I added the “ssl” expression to the services.py file where you recommended. After a reboot, I was able to immediately access the WebUI.

I assume, for a fresh installation the template nginx.conf file is generated out of nginx.conf.in or nginx-prod.conf.in? If so, then the deprecated SSL directive (ssl on) should be removed from there at the same time as the SSL directive is added to the template file(s) as well as the services.py (where you pointed out) to make this a complete transition to the newer directive format that nginx would like to see?

I can’t explain why I seem to be the only one with that issue at this time, but I also don’t understand whether/how niginx could comment out automatically the deprecated ssl on directive - maybe that’s the new feature that came with the November update on the Centos base?

If you want me to, I can open an issue on Github with reference to this post, unless you still feel it’s a one-off problem. However, I think, even if not urgent, the deprecated ssl directive should probably be taken care of at some point, even if it’s not a high priority compared to other burning issues for the OpenSUSE transition.

Thanks again, as always for the detailed analysis!

2 Likes

@Hooverdan Hello again.
Re:

I think an initial GitHub issue would be a good idea. There’s definitely something going on here and it would be nice to have this in an issue ready for when we can tend to it.

Cheers.

2 Likes

@phillxnet, here is the github issue - if you want me to change the title to something else, I can do that:

2 Likes

@Hooverdan Thanks, that looks just find.

And if we find the title needs a little tweak I can do that as we go and GitHub records the change, and who made it, so it’s all visible what’s happened. Best we have this I think in case the same emerges in time on our newer platform. And as you note in the issue we should probably move to using the newer format anyway.

Cheers.

2 Likes