User report of share and GUI failure

I’ve recently had a report, via support email, of Web-UI and share failure. Bringing this to the forum so that others may chip in. I’m pasting what I think are the most pertinent parts of the submitted logs downloaded from the Web-UI by the reporter. Currently that is the only information I have. I am going to invite the reporter to this thread so we might use the better-than-email forum facilities to have a go at getting them sorted.

All welcome to assist and I’m currently unsure of users technical level; what we have is:

I lost access to my shared folder and GUI, please help me.

I have requested, via email return, the output of the following commands:

btrfs fi show

and

btrfs fi usage /mnt2/rockstor_rockstor

Pasting the pertinent log entries as I see them:

2019-11-28 13:47:53 [10083] [INFO] Listening at: http://127.0.0.1:8000 (10083)
2019-11-28 13:47:53 [10083] [INFO] Using worker: sync
2019-11-28 13:47:53 [10185] [INFO] Booting worker with pid: 10185
2019-11-28 13:47:53 [10184] [INFO] Booting worker with pid: 10184
2019-12-31 18:45:24 [10052] [INFO] Starting gunicorn 0.16.1
2019-12-31 18:45:24 [10052] [INFO] Listening at: http://127.0.0.1:8000 (10052)
2019-12-31 18:45:24 [10052] [INFO] Using worker: sync
2019-12-31 18:45:24 [10156] [INFO] Booting worker with pid: 10156
2019-12-31 18:45:24 [10155] [INFO] Booting worker with pid: 10155
2020-01-20 08:59:58 [10052] [INFO] Handling signal: term
2020-01-20 08:59:58 [10155] [INFO] Worker exiting (pid: 10155)
2020-01-20 08:59:58 [10156] [INFO] Worker exiting (pid: 10156)
2020-01-20 08:59:58 [10052] [INFO] Shutting down: Master
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 875, in emit
    self.flush()
  File "/usr/lib64/python2.7/logging/__init__.py", line 835, in flush
    self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file glogging.py, line 184
2020-01-20 09:01:33 [10082] [INFO] Using worker: sync
2020-01-20 09:01:33 [10106] [INFO] Booting worker with pid: 10106
2020-01-20 09:01:33 [10105] [INFO] Booting worker with pid: 10105

The above gunicorn.log ends as it does above.

So initially this looks like the system disk is out of space, has gone read only, or is failing; and we also have the following db access happening at the same time:

rockstor.log

DoesNotExist: NetworkConnection matching query does not exist.
[20/Jan/2020 08:56:55] ERROR [storageadmin.views.network:157] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 154, in update_connection
    name=dconfig['connection'])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /o/token/ method: POST
[20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:33] FATAL:  the database system is in recovery mode
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper
    return bound_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/opt/rockstor/eggs/django_braces-1.11.0-py2.7.egg/braces/views/_forms.py", line 24, in dispatch
    return super(CsrfExemptMixin, self).dispatch(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper
    return bound_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
    return view(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/base.py", line 172, in post
    url, headers, body, status = self.create_token_response(request)
  File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/mixins.py", line 124, in create_token_response
    return core.create_token_response(request)
  File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_backends.py", line 126, in create_token_response
    headers)
  File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/base.py", line 64, in wrapper
    return f(endpoint, uri, *args, **kwargs)
  File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/token.py", line 100, in create_token_response
    request, self.default_token_type)
  File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/grant_types/client_credentials.py", line 80, in create_token_response
    token = token_handler.create_token(request, refresh_token=False)
  File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/tokens.py", line 276, in create_token
    self.request_validator.save_bearer_token(token, request)
  File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_validators.py", line 310, in save_bearer_token
    access_token.save()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/transaction.py", line 271, in __exit__
    connection.set_autocommit(True)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 294, in set_autocommit
    self.ensure_connection()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: FATAL:  the database system is in recovery mode

and a db with no space left may well announce itself to be “… in recovery mode” hence failing Web-UI.

There after we have a repeating:

[20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST
[20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):
...
[20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST
[20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):

Further from the supervisord.log we have at the same time:

2019-11-28 13:47:51,766 INFO supervisord started with pid 9700
2019-11-28 13:47:52,769 INFO spawned: 'nginx' with pid 10082
2019-11-28 13:47:52,771 INFO spawned: 'gunicorn' with pid 10083
2019-11-28 13:47:52,773 INFO spawned: 'data-collector' with pid 10084
2019-11-28 13:47:52,775 INFO spawned: 'ztask-daemon' with pid 10085
2019-11-28 13:47:54,804 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-11-28 13:47:54,804 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-11-28 13:47:57,807 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-11-28 13:47:57,808 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-12-31 18:45:22,556 CRIT Supervisor running as root (no user in config file)
2019-12-31 18:45:22,574 INFO RPC interface 'supervisor' initialized
2019-12-31 18:45:22,576 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-12-31 18:45:22,577 INFO supervisord started with pid 9669
2019-12-31 18:45:23,579 INFO spawned: 'nginx' with pid 10051
2019-12-31 18:45:23,582 INFO spawned: 'gunicorn' with pid 10052
2019-12-31 18:45:23,585 INFO spawned: 'data-collector' with pid 10053
2019-12-31 18:45:23,589 INFO spawned: 'ztask-daemon' with pid 10054
2019-12-31 18:45:25,615 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-12-31 18:45:25,615 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-12-31 18:45:28,619 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-12-31 18:45:28,620 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2020-01-20 08:59:58,123 WARN received SIGTERM indicating exit request
2020-01-20 08:59:58,123 INFO waiting for data-collector, nginx, gunicorn, ztask-daemon to die
2020-01-20 08:59:58,131 INFO exited: data-collector (terminated by SIGTERM; not expected)
2020-01-20 08:59:58,133 INFO exited: ztask-daemon (terminated by SIGTERM; not expected)
2020-01-20 08:59:58,140 INFO exited: nginx (exit status 0; expected)
2020-01-20 08:59:58,244 INFO exited: gunicorn (exit status 0; expected)
2020-01-20 09:01:31,233 CRIT Supervisor running as root (no user in config file)
2020-01-20 09:01:31,259 INFO RPC interface 'supervisor' initialized
2020-01-20 09:01:31,264 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-01-20 09:01:31,265 INFO supervisord started with pid 9294
2020-01-20 09:01:32,269 INFO spawned: 'nginx' with pid 10081
2020-01-20 09:01:32,273 INFO spawned: 'gunicorn' with pid 10082
2020-01-20 09:01:32,275 INFO spawned: 'data-collector' with pid 10083
2020-01-20 09:01:32,278 INFO spawned: 'ztask-daemon' with pid 10084
2020-01-20 09:01:37,364 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2020-01-20 09:01:37,364 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

So again failure of the supporting services to start with the exited entries above.

I may not have much time to address this issue myself so if anyone can chip in that would be great.

Cheers.

1 Like