I’ve recently had a report, via support email, of Web-UI and share failure. Bringing this to the forum so that others may chip in. I’m pasting what I think are the most pertinent parts of the submitted logs downloaded from the Web-UI by the reporter. Currently that is the only information I have. I am going to invite the reporter to this thread so we might use the better-than-email forum facilities to have a go at getting them sorted.
All welcome to assist and I’m currently unsure of users technical level; what we have is:
I lost access to my shared folder and GUI, please help me.
I have requested, via email return, the output of the following commands:
btrfs fi show
and
btrfs fi usage /mnt2/rockstor_rockstor
Pasting the pertinent log entries as I see them:
2019-11-28 13:47:53 [10083] [INFO] Listening at: http://127.0.0.1:8000 (10083)
2019-11-28 13:47:53 [10083] [INFO] Using worker: sync
2019-11-28 13:47:53 [10185] [INFO] Booting worker with pid: 10185
2019-11-28 13:47:53 [10184] [INFO] Booting worker with pid: 10184
2019-12-31 18:45:24 [10052] [INFO] Starting gunicorn 0.16.1
2019-12-31 18:45:24 [10052] [INFO] Listening at: http://127.0.0.1:8000 (10052)
2019-12-31 18:45:24 [10052] [INFO] Using worker: sync
2019-12-31 18:45:24 [10156] [INFO] Booting worker with pid: 10156
2019-12-31 18:45:24 [10155] [INFO] Booting worker with pid: 10155
2020-01-20 08:59:58 [10052] [INFO] Handling signal: term
2020-01-20 08:59:58 [10155] [INFO] Worker exiting (pid: 10155)
2020-01-20 08:59:58 [10156] [INFO] Worker exiting (pid: 10156)
2020-01-20 08:59:58 [10052] [INFO] Shutting down: Master
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/__init__.py", line 875, in emit
self.flush()
File "/usr/lib64/python2.7/logging/__init__.py", line 835, in flush
self.stream.flush()
IOError: [Errno 28] No space left on device
Logged from file glogging.py, line 184
2020-01-20 09:01:33 [10082] [INFO] Using worker: sync
2020-01-20 09:01:33 [10106] [INFO] Booting worker with pid: 10106
2020-01-20 09:01:33 [10105] [INFO] Booting worker with pid: 10105
The above gunicorn.log ends as it does above.
So initially this looks like the system disk is out of space, has gone read only, or is failing; and we also have the following db access happening at the same time:
rockstor.log
DoesNotExist: NetworkConnection matching query does not exist.
[20/Jan/2020 08:56:55] ERROR [storageadmin.views.network:157] NetworkConnection matching query does not exist.
Traceback (most recent call last):
File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 154, in update_connection
name=dconfig['connection'])
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
[20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /o/token/ method: POST
[20/Jan/2020 09:01:34] ERROR [storageadmin.middleware:33] FATAL: the database system is in recovery mode
Traceback (most recent call last):
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper
return bound_func(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func
return func.__get__(self, type(self))(*args2, **kwargs2)
File "/opt/rockstor/eggs/django_braces-1.11.0-py2.7.egg/braces/views/_forms.py", line 24, in dispatch
return super(CsrfExemptMixin, self).dispatch(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 89, in dispatch
return handler(request, *args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 34, in _wrapper
return bound_func(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/debug.py", line 76, in sensitive_post_parameters_wrapper
return view(request, *args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 30, in bound_func
return func.__get__(self, type(self))(*args2, **kwargs2)
File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/base.py", line 172, in post
url, headers, body, status = self.create_token_response(request)
File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/views/mixins.py", line 124, in create_token_response
return core.create_token_response(request)
File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_backends.py", line 126, in create_token_response
headers)
File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/base.py", line 64, in wrapper
return f(endpoint, uri, *args, **kwargs)
File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/endpoints/token.py", line 100, in create_token_response
request, self.default_token_type)
File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/grant_types/client_credentials.py", line 80, in create_token_response
token = token_handler.create_token(request, refresh_token=False)
File "/opt/rockstor/eggs/oauthlib-1.0.1-py2.7.egg/oauthlib/oauth2/rfc6749/tokens.py", line 276, in create_token
self.request_validator.save_bearer_token(token, request)
File "/opt/rockstor/eggs/django_oauth_toolkit-0.9.0-py2.7.egg/oauth2_provider/oauth2_validators.py", line 310, in save_bearer_token
access_token.save()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 734, in save
force_update=force_update, update_fields=update_fields)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/base.py", line 762, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/transaction.py", line 271, in __exit__
connection.set_autocommit(True)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 294, in set_autocommit
self.ensure_connection()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: FATAL: the database system is in recovery mode
and a db with no space left may well announce itself to be “… in recovery mode” hence failing Web-UI.
There after we have a repeating:
[20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST
[20/Jan/2020 09:01:44] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):
...
[20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /login_submit method: POST
[20/Jan/2020 09:01:58] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):
Further from the supervisord.log we have at the same time:
2019-11-28 13:47:51,766 INFO supervisord started with pid 9700
2019-11-28 13:47:52,769 INFO spawned: 'nginx' with pid 10082
2019-11-28 13:47:52,771 INFO spawned: 'gunicorn' with pid 10083
2019-11-28 13:47:52,773 INFO spawned: 'data-collector' with pid 10084
2019-11-28 13:47:52,775 INFO spawned: 'ztask-daemon' with pid 10085
2019-11-28 13:47:54,804 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-11-28 13:47:54,804 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-11-28 13:47:57,807 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-11-28 13:47:57,808 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-12-31 18:45:22,556 CRIT Supervisor running as root (no user in config file)
2019-12-31 18:45:22,574 INFO RPC interface 'supervisor' initialized
2019-12-31 18:45:22,576 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-12-31 18:45:22,577 INFO supervisord started with pid 9669
2019-12-31 18:45:23,579 INFO spawned: 'nginx' with pid 10051
2019-12-31 18:45:23,582 INFO spawned: 'gunicorn' with pid 10052
2019-12-31 18:45:23,585 INFO spawned: 'data-collector' with pid 10053
2019-12-31 18:45:23,589 INFO spawned: 'ztask-daemon' with pid 10054
2019-12-31 18:45:25,615 INFO success: data-collector entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-12-31 18:45:25,615 INFO success: ztask-daemon entered RUNNING state, process has stayed up for > than 2 seconds (startsecs)
2019-12-31 18:45:28,619 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2019-12-31 18:45:28,620 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2020-01-20 08:59:58,123 WARN received SIGTERM indicating exit request
2020-01-20 08:59:58,123 INFO waiting for data-collector, nginx, gunicorn, ztask-daemon to die
2020-01-20 08:59:58,131 INFO exited: data-collector (terminated by SIGTERM; not expected)
2020-01-20 08:59:58,133 INFO exited: ztask-daemon (terminated by SIGTERM; not expected)
2020-01-20 08:59:58,140 INFO exited: nginx (exit status 0; expected)
2020-01-20 08:59:58,244 INFO exited: gunicorn (exit status 0; expected)
2020-01-20 09:01:31,233 CRIT Supervisor running as root (no user in config file)
2020-01-20 09:01:31,259 INFO RPC interface 'supervisor' initialized
2020-01-20 09:01:31,264 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-01-20 09:01:31,265 INFO supervisord started with pid 9294
2020-01-20 09:01:32,269 INFO spawned: 'nginx' with pid 10081
2020-01-20 09:01:32,273 INFO spawned: 'gunicorn' with pid 10082
2020-01-20 09:01:32,275 INFO spawned: 'data-collector' with pid 10083
2020-01-20 09:01:32,278 INFO spawned: 'ztask-daemon' with pid 10084
2020-01-20 09:01:37,364 INFO success: nginx entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2020-01-20 09:01:37,364 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
So again failure of the supporting services to start with the exited entries above.
I may not have much time to address this issue myself so if anyone can chip in that would be great.
Cheers.