Brief description of the problem
After a reboot, I can no longer access the web GUI. SSH works.
Detailed step by step instructions to reproduce the problem
When I attempt to access Rockstor via HTTPS, I get an error.
Web-UI screenshot
“Unable to connect” error message.
Error Traceback provided on the Web-UI
The system boots to a command line and I’m able to SSH in, but it won’t connect via web.
I’m booting from a USB drive (/dev/sda), which appears to be working. There seems to be some corruption on sdb, where the rock-on data is stored; a btrfs check /dev/sdb produces errors like the following:
parent transid verify failed on 12369920 wanted 2531442 found 2531450
Ignoring transid failure
parent transid verify failed on 12369920 wanted 2531442 found 2531450
Ignoring transid failure
Csum exists for 57777917952-57777922048 but there is no extent record
There are no extents for csum range 57780281344-57780285440
Csum exists for 57780281344-57780285440 but there is no extent record
There are no extents for csum range 57780293632-57780301824
Those two sets of errors appear many times with different numerical values.
And here’s the output of tail -n 100 /opt/rockstor/var/log/rockstor.log:
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001241150']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912270750']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001250250']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:47] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001300950']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912280750']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202002050450']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001280050']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-daily_202001270130']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_201912190850']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:06:48] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rockons/.snapshots/nextcloud-main/ncmain-hourly_202001281950']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[28/Mar/2020 20:11:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/bootstrap method: POST
[28/Mar/2020 20:11:23] ERROR [storageadmin.middleware:33] FATAL: the database system is in recovery mode
Traceback (most recent call last):
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
response = self.handle_exception(exc)
File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
response = handler(request, *args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/transaction.py", line 271, in __exit__
connection.set_autocommit(True)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 294, in set_autocommit
self.ensure_connection()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: FATAL: the database system is in recovery mode
[28/Mar/2020 20:45:48] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /login_submit method: POST
[28/Mar/2020 20:45:50] ERROR [storageadmin.middleware:33] could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Traceback (most recent call last):
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/rockstor/src/rockstor/storageadmin/views/home.py", line 37, in login_submit
user = authenticate(username=username, password=password)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/__init__.py", line 74, in authenticate
user = backend.authenticate(**credentials)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/backends.py", line 17, in authenticate
user = UserModel._default_manager.get_by_natural_key(username)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/contrib/auth/models.py", line 162, in get_by_natural_key
return self.get(**{self.model.USERNAME_FIELD: username})
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 328, in get
num = len(clone)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 144, in __len__
self._fetch_all()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 965, in _fetch_all
self._result_cache = list(self.iterator())
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/sql/compiler.py", line 838, in execute_sql
cursor = self.connection.cursor()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection
self.connect()
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
[28/Mar/2020 20:50:12] ERROR [scripts.scheduled_tasks.snapshot:133] Failed to create snapshot at shares/12/snapshots/ncmain-hourly_202003282050
[28/Mar/2020 20:50:13] ERROR [scripts.scheduled_tasks.snapshot:134] Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None
Traceback (most recent call last):
File "/opt/rockstor/src/rockstor/scripts/scheduled_tasks/snapshot.py", line 128, in main
save_error=False)
File "/opt/rockstor/src/rockstor/cli/api_wrapper.py", line 75, in api_call
self.set_token()
File "/opt/rockstor/src/rockstor/cli/api_wrapper.py", line 69, in set_token
raise Exception(msg)
Exception: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None
Output of systemctl status -l rockstor-pre rockstor rockstor-bootstrap:
● rockstor-pre.service - Tasks required prior to starting Rockstor
Loaded: loaded (/etc/systemd/system/rockstor-pre.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2020-03-28 22:51:30 MST; 19min ago
Process: 3051 ExecStart=/opt/rockstor/bin/initrock (code=exited, status=1/FAILURE)
Main PID: 3051 (code=exited, status=1/FAILURE)
Mar 28 22:51:30 rockstor initrock[3051]: sys.exit(scripts.initrock.main())
Mar 28 22:51:30 rockstor initrock[3051]: File "/opt/rockstor/src/rockstor/scripts/initrock.py", line 392, in main
Mar 28 22:51:30 rockstor initrock[3051]: fake_initial_migration_cmd + ['--database=default', 'contenttypes'])
Mar 28 22:51:30 rockstor initrock[3051]: File "/opt/rockstor/src/rockstor/system/osi.py", line 121, in run_command
Mar 28 22:51:30 rockstor initrock[3051]: raise CommandException(cmd, out, err, rc)
Mar 28 22:51:30 rockstor initrock[3051]: system.exceptions.CommandException: Error running a command. cmd = /opt/rockstor/bin/django migrate --noinput --fake-initial --database=default contenttypes. rc = 1. stdout = ['']. stderr = ['Traceback (most recent call last):', ' File "/opt/rockstor/bin/django", line 44, in <module>', " sys.exit(djangorecipe.manage.main('rockstor.settings'))", ' File "/opt/rockstor/eggs/djangorecipe-1.9-py2.7.egg/djangorecipe/manage.py", line 9, in main', ' management.execute_from_command_line(sys.argv)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 354, in execute_from_command_line', ' utility.execute()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 346, in execute', ' self.fetch_command(subcommand).run_from_argv(self.argv)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 394, in run_from_argv', ' self.execute(*args, **cmd_options)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 445, in execute', ' output = self.handle(*args, **options)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/commands/migrate.py", line 93, in handle', ' executor = MigrationExecutor(connection, self.migration_progress_callback)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/executor.py", line 19, in __init__', ' self.loader = MigrationLoader(self.connection)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/loader.py", line 47, in __init__', ' self.build_graph()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/loader.py", line 191, in build_graph', ' self.applied_migrations = recorder.applied_migrations()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/recorder.py", line 59, in applied_migrations', ' self.ensure_schema()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/migrations/recorder.py", line 49, in ensure_schema', ' if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 164, in cursor', ' cursor = self.make_cursor(self._cursor())', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 135, in _cursor', ' self.ensure_connection()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection', ' self.connect()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__', ' six.reraise(dj_exc_type, dj_exc_value, traceback)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 130, in ensure_connection', ' self.connect()', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/base/base.py", line 119, in connect', ' self.connection = self.get_new_connection(conn_params)', ' File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection', ' connection = Database.connect(**conn_params)', ' File "/opt/rockstor/eggs/psycopg2-2.6-py2.7-linux-x86_64.egg/psycopg2/__init__.py", line 164, in connect', ' conn = _connect(dsn, connection_factory=connection_factory, async=async)', 'django.db.utils.OperationalError: FATAL: could not write init file', '', '']
Mar 28 22:51:30 rockstor systemd[1]: rockstor-pre.service: main process exited, code=exited, status=1/FAILURE
Mar 28 22:51:30 rockstor systemd[1]: Failed to start Tasks required prior to starting Rockstor.
Mar 28 22:51:30 rockstor systemd[1]: Unit rockstor-pre.service entered failed state.
Mar 28 22:51:30 rockstor systemd[1]: rockstor-pre.service failed.
● rockstor.service - RockStor startup script
Loaded: loaded (/etc/systemd/system/rockstor.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Mar 28 22:51:30 rockstor systemd[1]: Dependency failed for RockStor startup script.
Mar 28 22:51:30 rockstor systemd[1]: Job rockstor.service/start failed with result 'dependency'.
● rockstor-bootstrap.service - Rockstor bootstrapping tasks
Loaded: loaded (/etc/systemd/system/rockstor-bootstrap.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Mar 28 22:51:30 rockstor systemd[1]: Dependency failed for Rockstor bootstrapping tasks.
Mar 28 22:51:30 rockstor systemd[1]: Job rockstor-bootstrap.service/start failed with result 'dependency'.