Can't stop Rockons or install new ones

maxhq · June 3, 2017, 6:35pm

Hi there,

for some months I have two Rockons running: Gogs and Syncthing, all is fine.
Now I want to stop them or install new ones, both actions fail with similar errors.

When I try to stop Gogs Rockon, the “Waiting” gear starts spinning but after a while disappears with the Rockon still running.
I mainly get the following errors:

==> /opt/rockstor/var/log/supervisord_ztask-daemon_stderr.log <==
ERROR:ztaskd:Error calling storageadmin.views.rockon_helpers.stop. Details:
cannot import name get_uuid_name_map

==> /opt/rockstor/var/log/rockstor.log <==
[03/Jun/2017 20:23:18] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /api/commands/refresh-snapshot-state method: POST
[03/Jun/2017 20:23:18] ERROR [storageadmin.middleware:33] need more than 0 values to unpack

Does anybody have an idea?

Full log entries:

==> /opt/rockstor/var/log/supervisord_ztask-daemon_stderr.log <==
ERROR:ztaskd:Error calling storageadmin.views.rockon_helpers.stop. Details:
cannot import name get_uuid_name_map
WARNING:py.warnings:/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/fields/__init__.py:1474: RuntimeWarning: DateTimeField Task.failed received a naive datetime (2017-06-03 18:23:08.367680) while time zone support is active.
  RuntimeWarning)

Traceback (most recent call last):
  File "/opt/rockstor/eggs/django_ztask-0.1.5-py2.7.egg/django_ztask/management/commands/ztaskd.py", line 114, in _call_function
    __import__(module_name)
  File "/opt/rockstor/src/rockstor/storageadmin/views/__init__.py", line 23, in <module>
    from disk import (DiskMixin, DiskListView, DiskDetailView)  # noqa F401
  File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 31, in <module>
    from system.luks import luks_format_disk, get_unlocked_luks_containers_uuids, \
  File "/opt/rockstor/src/rockstor/system/luks.py", line 23, in <module>
    from system.osi import run_command, get_uuid_name_map
ImportError: cannot import name get_uuid_name_map

==> /opt/rockstor/var/log/rockstor.log <==
[03/Jun/2017 20:23:18] ERROR [storageadmin.middleware:32] Exception occured while processing a request. Path: /api/commands/refresh-snapshot-state method: POST
[03/Jun/2017 20:23:18] ERROR [storageadmin.middleware:33] need more than 0 values to unpack
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 276, in post
    status = True
  File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 150, in import_snapshots
    rusage, eusage = volume_usage(share.pool, snaps_d[s][0])
ValueError: need more than 0 values to unpack

phillxnet · June 6, 2017, 11:13am

@maxhq Hello again. Sorry for slow response but just spotted something in your log.

First off have you rebooted after a recent update? The docker system has malfunctioned a number of times directly after an update; plus from the following:

It looks like some of your code has upgrade and others not, ie the import statement in luks.py (new code file) can’t find the new function get_uuid_name_map in system/osi. Which is odd but can happen, you still have active an old version of system/osi but the new luks.py requires this as indicated by the import statement. They were both part of the same pull request / merge / update however. Not sure if this is the cause of your docker issues but it’s a definite indication that a reboot is in order to hopefully force the use of all new code. That and the know issue of the Rock-on system misbehaving after some updates.

Hope that helps.

maxhq · June 6, 2017, 12:08pm

Hi @phillxnet and thanks for answering.
Indeed I did not reboot. After rebooting now there are other errors which prevent me from (un-)installing Rockons:

ztask-daemon cannot be started, Rockstor UI says:

ztask daemon is not running and could not be started

Traceback (most recent call last):
File “/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py”, line 34, in run
client, addr = self.socket.accept()
File “/usr/lib64/python2.7/socket.py”, line 202, in accept
sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable

And in /opt/rockstor/var/log/supervisord_ztask-daemon_stderr.log:

Traceback (most recent call last):
  File "/opt/rockstor/bin/django", line 44, in <module>
    sys.exit(djangorecipe.manage.main('rockstor.settings'))
  File "/opt/rockstor/eggs/djangorecipe-1.9-py2.7.egg/djangorecipe/manage.py", line 9, in main
    management.execute_from_command_line(sys.argv)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/opt/rockstor/eggs/django_ztask-0.1.5-py2.7.egg/django_ztask/management/commands/ztaskd.py", line 43, in handle
    self._handle(use_reloader, replay_failed)
  File "/opt/rockstor/eggs/django_ztask-0.1.5-py2.7.egg/django_ztask/management/commands/ztaskd.py", line 87, in _handle
    self.io_loop.add_handler(socket, _queue_handler, self.io_loop.READ)
  File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 703, in add_handler
    self._impl.register(fd, events | self.ERROR)
TypeError: argument must be an int, or have a fileno() method.

The last error from my first post is also still there:

Exception occured while processing a request. Path: /api/commands/refresh-snapshot-state method: POST
...
ValueError: need more than 0 values to unpack

maxhq · June 6, 2017, 12:14pm

I just saw that there were earlier errors just after the reboot:

Jun  6 13:54:40 Sigurd bootstrap: BTRFS device scan complete
Jun  6 13:54:40 Sigurd bootstrap: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None
... (last message repeated several times) ...
Jun  6 13:54:52 Sigurd bootstrap: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: Exception while setting access_token for url(http://127.0.0.1:8000): HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: /o/token/ (Caused by <class 'socket.error'>: [Errno 111] Connection refused). content: None
Jun  6 13:54:52 Sigurd systemd: rockstor-bootstrap.service: main process exited, code=exited, status=1/FAILURE
Jun  6 13:54:52 Sigurd systemd: Failed to start Rockstor bootstrapping tasks.
Jun  6 13:54:52 Sigurd systemd: Unit rockstor-bootstrap.service entered failed state.
Jun  6 13:54:52 Sigurd systemd: rockstor-bootstrap.service failed.

[EDIT]

A manual systemctl start rockstor-bootstrap solved this.

phillxnet · June 6, 2017, 12:35pm

@maxhq OK, so what is the current state of your Rockstor function / malfunction?
It should of course start up just fine! Is your system disk or cpu particularly slow? Just pondering the 15 failed attempts in the background of rockstor-bootstrap.

maxhq · June 6, 2017, 1:10pm

Rockstor itself is up and running, CPU usage fine. BTRFS volumes and Samba shares mounted and fine.
Just the Rockons don’t work.

maxhq · June 12, 2017, 10:43pm

I am sorry for bugging you, but do you have any ideas what I could try to cleanly remove the Rockon and reinstall it? Can I safely remove it through docker or will Rockstor get confused then?

Edit

I just removed the Rockons via docker command line. Now I am still stuck because I cannot reinstall them:

1. [Update] button on the Rockons page fails

with error message:

Errors occurred while processing updates for following Rock-ons. Syncthing: ['Cannot add/remove volume definitions of the container(syncthing) as it belongs to an installed Rock-on(Syncthing). Uninstall it first and try again.

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 106, in post
self._create_update_meta(r, rockons[r])
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 267, in _create_update_meta
handle_exception(Exception(e_msg), self.request)
  File "/opt/rockstor/src/rockstor/storageadmin/util.py", line 46, in handle_exception
raise RockStorAPIException(detail=e_msg, trace=traceback.format_exc())

RockStorAPIException: ['Cannot add/remove volume definitions of the container(syncthing) as it belongs to an installed Rock-on(Syncthing). Uninstall it first and try again.', 'Traceback (most recent call last):\n  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 106, in post\n    self._create_update_meta(r, rockons[r])\n  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner\n    return func(*args, **kwargs)\n  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 267, in _create_update_meta\n    handle_exception(Exception(e_msg), self.request)\n  File "/opt/rockstor/src/rockstor/storageadmin/util.py", line 46, in handle_exception\n    raise RockStorAPIException(detail=e_msg, trace=traceback.format_exc())\nRockStorAPIException: [\'Cannot add/remove volume definitions of the container(gogs) as it belongs to an installed Rock-on(Gogs). Uninstall it first and try again.\', \'Traceback (most recent call last):\\n  File "/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py", line 34, in run\\n    client, addr = self.socket.accept()\\n  File "/usr/lib64/python2.7/socket.py", line 202, in accept\\n    sock, addr = self._sock.accept()\\nerror: [Errno 11] Resource temporarily unavailable\\n\']\n']

2. [Uninstall] button on Rockons page fails

with the ztask error message that I posted here again with details.

maxhq · July 10, 2017, 9:53pm

Did someone else come up with this error in the meantime?
Even after reboots I still get the same error message when trying to uninstall the non-functioning Rockon. (See no. 2 above).

Is there any workaround to delete a Rockon? Is it enough to change the entry in the “storageadmin_rockon” PostgreSQL table and remove the docker container+image?

maxhq · August 1, 2017, 9:10pm

In case anyone gets the same message on a custom installation:
My problem was another Yum repo (saltstack-repo) which installed a newer version of python-tornado that caused the error.
I solved it with yum downgrade python-tornado.