Tring to access /home#rockons. Reboot did not change anything, additional the Rockon service had to be started manually.
Detailed step by step instructions to reproduce the problem
I’m only guessing… but i setup up watchtower and started adding labels to the rockons. After adding a Label to a Rockon which contains more than one docker container i started seeing issues in the UI. The container was reachable but the Rockon was stuck in the status “Starting”.
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/generics.py", line 243, in get
return self.list(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/rockstor/.venv/lib/python3.11/site-packages/rest_framework/mixins.py", line 38, in list
queryset = self.filter_queryset(self.get_queryset())
^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/opt/rockstor/src/rockstor/storageadmin/views/rockon.py", line 103, in get_queryset
ro.status = rockon_status(ro)
^^^^^^^^^^^^^^^^^
File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 59, in rockon_status
co = DContainer.objects.filter(rockon=ro).order_by("-launch_order")[0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/opt/rockstor/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 450, in __getitem__
return qs._result_cache[0]
~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
There was an issue for @freaktechnik a while ago, where there ended up being multiple instances of Rockons in the database causing issues. Not sure, whether the outcome of using watchtower on multi-docker Rockons could cause something similar:
If your theory is correct, then it could be that Watchtower gets confused, updating only some of the multi-container pieces and not other, and not cleaning up correctly, resulting in some duplication.
May be when you are back home, you can remove/reinstall the multi-docker Rockon(s) (I hope, you don’t have too many) and see whether that addresses the problem, using:
cd /opt/rockstor
$ poetry run delete-rockon <rockon name - case-sensitive>
I’m also running many multi container rockons with watchtower and haven’t really seen an issue like that at all - especially since the container I encountered the issue with was a single-container rockon. I know watchtower doesn’t know about the connection between multi-container rockons and will just update a single one. However, in the past it’s always cleaned up properly for me and only stuff like docker networks would get broken when watchtower runs (which I’d work around with the watchtower labels as necessary).
I recall my issue happening when I had removed the rockon, and then reinstalled from definition but after running a scrub on the system disk because I had to restore on a new disk it broke or similar. So I wasn’t too surprised to see the database issue (it was purely a database issue, only one instance of the container existed) and thus didn’t follow up further.
Do you use set the Tag for the watchtower Rockon to define which container should/should not be updated @freaktechnik ? com.centurylinklabs.watchtower.enable
Most of my Rockons use multi-container definitions.
Guess i’m going to spend some time to set them up again. Probably next week. The Rockon are working for now. Just the UI is broken.
I had for a time used a disable label to exclude a specific container from the updates because as described, it would break the network link that was required for it to communicate with its database… I no longer need it since the multi container setup isn’t relying on the network link anymore, so the containers can be updated by watchtower just fine.
You could possibly make a Rockstor configuration backup, and then apply it. It should skip the items (samba, users, etc.) and just apply the Rockon configs. I have done that once in the past, but, of course YMMV…
