the rockons API endpoint errors, with an empty error description in the UI. In rockstor.log I get the following:
[29/Oct/2024 01:37:28] ERROR [storageadmin.middleware:33] Exception occurred while processing a request. Path: /api/rockons method: GET
[29/Oct/2024 01:37:28] ERROR [storageadmin.middleware:34] list index out of range
Traceback (most recent call last):
File "/opt/rockstor/.venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/rest_framework/views.py", line 495, in dispatch
response = self.handle_exception(exc)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/rest_framework/views.py", line 455, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/rest_framework/views.py", line 466, in raise_uncaught_exception
raise exc
IndexError: list index out of range
As I have a couple of custom rockon definitions I wouldn’t exclude user error here, but the trace doesn’t seem to give any useful info of what might be wrong to me.
Reading through the full log since a reboot, there’s a bunch of errors in the style of
CommandException: Error running a command. cmd = /usr/bin/nmcli d show vethf96fb47. rc = 10. stdout = ['']. stde
rr = ["Error: Device 'vethf96fb47' not found.", '']
[29/Oct/2024 01:33:38] ERROR [storageadmin.views.network:213] NetworkConnection matching query does not exist.
Traceback (most recent call last):
File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 207, in update_connection
name=dconfig["connection"]
File "/opt/rockstor/.venv/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/rockstor/.venv/lib/python2.7/site-packages/django/db/models/query.py", line 380, in get
self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.
which I guess means my docker networks and what rockstor expects got desynced?
From what I can tell of the running containers all their networks that they’re actually using exist. Since I can’t see the rockons page due to this error I can’t tell if there’s any other issues with networks, or check if the configuration in rockstor matches what’s replicated with docker.
Yes, 4.6.1-0 with “Linux 5.14.21-150500.55.73-default”
A brief thought here on these veth devices; sorry for being so brief, I’m really short on time, unfortunately.
It would need confirmation, but if I remember correctly, the veth devices are created by docker when the container is started, or when docker “turns on” at boot, depending on the nature of the docker network. Now, one also needs to keep in mind that Rockstor checks the network and devices listed by nmcli and updates its database accordingly so that the networks/devices that exist on the machine are used as the ground truth. Combining the two, you may see some sort of race condition where Rockstor sees that these veth devices no longer exist at boot as Docker has not initialized them yet (hasn’t fully turned on yet).
Sorry I can’t provide a similar insight on your original issue yet, though… Still wondering about what is happening.
Interestingly the networks page also errored before a reboot, but is now fine after a reboot. So I guess that repaired itself by updating the database to match reality?
