Broken rockons after snapshot removal

Rock-ons
1

Updated to 3.8.16-16 and have not been able to get my Rockons to run. Service is running but none of the installed Rockons will run. I have 3 installed, Emby, Openvpn, and ZoneMinder. Before the upgrade Openvpn, and ZoneMinder were both running reliably. Emby was not, and I hadn’t put any effort into fixing yet. When I toggle the Openvpn service to on it doesn’t start, but generates these log entries in Rockstor Log. During my update I went on a cleaning frenzy and may have deleted the snapshots since I don’t have any older then February, but I’ve had my server up over a year. I didn’t realize they were critical to Rockon operations, may be a chance to add a banner somewhere to prevent users like me from shooting ourselves in the foot.

[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:39:43] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
[17/Mar/2017 12:39:56] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:39:56] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:39:57] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
[17/Mar/2017 12:39:57] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:39:57] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:39:57] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
[17/Mar/2017 12:40:05] ERROR [storageadmin.views.rockon_helpers:78] Exception while starting the rockon(OpenVPN)
[17/Mar/2017 12:40:05] ERROR [storageadmin.views.rockon_helpers:79] Error running a command. cmd = /usr/bin/docker start ovpn-data. rc = 1. stdout = [’’]. stderr = [‘Error response from daemon: No such container: ovpn-data’, ‘Error: failed to start containers: ovpn-data’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 76, in generic_start
run_command([DOCKER, ‘start’, c.name])
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker start ovpn-data. rc = 1. stdout = [’’]. stderr = [‘Error response from daemon: No such container: ovpn-data’, ‘Error: failed to start containers: ovpn-data’, ‘’]
[17/Mar/2017 12:40:05] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:40:05] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:40:20] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} embyserver. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: embyserver’, ‘’]
[17/Mar/2017 12:40:21] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} zoneminder. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: zoneminder’, ‘’]
[17/Mar/2017 12:40:21] ERROR [storageadmin.views.rockon_utils:49] Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py”, line 31, in container_status
’{{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }}’, name]) # noqa E501
File “/opt/rockstor/src/rockstor/system/osi.py”, line 110, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f {{range $key, $value := .State}}{{$key}}:{{$value}},{{ end }} openvpn. rc = 1. stdout = [’’, ‘’]. stderr = [‘Error: No such object: openvpn’, ‘’]

A docker history generates this:
[root@jonesville ~]# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 16
Server Version: 1.13.1
Storage Driver: btrfs
Build Version: Btrfs v4.4.1
Library Version: 101
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.8.7-1.el7.elrepo.x86_64
Operating System: Rockstor 3 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.64 GiB
Name: jonesville
ID: LLAA:U6LM:2Y3H:VKER:PG6N:3Y4C:5ZR5:DK3S:676K:FXON:TORP:OHJZ
Docker Root Dir: /mnt2/Rockons
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

I’ve been out to /mnt2/Rockons and have 4 entries that I assume are containers (which is 1 more than I expected, and this image list looks like it’s short a few names
[root@jonesville containers]# ls
654510a8e215ba25dd023ce584002d4659acb900d1772365d93c0cf021be0f0c b01024b71dff5a7e6825bb8395e5a9171fc7944fb045795f97427b958cbdf24b
8782c9fb56efbf53a8107603387d6e0cd9e5f7e861559177332338b3270a8740 d8773256a60d9172341f34d489be842e9f929c4751c8775e80d69f0f04f12517
[root@jonesville containers]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
emby/embyserver latest 3104abfc9331 5 weeks ago 625 MB
50fbb84d0602 10 months ago 125 MB
22935084d814 12 months ago 1.11 MB
9c57fbc62cb4 12 months ago 5.25 MB
1b5f57474aa5 21 months ago 188 MB
1344c89b2d74 22 months ago 125 MB
[root@jonesville containers]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@jonesville containers]#

Looks like I need to reestablish some relationships, but not sure how to start.
Any help greatly appreciated.

2

@D_Jones Hello again.
I have finally created an issue set for this usability shortfall, I was fairly sure we already had one in rockstor-core but apparently not, so here it is:


and it’s partner in the rockstor-doc repo:

Would you mind changing the title of this thread to reflect only the snapshot removal (ie “remove update or”) as this is enough to explain your reported issue here. Also we have had a long standing problem with needing to reboot after each update which affected docker functionality but usually simply switching it on again sorted this but not always.

As for getting things back in order I would suggest making a new rock-ons-root share and re-establishing your rock-ons using their existing config and data shares. Attempting to uninstall what ones you can would be good first of course. That way the root or ‘system part’ of all the rockons / dockers should be re-established on the new differently named share.

Hope that helps and we definitely have a way to go on the robustness / available clean up tools front for the Rock-ons / docker subsystem but hopefully bit by bit we can get this into proper shape.

Thanks for reporting your issues.

1 Like