Lost ability to start all my rockons after updating system

Brief description of the problem

I Lost ability to start all my rockons after updating system.

Detailed step by step instructions to reproduce the problem

  1. I clicked the blinking update logo
  2. Clicked update
  3. Closed browser
  4. Lost internet connection (due to Pi-hole shutting off)
  5. restarted machine after resetting my router (thought my router was bad)

Web-UI screenshot

Error Traceback provided on the Web-UI

``` [18/Jun/2021 17:04:07] ERROR [system.osi:196] non-zero code(125) returned by command: ['/usr/bin/docker', 'run', '--name', 'ovpn-data', '-v', '/etc/openvpn', 'busybox:latest']. output: [''] error: ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] [18/Jun/2021 17:04:07] ERROR [storageadmin.views.rockon_helpers:174] Error running a command. cmd = /usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest. rc = 125. stdout = ['']. stderr = ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 171, in install globals().get("%s_install" % rockon.name.lower(), generic_install)(rockon) File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 359, in openvpn_install run_command(volc_cmd, log=True) File "/opt/rockstor/src/rockstor/system/osi.py", line 198, in run_command raise CommandException(cmd, out, err, rc) CommandException: Error running a command. cmd = /usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest. rc = 125. stdout = ['']. stderr = ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] [18/Jun/2021 17:04:51] INFO [system.osi:1337] Ignoring rc=-15 from command (['/sbin/shutdown', '-r', 'now']). [18/Jun/2021 17:07:47] ERROR [system.osi:196] non-zero code(125) returned by command: ['/usr/bin/docker', 'run', '--name', 'ovpn-data', '-v', '/etc/openvpn', 'busybox:latest']. output: [''] error: ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] [18/Jun/2021 17:07:47] ERROR [storageadmin.views.rockon_helpers:174] Error running a command. cmd = /usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest. rc = 125. stdout = ['']. stderr = ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 171, in install globals().get("%s_install" % rockon.name.lower(), generic_install)(rockon) File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 359, in openvpn_install run_command(volc_cmd, log=True) File "/opt/rockstor/src/rockstor/system/osi.py", line 198, in run_command raise CommandException(cmd, out, err, rc) CommandException: Error running a command. cmd = /usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest. rc = 125. stdout = ['']. stderr = ['docker: Error response from daemon: Conflict. The container name "/ovpn-data" is already in use by container "e694ed681315f534a637e83ea18f43a5e6e5f3045bbe129f1e0bfdfa7057aa41". You have to remove (or rename) that container to be able to reuse that name.', "See 'docker run --help'.", ''] [18/Jun/2021 17:11:32] ERROR [system.osi:196] non-zero code(1) returned by command: ['/usr/bin/docker', 'start', 'pi-hole']. output: [''] error: ['Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (ad31ee4a461ceb42c4d41314639f0278e2d899a4023391691345b93f1b4a6828): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol', 'Error: failed to start containers: pi-hole', ''] [18/Jun/2021 17:11:32] ERROR [storageadmin.views.rockon_helpers:95] Exception while starting the rockon (Pi-Hole). [18/Jun/2021 17:11:32] ERROR [storageadmin.views.rockon_helpers:96] Error running a command. cmd = /usr/bin/docker start pi-hole. rc = 1. stdout = ['']. stderr = ['Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (ad31ee4a461ceb42c4d41314639f0278e2d899a4023391691345b93f1b4a6828): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol', 'Error: failed to start containers: pi-hole', ''] Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 93, in generic_start run_command([DOCKER, "start", c.name], log=True) File "/opt/rockstor/src/rockstor/system/osi.py", line 198, in run_command raise CommandException(cmd, out, err, rc) CommandException: Error running a command. cmd = /usr/bin/docker start pi-hole. rc = 1. stdout = ['']. stderr = ['Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (ad31ee4a461ceb42c4d41314639f0278e2d899a4023391691345b93f1b4a6828): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol', 'Error: failed to start containers: pi-hole', ''] ```

@Emmanuel_perez Hello again.

So you may have a failed system update, we haven’t yet seen them in our “Built on openSUSE” variant but it’s entirely possible. It may be that your initiated update was not yet complete and you then restarted mid update.

What is the output, run as root on the Rockstor 4 machine, of:

zypper info rockstor

and

zypper up --no-recommends

Also worth taking a look at the main system journal via

journalctl

However from your log output the following looks iffy:

What is the output of for example:

/usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest

Hope that helps, at least to start out the ‘investigation’. Also note that there were a few upstream docker updates of late (in Leap 15.2) so there may be something tripping us up there.

Hey @phillxnet heres what i got, thanks for the help
Loading repository data…
Reading installed packages…

Information for package rockstor:

Repository : Rockstor-Testing
Name : rockstor
Version : 4.0.7-0
Arch : x86_64
Vendor : YewTreeApps
Installed Size : 74.4 MiB
Installed : Yes
Status : up-to-date
Source package : rockstor-4.0.7-0.src
Summary : Btrfs Network Attached Storage (NAS) Appliance.
Description :
Software raid, snapshot capable NAS solution with built-in file
integrity protection.
Allows for file sharing between network attached devices.

Tesseract:/home/epmachuca # zypper up --no-recommends
Loading repository data…
Reading installed packages…

The following 5 package updates will NOT be installed:
bash bash-completion dracut-kiwi-lib dracut-kiwi-oem-dump
dracut-kiwi-oem-repart
Nothing to do.
Tesseract:/home/epmachuca # journalctl
– Logs begin at Sat 2021-06-19 16:18:53 CDT, end at Sun 2021-06-20
Jun 19 16:18:53 Tesseract kernel: Linux version 5.3.18-lp152.78-def
Jun 19 16:18:53 Tesseract kernel: Command line: BOOT_IMAGE=/boot/vm
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Supporting XSAVE feature
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Supporting XSAVE feature
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Supporting XSAVE feature
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Supporting XSAVE feature
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Supporting XSAVE feature
Jun 19 16:18:53 Tesseract kernel: x86/fpu: xstate_offset[2]: 576,
Jun 19 16:18:53 Tesseract kernel: x86/fpu: xstate_offset[3]: 832,
Jun 19 16:18:53 Tesseract kernel: x86/fpu: xstate_offset[4]: 896,
Jun 19 16:18:53 Tesseract kernel: x86/fpu: Enabled xstate features
Jun 19 16:18:53 Tesseract kernel: BIOS-provided physical RAM map:
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000000000000
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000000005f00
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000000006000
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x00000000000a000
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000000010000
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000003246100
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000003246200
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000003246300
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x0000000038c3100
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x000000003909400
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x00000000391ff00
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x00000000395b800
Jun 19 16:18:53 Tesseract kernel: BIOS-e820: [mem 0x0000000039eff00

I removed opvpn when trying to troubleshoot my self so its not currently installed.

I also upgraded to 4.0.7 in the process as well.

/usr/bin/docker run --name ovpn-data -v /etc/openvpn busybox:latest
docker: Error response from daemon: Conflict. The container name “/ovpn-data” is already in use by container “c1746540ef1
341c97b34b3a91b2b8e4caef0fefa5ec8f455831546fbd57844f5”. You have to remove (or rename) that container to be able to reuse
that name.
See ‘docker run --help’.

I have experienced exactly the same problem after updating my Rockstor system. Several Rock-ons stopped with 255 error, just like Emmanuel_perez.
Also tested with another Rockstor appliance on which I hadn’t enabled Rock-ons yet:

  • I updated the system by clicking the blinking update logo top-right.
  • I activated Rock-on service
  • Tried installing a Rock-on (Scrutiny), the UI switches to the Installed tab, shows Installing, but then it’s empty.
2 Likes

Any recommendations?

So I installed rockstor on a spare machine with 1tB hdd, all i did was try to install pi-hole with no luck. So I dont think its an issue with my original machine.
Also I disabled stable updates on the original machine to see if it would help.

Here’s what i got on the logs on the spare machine.

    cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
IntegrityError: duplicate key value violates unique constraint "storageadmin_snapshot_share_id_name_key"
DETAIL:  Key (share_id, name)=(2, 69933f85aace42b27c65feb77e55fcb9aa130bc34f0adf1f5eb74e4dd19b0dc5) already exists.

[25/Jun/2021 12:39:19] ERROR [system.osi:199] non-zero code(125) returned by command: ['/usr/bin/docker', 'run', '-d', '--restart=unless-stopped', '--name', 'pi-hole', '-v', '/mnt2/dnsmasq_config:/etc/dnsmasq.d', '-v', '/mnt2/pihole_config:/etc/pihole', '-v', '/etc/localtime:/etc/localtime:ro', '-p', '86:80/tcp', '-p', '86:80/udp', '-p', '53:53/tcp', '-p', '53:53/udp', '--cap-add', 'NET_ADMIN', '--dns', '127.0.0.1', '--dns', '8.8.8.8', '-e', 'IPv6=False', '-e', 'WEBPASSWORD=emachuca123', '-e', 'ServerIP=192.168.1.165', 'pihole/pihole:latest']. output: ['1567137ea7f7d0532db7e6286b0e6b79ba5166e6da03fa52da5df5901b25cb5f', ''] error: ['WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.', 'docker: Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (035681160de070ffce5f1b4848257acf06a952d69585e5d53ff8e0b8938c4003): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol.', '']
[25/Jun/2021 12:39:19] ERROR [storageadmin.views.rockon_helpers:207] Error running a command. cmd = /usr/bin/docker run -d --restart=unless-stopped --name pi-hole -v /mnt2/dnsmasq_config:/etc/dnsmasq.d -v /mnt2/pihole_config:/etc/pihole -v /etc/localtime:/etc/localtime:ro -p 86:80/tcp -p 86:80/udp -p 53:53/tcp -p 53:53/udp --cap-add NET_ADMIN --dns 127.0.0.1 --dns 8.8.8.8 -e IPv6=False -e WEBPASSWORD=emachuca123 -e ServerIP=192.168.1.165 pihole/pihole:latest. rc = 125. stdout = ['1567137ea7f7d0532db7e6286b0e6b79ba5166e6da03fa52da5df5901b25cb5f', '']. stderr = ['WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.', 'docker: Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (035681160de070ffce5f1b4848257acf06a952d69585e5d53ff8e0b8938c4003): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol.', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 204, in install
    globals().get("{}_install".format(rockon.name.lower()), generic_install)(rockon)
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py", line 390, in generic_install
    run_command(cmd, log=True)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 201, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker run -d --restart=unless-stopped --name pi-hole -v /mnt2/dnsmasq_config:/etc/dnsmasq.d -v /mnt2/pihole_config:/etc/pihole -v /etc/localtime:/etc/localtime:ro -p 86:80/tcp -p 86:80/udp -p 53:53/tcp -p 53:53/udp --cap-add NET_ADMIN --dns 127.0.0.1 --dns 8.8.8.8 -e IPv6=False -e WEBPASSWORD=emachuca123 -e ServerIP=192.168.1.165 pihole/pihole:latest. rc = 125. stdout = ['1567137ea7f7d0532db7e6286b0e6b79ba5166e6da03fa52da5df5901b25cb5f', '']. stderr = ['WARNING: Localhost DNS setting (--dns=127.0.0.1) may fail in containers.', 'docker: Error response from daemon: driver failed programming external connectivity on endpoint pi-hole (035681160de070ffce5f1b4848257acf06a952d69585e5d53ff8e0b8938c4003): Error starting userland proxy: listen tcp6 [::]:86: socket: address family not supported by protocol.', '']
[25/Jun/2021 12:39:19] INFO [storageadmin.tasks:63] Task [install] completed OK
[25/Jun/2021 12:39:44] ERROR [storageadmin.views.network:213] NetworkConnection matching query does not exist.
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/network.py", line 207, in update_connection
    name=dconfig["connection"]
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
DoesNotExist: NetworkConnection matching query does not exist.

I have found the cause of our misery! Apparently by updating our system, a new version of Docker (20.10.6) is installed. I found out (hooray for google) that this version has a problem with disabled IPv6: it ignores this and tries to bind ports using IPv6, even if IPv6 has been disabled on the system. This causes all the errors.

The only solutions are: enabling IPv6 or downgrading Docker. Because IPv6 on Rockstor is disabled intentionally, I chose to downgrade Docker and this worked! all my Rock-ons are functioning now.

In system shell, I entered: sudo zypper install --oldpackage docker-19.03.15_ce-lp152.2.9.1.x86_64
this installs the previous version available in Leap 15.2.

5 Likes

@sanderweel You are my hero! This worked for me for most of my rockons…

I had to uninstall each one and reinstall them one by one, this is my fault since i tried a few things trying to get the rockons to run. I ran “prune” command for dockers.

I still was not able to install Open vpn, and Transmission is stuck on “uninstalling”

  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py", line 37, in container_status
    name,
  File "/opt/rockstor/src/rockstor/system/osi.py", line 201, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
[25/Jun/2021 14:35:50] INFO [storageadmin.views.rockon:103] Rockon (Transmission) state pending and no pending task: assuming task is mid execution.
[25/Jun/2021 14:35:53] ERROR [storageadmin.views.rockon_utils:56] Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py", line 37, in container_status
    name,
  File "/opt/rockstor/src/rockstor/system/osi.py", line 201, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
[25/Jun/2021 14:35:53] INFO [storageadmin.views.rockon:103] Rockon (Transmission) state pending and no pending task: assuming task is mid execution.
[25/Jun/2021 14:35:58] ERROR [storageadmin.views.rockon_utils:56] Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py", line 37, in container_status
    name,
  File "/opt/rockstor/src/rockstor/system/osi.py", line 201, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
[25/Jun/2021 14:35:58] INFO [storageadmin.views.rockon:103] Rockon (Transmission) state pending and no pending task: assuming task is mid execution.
[25/Jun/2021 14:36:02] ERROR [storageadmin.views.rockon_utils:56] Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/rockon_utils.py", line 37, in container_status
    name,
  File "/opt/rockstor/src/rockstor/system/osi.py", line 201, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker inspect -f Error:{{.State.Error}},ExitCode:{{.State.ExitCode}},Running:{{.State.Running}} resilio-sync. rc = 1. stdout = ['', '']. stderr = ['Error: No such object: resilio-sync', '']
[25/Jun/2021 14:36:03] INFO [storageadmin.views.rockon:103] Rockon (Transmission) state pending and no pending task: assuming task is mid execution.

@sanderweel, @Emmanuel_perez, sorry for the silence on that issue; I personally am extremely short on time lately and couldn’t find the time to properly help with that. Thankfully, @sanderweel made a very nice find, well done!

It’s unfortunate that this happened; it’s apparently a bug on docker’s 20.10.6. Thanks to @sanderweel’s finding, it turns out it is a known issue:
https://github.com/moby/moby/issues/42288

Fortunately, a fix has already been merged:
https://github.com/moby/moby/pull/42413

… and is included in the next release 20.10.7:

Hopefully it’ll land in the openSUSE repos soon.

1 Like

Thanks @Flox, What can I do about the “transmission” rockon stuck on uninstalling?

@sanderweel Nice find, and thanks from me also for sharing, much appreciated. We do hope to run IPv6 enabled in time but with our current CentOS → “Built on openSUSE” transition we just had to cut our losses on this one and disable IPv6 whole-sale via the kerrnel command Grub config for now. At least until we can support such configurations within our own network config Web-UI component which currently is purely IPv4.

@Emmanuel_perez
Re:

Hopefully I can chip in on this one. We have an ‘emergency’ script that does a force clean of a ‘stuck’ state of a Rock-on. See:

I’ve now created the following documentation issue re ‘stuck’ Rock-ons:
https://github.com/rockstor/rockstor-doc/issues/276

Hope that helps.

2 Likes

I actually tried this with no success, found the post a while back and had bookmarked it.

Im more than sure that I messed up the dockers beyond my capabilities of repairing it. I ended up reinstalling the rockstor 4.0.4
As of now everything is working great.

1 Like

Has this been fixed yet? I’ve been having to update to new docker (only to get the new Rockstor updates) then having to go downgrade.

@Emmanuel_perez Hello again.
Re:

Not as far as I’m aware. @Flox has an eye on upstream and we are awaiting it’s merge into Leap’s repos. For the time being you could 'addlock" the docker rpm version using zypper once you have manually downgraded. We actually do this in one scenario with the Rockstor rpm version here:

Then once this is resolved upstream you can just 'removelock" via zypper.

Hope that helps.

4 Likes

Just to update, I updated my system to the new 4.0.8 version of rockstor and I am happy to say that all my rock-ons/docker containers are all working as they should.

After updating to 4.0.8…

I removed the addlock and updated docker and all the rock-ons are working as they should

Thank you @phillxnet and the rockstor team for all the hard work!

For my “Teaming” issue, I still have issues with it and will update my open thread on that.

3 Likes