[SOLVED] Rockon installations fail (Rockstor 3.9.2-57)

[StephenBrown2 (Stephen Brown II)]

Hello!

Background: I had my Rockstor set up about 3 years ago under the Testing (free) channel, but it was in storage for a while, and when I tried to bring it online again the OS disk wouldn’t boot due to superblock issues, so I re-installed and subscribed to the Stable channel, though the data disks have remained the same, and I have added a new disk to the main pool (It is currently rebalancing).
Due to the re-install, I’ve lost my previous configuration, though I understand rock-on config backup was added between my last use and now, so no big loss.

The issue now is, I had Emby and Deluge running on the previous install, but I wanted to try out Jellyfin, so I made my own Rock-on file based on Emby’s. When I attempted to install it, it failed, pointing me to check the logs. I just wanted a running media server, so I tried the same with Emby, as I figured the given Rockon json might be better, but no luck. In fact it gave the same error in the logs.
Checking /opt/rockstor/var/log/rockstor.log reveals:

[07/Jun/2020 14:20:38] ERROR [storageadmin.views.rockon:85] Rockon (Jellyfin (Linuxserver.io)) is in pending state but there is no pending or failed task for it.
[07/Jun/2020 14:26:17] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

I’ve scanned the forums for posts similar to this, but haven’t found any with the same error, so I’m wondering if I’m missing something or somehow my fresh install got borked? Do let me know other logs I might check to help troubleshoot.

[Hooverdan (Dan W.)]

@StephenBrown2, welcome to the community forums.

You could check here, as well to see what the docker daemon had to “say”:

journalctl -xe | grep ‘dockerd’

I am assuming that the RockOn service itself is running?

There was a post with a pending state that you might have to go through (not exactly the same, but appears similar):

Rock-ons issues

Hi @maclm, This error indicates a rather complex state in which a previous task for the rock-on (usually its installation) was pending, but somehow “vanished” without throwing an error that Rockstor could catch. It’s a rather tricky one to reproduce and thus better deal with it, but there are plans for big improvements in the future in this domain. Now, to get you back on track, I would start by resetting the database information related to these rock-ons. We have an as-yet undocumented script…

[StephenBrown2 (Stephen Brown II)]

Yes, I enabled the Rockon service, and docker is running:

[root@rockstor ~]# systemctl list-unit-files | grep -E 'docker|rock'
docker.service                                enabled
rockstor-bootstrap.service                    enabled
rockstor-fstrim.service                       static
rockstor-pre.service                          enabled
rockstor.service                              enabled
docker.socket                                 disabled
rockstor-fstrim.timer                         disabled

Unfortunately journalctl -xe | grep dockerd has nothing to say, zero output. Also no docker images exist:

[root@rockstor ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@rockstor ~]# docker images ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

I hadn’t tried the delete-rockon script because of the above, but running it anyways, I get some errors in the logs, and then the same message when trying to reinstall:

[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'embyserver']. output: [''] error: ['Error response from daemon: No such container: embyserver', '']
[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'embyserver']. output: [''] error: ['Error response from daemon: No such container: embyserver', '']
[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rmi', 'emby/embyserver:latest']. output: [''] error: ['Error: No such image: emby/embyserver:latest', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'jellyfin']. output: [''] error: ['Error response from daemon: No such container: jellyfin', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'jellyfin']. output: [''] error: ['Error response from daemon: No such container: jellyfin', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rmi', 'linuxserver/jellyfin:latest']. output: [''] error: ['Error: No such image: linuxserver/jellyfin:latest', '']
[07/Jun/2020 15:36:47] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

Since this is a fresh install, I wouldn’t expect there to be any info in the DB for any Rock-ons as of yet. Regardless, certainly looking forward for the OpenSUSE build: Rock-ons issues - #6 by Flox

[StephenBrown2 (Stephen Brown II)]

Testing on my own, docker pull works fine for both images:

[root@rockstor ~]# docker pull linuxserver/jellyfin:latest
latest: Pulling from linuxserver/jellyfin
eade91583e3e: Pull complete
5318222e0b9e: Pull complete
b62b45a46472: Pull complete
119a408efe7a: Pull complete
6a4f40ecbebf: Pull complete
0d945d10e8c3: Pull complete
Digest: sha256:08189868096461e2424ada2bd5363462beb561656a787cf2f9449ca251efeb54
Status: Downloaded newer image for linuxserver/jellyfin:latest
[root@rockstor ~]# docker pull emby/embyserver:latest
latest: Pulling from emby/embyserver
57c14dd66db0: Pull complete
11f641f811ab: Pull complete
304caacb4011: Pull complete
a7d7f597d051: Extracting [=====>                                             ]  32.77kB/284.9kB
29630ca1f53a: Download complete
cb6b62e68549: Download complete

(The image pull did complete for emby as well)

Digest: sha256:c9aa10e6edea05d1b075d1cbdd5ca6db218e0238a77aa6c75ce592ca83a0f040
Status: Downloaded newer image for emby/embyserver:latest

EDIT:

Logs appear to have more issues, not sure if related:

[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/refresh-share-state method: POST
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:33] Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 348, in post
    import_shares(p, request)
  File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 86, in import_shares
    shares_in_pool = shares_info(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 721, in shares_info
    snap_idmap[vol_id])
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 768, in parse_snap_details
    writable = not get_property(full_snap_path, 'ro')
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1844, in get_property
    o, e, rc = run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 176, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/refresh-snapshot-state method: POST
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:33] Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 353, in post
    import_snapshots(share)
  File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 209, in import_snapshots
    share.name)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 819, in snaps_info
    stripped_path)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 768, in parse_snap_details
    writable = not get_property(full_snap_path, 'ro')
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1844, in get_property
    o, e, rc = run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 176, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
[07/Jun/2020 15:55:46] ERROR [storageadmin.views.rockon:85] Rockon (Folding@home Linuxserver.io) is in pending state but there is no pending or failed task for it.
[07/Jun/2020 16:36:30] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

[Flox]

Hi @StephenBrown2,

Welcome to the forums! Before starting, I’d like to apologize for the too-long delay in getting to reviewing your Github PR… I personally have been completely taken by our effort on the openSUSE rebase that led me with not enough time to take care of all I’d like.

Now, the symptoms you are unfortunately experiencing are indeed rather curious and I can think of two things that might be related.

The first is:

@StephenBrown2

(It is currently rebalancing).

I know balancing is a rather intensive task for your system, and based on the amount of work to do, it will impact your system negatively while it is still running. For instance, have a look at one of @phillxnet’s recent posts and its included screenshot:

Error running a command. (Create a Share)

@b8two Hello again. Re: This simply means that btrfs had to wait for something for more than 120 seconds which is not expected on adequate hardware. This is normal. A drive removal is a form of balance that sets the target size of the to-be-removed drive as zero. Hence info is read from that drive but not, generally, written. And over time the read info is re-located on one of the other drives until no info is left where upon it is not longer part of the drive. Rockstor referes to this as a…

As a result, there may have been some timeouts happening while you were trying to install your Rock-ons, which ultimately resulted in some sync errors between what containers are running on your system and what was registered in Rockstor’s database. Pending experts’ opinions (@phillxnet?) I would recommend waiting for the balance to finish.

The other thing I can think of, would be some sync issues if you re-use the same rock-ons root share in your re-installed system (when compared to before the re-installation). In a Rockstor re-installation scenario, I generally prefer to delete this rockons_root share and re-create it, so that I’m sure there won’t be any kind of conflict with pre-existing docker container layers. As docker containers are designed to be nuked and rebuilt anyway, I’ve never hesitated to do that in a Rockstor reinstallation scenario.

@StephenBrown2

Since this is a fresh install, I wouldn’t expect there to be any info in the DB for any Rock-ons as of yet.

Correct. The related database models are blank upon Rockstor re-installation, and re-populated afresh by clicking on the “Update” button in the top right corner of the Rock-ons page. If interested, you will find more information in the wiki post below:

Rock-on framework implementation

Note: this document will be split in several posts in order to fit within characters’ limit. Make sure to browse the second post as well for more information . Preambule This is a wikified post documenting how Rockstor implements the rock-on framework, and should thus be considered a live document expected to be updated as necessary. As a user documentation already exists on how to use the framework (see link below), however, this write-up intends to first summarize the overall rock-on …

In this post, you will also see that when you click “Submit” at the end of a rock-on installation, Rockstor will run the docker pull image-name command for all containers listed in a given rock-on. Now, note that by default, this docker pull command will not run if it believes the image is already on the system. While this might be a possibility one can envision if you re-used a rock-ons_root share with pre-existing container layers, the fact that the images downloaded successfully when you triggered it manually goes against this hypothesis.

There still seems to be some out-of-sync issues going on, however, so it might be worth trying to reset it completely (and possibly waiting for your balance to finish):

1. Turn off rock-ons service
2. /opt/rockstor/bin/delete-rockon "Emby Server" (and any other rock-on for which you saw this error)
3. Go to Rock-ons > Click on “Update” in the top right
4. Go to Storage > Shares > <rock-ons_root> > Snapshots > delete all snapshots
5. Delete <rock-ons_root>
6. Create a new share to be used for rock-ons root
7. Configure the Rock-ons service using the share created in step 6 above as rock-ons_root
8. Turn on the Rock-ons service
9. Go to Rock-ons page and click on the “Update” button in the top right.

Everything should now be reset and ready to go.

Hope this helps,

[Flox]

EDIT:

Logs appear to have more issues, not sure if related:

I just saw you added more details, so apologies for missing that.

I assume the Rock-Ons share is the share you used as rock-ons_root? That might support my previous message…

[StephenBrown2 (Stephen Brown II)]

I’d like to apologize for the too-long delay in getting to reviewing your Github PR… I personally have been completely taken by our effort on the openSUSE rebase that led me with not enough time to take care of all I’d like.

No worries, I think the OpenSUSE rebase is much more important than other things, and there are only so many people working on this project.

balancing is a rather intensive task

Yes, I thought that might have something to do with it, so I was sure to include that. It’s a rather large disk I’ve added, so I’ll wait till it’s finished to try again.

btrfs had to wait for something for more than 120 seconds

The errors were seen immediately after clicking “Submit” in the Install Rockon modal, though.

if you re-used a rock-ons_root share with pre-existing container layers, the fact that the images downloaded successfully when you triggered it manually goes against this hypothesis.

Yes, I had to recreate the Rock-On share, as I believe I had created it on the OS disk in the first install, which of course was wiped during re-install. This time I created it in the data pool.

Since the balance is still running (at just under 2% an hour, it’ll be a while…), and the rock-ons share is a new one, I’ll first wait for the balance to complete, then try installing again, and if that fails, I’ll go through with clearing out and recreating the rock-ons root and all snapshots.

I assume the Rock-Ons share is the share you used as rock-ons_root? That might support my previous message…

Yes, that’s the new one I created after the re-install. Thinking about it, I might want to do like I did before and create the rock-on_root on the OS disk again, to avoid performance issues like this in the future. Thoughts?

[Flox]

@StephenBrown2

The errors were seen immediately after clicking “Submit” in the Install Rockon modal, though.

Flox:

Good point… and that makes sense based on the new logs you pasted as there seems to be some lower-level problem happening with this share. As I’m not familiar with the inner workings of balance, I’m not sure to what degree the no such file or directory error you’re seeing is expected/abnormal/temporary… maybe @phillxnet can shed some light onto this.

@StephenBrown2

Thinking about it, I might want to do like I did before and create the rock-on_root on the OS disk again, to avoid performance issues like this in the future. Thoughts?

That’s a very good question… It actually is more complex than one might think. The system pool is a rather special case and good practices would recommend to separate data from OS, hence not using the system (ROOT) pool for things such as the rock-ons root. Using it as such, however, does have a lot of appeal in terms of convenience and @phillxnet has done a lot of work to better approach this situation (see link below). As he is the expert, he is better placed to give advice on it, but I believe the bottom line is that not everything is as simple on the system pool as it is on a data pool…

[StephenBrown2 (Stephen Brown II)]

@Flox

The system pool is a rather special case and good practices would recommend to separate data from OS, hence not using the system (ROOT) pool for things such as the rock-ons root. Using it as such, however, does have a lot of appeal in terms of convenience and @phillxnet has done a lot of work to better approach this situation (see link below). As he is the expert, he is better placed to give advice on it, but I believe the bottom line is that not everything is as simple on the system pool as it is on a data pool…

Yes, I currently have a data pool (caverna), and the OS pool (rockstor_rockstor)

I can and should probably clean up the shares I’m not using, that were simply imported from before, though I had wanted to re-use the config shares mounted in the Rockons.

And of course, the warning is appropriate:

[phillxnet (Philip Guyton)]

@StephenBrown2 Hello here,

I can chip in with regard to @Flox’s comment:

I would recommend waiting for the balance to finish.

and

Since the balance is still running (at just under 2% an hour, it’ll be a while…),

You can speed things up by disabling quotas on that pool. I see they are enabled from your screen shots.

… but I believe the bottom line is that not everything is as simple on the system pool as it is on a data pool…

Yes, we have some work to do there in migrating our btrfs in partition work to that of the system pool code. That should help to simplify things a little. But as is appreciated there I think, the main problem is folks putting non throw away data on the system drive. Docker images can just be re-fetched so are pretty much throw away in that sense which makes the rock-ons-root share a likely candidate, sort of.

… though I had wanted to re-use the config shares mounted in the Rockons.

Yes, that’s best as the Rock-ons can then, usually, just pick-up where they left off.

And of course, the warning is appropriate:

Yes, we need to improve that at bit to give more of an indication that it’s for throw away data only really. But it’s in the queue to do.

With regard to this strange Rock-on behaviour. All I can think is that you may not have yet rebooted after updating from the installer. We changed from docker (installer & CentOS testing channel) to docker-ce (Stable Channel) and that needs a reboot to settle in. But if you are mid balance operation then best just leave it be until that has finished.

Hope that helps.

[StephenBrown2 (Stephen Brown II)]

Docker images can just be re-fetched so are pretty much throw away in that sense which makes the rock-ons-root share a likely candidate, sort of.

Yeah, that’s what I was thinking as well. Still gonna wait for the rebalance to finish.

speed things up by disabling quotas on that pool.

Done, we’ll see how that affects things. 3 days is a long while.

it’s in the queue to do

Make sure it’s after/not blocking the OpenSUSE refresh! I want my 5.6 kernel!

[StephenBrown2 (Stephen Brown II)]

Yup, as it turns out, the balancing was the issue. It completed this morning, I deleted all the Rockons from the database, refreshed, and was able to install Emby and my custom Jellyfin RockOn with ease!

I don’t see an option to add [SOLVED] to the title though, so if an admin needs to do that, please go ahead.

[Flox]

Glad it’s all sorted now!

Once we have the rebase out and the next big update to our Django on the way, my hope is that we should be able to surface to the user this kind of heavy and resource-intensive operations running in the background to alleviate the burden caused by this kind of situation.

@StephenBrown2

my custom Jellyfin RockOn

Nicely done on getting that Rock-on. I’ve personally be meaning to try it for a rather long time but never took the time to do so. If you’re willing to share it with the community, feel free to create a pull request in our rockon-registry repository .
https://github.com/rockstor/rockon-registry/pulls

Cheers!