[SOLVED] Rockon installations fail (Rockstor 3.9.2-57)

Hello!

Background: I had my Rockstor set up about 3 years ago under the Testing (free) channel, but it was in storage for a while, and when I tried to bring it online again the OS disk wouldn’t boot due to superblock issues, so I re-installed and subscribed to the Stable channel, though the data disks have remained the same, and I have added a new disk to the main pool (It is currently rebalancing).
Due to the re-install, I’ve lost my previous configuration, though I understand rock-on config backup was added between my last use and now, so no big loss.

The issue now is, I had Emby and Deluge running on the previous install, but I wanted to try out Jellyfin, so I made my own Rock-on file based on Emby’s. When I attempted to install it, it failed, pointing me to check the logs. I just wanted a running media server, so I tried the same with Emby, as I figured the given Rockon json might be better, but no luck. In fact it gave the same error in the logs.
Checking /opt/rockstor/var/log/rockstor.log reveals:

[07/Jun/2020 14:20:38] ERROR [storageadmin.views.rockon:85] Rockon (Jellyfin (Linuxserver.io)) is in pending state but there is no pending or failed task for it.
[07/Jun/2020 14:26:17] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

I’ve scanned the forums for posts similar to this, but haven’t found any with the same error, so I’m wondering if I’m missing something or somehow my fresh install got borked? Do let me know other logs I might check to help troubleshoot.

@StephenBrown2, welcome to the community forums.

You could check here, as well to see what the docker daemon had to “say”:

journalctl -xe | grep ‘dockerd’

I am assuming that the RockOn service itself is running?

There was a post with a pending state that you might have to go through (not exactly the same, but appears similar):

2 Likes

Yes, I enabled the Rockon service, and docker is running:

[root@rockstor ~]# systemctl list-unit-files | grep -E 'docker|rock'
docker.service                                enabled
rockstor-bootstrap.service                    enabled
rockstor-fstrim.service                       static
rockstor-pre.service                          enabled
rockstor.service                              enabled
docker.socket                                 disabled
rockstor-fstrim.timer                         disabled

Unfortunately journalctl -xe | grep dockerd has nothing to say, zero output. Also no docker images exist:

[root@rockstor ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@rockstor ~]# docker images ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

I hadn’t tried the delete-rockon script because of the above, but running it anyways, I get some errors in the logs, and then the same message when trying to reinstall:

[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'embyserver']. output: [''] error: ['Error response from daemon: No such container: embyserver', '']
[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'embyserver']. output: [''] error: ['Error response from daemon: No such container: embyserver', '']
[07/Jun/2020 15:34:11] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rmi', 'emby/embyserver:latest']. output: [''] error: ['Error: No such image: emby/embyserver:latest', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'jellyfin']. output: [''] error: ['Error response from daemon: No such container: jellyfin', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'jellyfin']. output: [''] error: ['Error response from daemon: No such container: jellyfin', '']
[07/Jun/2020 15:34:23] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/bin/docker', 'rmi', 'linuxserver/jellyfin:latest']. output: [''] error: ['Error: No such image: linuxserver/jellyfin:latest', '']
[07/Jun/2020 15:36:47] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

Since this is a fresh install, I wouldn’t expect there to be any info in the DB for any Rock-ons as of yet. Regardless, certainly looking forward for the OpenSUSE build: Rock-ons issues :smile:

Testing on my own, docker pull works fine for both images:

[root@rockstor ~]# docker pull linuxserver/jellyfin:latest
latest: Pulling from linuxserver/jellyfin
eade91583e3e: Pull complete
5318222e0b9e: Pull complete
b62b45a46472: Pull complete
119a408efe7a: Pull complete
6a4f40ecbebf: Pull complete
0d945d10e8c3: Pull complete
Digest: sha256:08189868096461e2424ada2bd5363462beb561656a787cf2f9449ca251efeb54
Status: Downloaded newer image for linuxserver/jellyfin:latest
[root@rockstor ~]# docker pull emby/embyserver:latest
latest: Pulling from emby/embyserver
57c14dd66db0: Pull complete
11f641f811ab: Pull complete
304caacb4011: Pull complete
a7d7f597d051: Extracting [=====>                                             ]  32.77kB/284.9kB
29630ca1f53a: Download complete
cb6b62e68549: Download complete

(The image pull did complete for emby as well)

Digest: sha256:c9aa10e6edea05d1b075d1cbdd5ca6db218e0238a77aa6c75ce592ca83a0f040
Status: Downloaded newer image for emby/embyserver:latest

EDIT:

Logs appear to have more issues, not sure if related:

[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/refresh-share-state method: POST
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:33] Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 348, in post
    import_shares(p, request)
  File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 86, in import_shares
    shares_in_pool = shares_info(pool)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 721, in shares_info
    snap_idmap[vol_id])
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 768, in parse_snap_details
    writable = not get_property(full_snap_path, 'ro')
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1844, in get_property
    o, e, rc = run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 176, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/commands/refresh-snapshot-state method: POST
[07/Jun/2020 15:53:22] ERROR [storageadmin.middleware:33] Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
Traceback (most recent call last):
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/rockstor/src/rockstor/storageadmin/views/command.py", line 353, in post
    import_snapshots(share)
  File "/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py", line 209, in import_snapshots
    share.name)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 819, in snaps_info
    stripped_path)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 768, in parse_snap_details
    writable = not get_property(full_snap_path, 'ro')
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1844, in get_property
    o, e, rc = run_command(cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 176, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/caverna/Rock-Ons/btrfs/subvolumes/b360febde5f89dd36ffeb10ab2dbc21efa5235bf87cf3a265f5760b651d0c197: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', '    Gets a property from a btrfs object.', '', '    If no name is specified, all properties for the given object are', '    printed.', '    A filesystem object can be a the filesystem itself, a subvolume,', "    an inode or a device. The '-t <type>' option can be used to explicitly", '    specify what type of object you meant. This is only needed when a', '    property could be set for more then one object type. Possible types', '    are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
[07/Jun/2020 15:55:46] ERROR [storageadmin.views.rockon:85] Rockon (Folding@home Linuxserver.io) is in pending state but there is no pending or failed task for it.
[07/Jun/2020 16:36:30] ERROR [storageadmin.views.rockon:85] Rockon (Emby server) is in pending state but there is no pending or failed task for it.

Hi @StephenBrown2,

Welcome to the forums! Before starting, I’d like to apologize for the too-long delay in getting to reviewing your Github PR… I personally have been completely taken by our effort on the openSUSE rebase that led me with not enough time to take care of all I’d like.

Now, the symptoms you are unfortunately experiencing are indeed rather curious and I can think of two things that might be related.

The first is:

I know balancing is a rather intensive task for your system, and based on the amount of work to do, it will impact your system negatively while it is still running. For instance, have a look at one of @phillxnet’s recent posts and its included screenshot:

As a result, there may have been some timeouts happening while you were trying to install your Rock-ons, which ultimately resulted in some sync errors between what containers are running on your system and what was registered in Rockstor’s database. Pending experts’ opinions (@phillxnet?) I would recommend waiting for the balance to finish.

The other thing I can think of, would be some sync issues if you re-use the same rock-ons root share in your re-installed system (when compared to before the re-installation). In a Rockstor re-installation scenario, I generally prefer to delete this rockons_root share and re-create it, so that I’m sure there won’t be any kind of conflict with pre-existing docker container layers. As docker containers are designed to be nuked and rebuilt anyway, I’ve never hesitated to do that in a Rockstor reinstallation scenario.

Correct. The related database models are blank upon Rockstor re-installation, and re-populated afresh by clicking on the “Update” button in the top right corner of the Rock-ons page. If interested, you will find more information in the wiki post below:

In this post, you will also see that when you click “Submit” at the end of a rock-on installation, Rockstor will run the docker pull image-name command for all containers listed in a given rock-on. Now, note that by default, this docker pull command will not run if it believes the image is already on the system. While this might be a possibility one can envision if you re-used a rock-ons_root share with pre-existing container layers, the fact that the images downloaded successfully when you triggered it manually goes against this hypothesis.

There still seems to be some out-of-sync issues going on, however, so it might be worth trying to reset it completely (and possibly waiting for your balance to finish):

  1. Turn off rock-ons service
  2. /opt/rockstor/bin/delete-rockon "Emby Server" (and any other rock-on for which you saw this error)
  3. Go to Rock-ons > Click on “Update” in the top right
  4. Go to Storage > Shares > <rock-ons_root> > Snapshots > delete all snapshots
  5. Delete <rock-ons_root>
  6. Create a new share to be used for rock-ons root
  7. Configure the Rock-ons service using the share created in step 6 above as rock-ons_root
  8. Turn on the Rock-ons service
  9. Go to Rock-ons page and click on the “Update” button in the top right.

Everything should now be reset and ready to go.

Hope this helps,

I just saw you added more details, so apologies for missing that.

I assume the Rock-Ons share is the share you used as rock-ons_root? That might support my previous message…

No worries, I think the OpenSUSE rebase is much more important than other things, and there are only so many people working on this project.

Yes, I thought that might have something to do with it, so I was sure to include that. It’s a rather large disk I’ve added, so I’ll wait till it’s finished to try again.

The errors were seen immediately after clicking “Submit” in the Install Rockon modal, though.

Yes, I had to recreate the Rock-On share, as I believe I had created it on the OS disk in the first install, which of course was wiped during re-install. This time I created it in the data pool.

Since the balance is still running (at just under 2% an hour, it’ll be a while…), and the rock-ons share is a new one, I’ll first wait for the balance to complete, then try installing again, and if that fails, I’ll go through with clearing out and recreating the rock-ons root and all snapshots.

Yes, that’s the new one I created after the re-install. Thinking about it, I might want to do like I did before and create the rock-on_root on the OS disk again, to avoid performance issues like this in the future. Thoughts?

Good point… and that makes sense based on the new logs you pasted as there seems to be some lower-level problem happening with this share. As I’m not familiar with the inner workings of balance, I’m not sure to what degree the no such file or directory error you’re seeing is expected/abnormal/temporary… maybe @phillxnet can shed some light onto this.

That’s a very good question… It actually is more complex than one might think. The system pool is a rather special case and good practices would recommend to separate data from OS, hence not using the system (ROOT) pool for things such as the rock-ons root. Using it as such, however, does have a lot of appeal in terms of convenience and @phillxnet has done a lot of work to better approach this situation (see link below). As he is the expert, he is better placed to give advice on it, but I believe the bottom line is that not everything is as simple on the system pool as it is on a data pool…

Yes, I currently have a data pool (caverna), and the OS pool (rockstor_rockstor)
17
I can and should probably clean up the shares I’m not using, that were simply imported from before, though I had wanted to re-use the config shares mounted in the Rockons.
45
And of course, the warning is appropriate:
13

@StephenBrown2 Hello here,

I can chip in with regard to @Flox’s comment:

and

You can speed things up by disabling quotas on that pool. I see they are enabled from your screen shots.

Yes, we have some work to do there in migrating our btrfs in partition work to that of the system pool code. That should help to simplify things a little. But as is appreciated there I think, the main problem is folks putting non throw away data on the system drive. Docker images can just be re-fetched so are pretty much throw away in that sense which makes the rock-ons-root share a likely candidate, sort of.

Yes, that’s best as the Rock-ons can then, usually, just pick-up where they left off.

Yes, we need to improve that at bit to give more of an indication that it’s for throw away data only really. But it’s in the queue to do.

With regard to this strange Rock-on behaviour. All I can think is that you may not have yet rebooted after updating from the installer. We changed from docker (installer & CentOS testing channel) to docker-ce (Stable Channel) and that needs a reboot to settle in. But if you are mid balance operation then best just leave it be until that has finished.

Hope that helps.

Yeah, that’s what I was thinking as well. Still gonna wait for the rebalance to finish.

Done, we’ll see how that affects things. 3 days is a long while.

Make sure it’s after/not blocking the OpenSUSE refresh! I want my 5.6 kernel! :grin:

1 Like

Yup, as it turns out, the balancing was the issue. It completed this morning, I deleted all the Rockons from the database, refreshed, and was able to install Emby and my custom Jellyfin RockOn with ease!

I don’t see an option to add [SOLVED] to the title though, so if an admin needs to do that, please go ahead.

2 Likes

Glad it’s all sorted now!

Once we have the rebase out and the next big update to our Django on the way, my hope is that we should be able to surface to the user this kind of heavy and resource-intensive operations running in the background to alleviate the burden caused by this kind of situation.

Nicely done on getting that Rock-on. I’ve personally be meaning to try it for a rather long time but never took the time to do so. If you’re willing to share it with the community, feel free to create a pull request in our rockon-registry repository :wink: .

Cheers!

2 Likes