Final outcome: All drives passed the extended SMART diags with no errors. Deleting and recreating the pool solved the problem and (after 20 hours of data restore, 4 hours running a balance, and letting Emby rescan files for night), all is well.
I ran onto 1 weird issue: the Rockons service failed; the Emby rockon was missing and errored out when trying to reinstall it. I deleted everything from the Rockons share and the Emby shares & started over. I can’t say why the restore of those shares failed but I’d had had enough of this issue. (For Emby, I was able to restore my playlists - recreating those would have been the biggest pain.)
So at the end of the day (or the fortnight), all is well again.
[02/Dec/2021 15:45:32] ERROR [storageadmin.middleware:34] Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/Pool1/Rockons/btrfs/subvolumes/ba021924c63ed21be974dde66822888967e50ee2e5a19f2d248565a0d7cfda83 ro. rc = 1. stdout = ['']. stderr = ['ERROR: failed to open /mnt2/Pool1/Rockons/btrfs/subvolumes/ba021924c63ed21be974dde66822888967e50ee2e5a19f2d248565a0d7cfda83: No such file or directory', 'ERROR: failed to detect object type: No such file or directory', 'usage: btrfs property get [-t <type>] <object> [<name>]', '', ' Gets a property from a btrfs object.', '', ' If no name is specified, all properties for the given object are', ' printed.', ' A filesystem object can be a the filesystem itself, a subvolume,', " an inode or a device. The '-t <type>' option can be used to explicitly", ' specify what type of object you meant. This is only needed when a', ' property could be set for more then one object type. Possible types', ' are s[ubvol], f[ilesystem], i[node] and d[evice].', '', '']
Traceback (most recent call last):
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py”, line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py”, line 58, in wrapped_view
return view_func(*args, **kwargs)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py”, line 71, in view
return self.dispatch(request, *args, **kwargs)
File “/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py”, line 452, in dispatch
response = self.handle_exception(exc)
File “/opt/rockstor/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py”, line 449, in dispatch
response = handler(request, *args, **kwargs)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py”, line 145, in inner
return func(*args, **kwargs)
File “/opt/rockstor/src/rockstor/storageadmin/views/command.py”, line 415, in post
import_snapshots(share)
File “/opt/rockstor/src/rockstor/storageadmin/views/share_helpers.py”, line 243, in import_snapshots
snaps_d = snaps_info(share.pool.mnt_pt, share.name)
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 883, in snaps_info
pool_mnt_pt, stripped_path
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 829, in parse_snap_details
writable = not get_property(full_snap_path, “ro”)
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 2017, in get_property
o, e, rc = run_command(cmd)
File “/opt/rockstor/src/rockstor/system/osi.py”, line 176, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/btrfs property get /mnt2/Pool1/Rockons/btrfs/subvolumes/ba021924c63ed21be974dde66822888967e50ee2e5a19f2d248565a0d7cfda83 ro. rc = 1. stdout = [’’]. stderr = [‘ERROR: failed to open /mnt2/Pool1/Rockons/btrfs/subvolumes/ba021924c63ed21be974dde66822888967e50ee2e5a19f2d248565a0d7cfda83: No such file or directory’, ‘ERROR: failed to detect object type: No such file or directory’, ‘usage: btrfs property get [-t ] []’, ‘’, ’ Gets a property from a btrfs object.’, ‘’, ’ If no name is specified, all properties for the given object are’, ’ printed.’, ’ A filesystem object can be a the filesystem itself, a subvolume,’, " an inode or a device. The ‘-t ’ option can be used to explicitly", ’ specify what type of object you meant. This is only needed when a’, ’ property could be set for more then one object type. Possible types’, ’ are s[ubvol], f[ilesystem], i[node] and d[evice].’, ‘’, ‘’]
[02/Dec/2021 15:45:33] ERROR [system.osi:174] non-zero code(125) returned by command: [’/usr/bin/docker’, ‘run’, ‘-d’, ‘–restart=unless-stopped’, ‘–name’, ‘embyserver’, ‘-v’, ‘/mnt2/EmbyMedia:/media’, ‘-v’, ‘/mnt2/EmbyConfig:/config’, ‘-v’, ‘/etc/localtime:/etc/localtime:ro’, ‘-p’, ‘8096:8096/tcp’, ‘-p’, ‘8920:8920/tcp’, ‘–net=host’, ‘-e’, ‘GID=100’, ‘-e’, ‘UID=1000’, ‘-e’, ‘GIDLIST=1001’, ‘emby/embyserver:latest’]. output: [’’] error: [‘docker: Error response from daemon: Conflict. The container name “/embyserver” is already in use by container “4659bd2636c98c0601a0c94ebab8a121a5891f0a46b78e9921fd25dd93562419”. You have to remove (or rename) that container to be able to reuse that name.’, “See ‘docker run --help’.”, ‘’]
[02/Dec/2021 15:45:33] ERROR [storageadmin.views.rockon_helpers:144] Error running a command. cmd = /usr/bin/docker run -d --restart=unless-stopped --name embyserver -v /mnt2/EmbyMedia:/media -v /mnt2/EmbyConfig:/config -v /etc/localtime:/etc/localtime:ro -p 8096:8096/tcp -p 8920:8920/tcp --net=host -e GID=100 -e UID=1000 -e GIDLIST=1001 emby/embyserver:latest. rc = 125. stdout = [’’]. stderr = [‘docker: Error response from daemon: Conflict. The container name “/embyserver” is already in use by container “4659bd2636c98c0601a0c94ebab8a121a5891f0a46b78e9921fd25dd93562419”. You have to remove (or rename) that container to be able to reuse that name.’, “See ‘docker run --help’.”, ‘’]
Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 141, in install
globals().get("%s_install" % rockon.name.lower(), generic_install)(rockon)
File “/opt/rockstor/src/rockstor/storageadmin/views/rockon_helpers.py”, line 279, in generic_install
run_command(cmd, log=True)
File “/opt/rockstor/src/rockstor/system/osi.py”, line 176, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/docker run -d --restart=unless-stopped --name embyserver -v /mnt2/EmbyMedia:/media -v /mnt2/EmbyConfig:/config -v /etc/localtime:/etc/localtime:ro -p 8096:8096/tcp -p 8920:8920/tcp --net=host -e GID=100 -e UID=1000 -e GIDLIST=1001 emby/embyserver:latest. rc = 125. stdout = [’’]. stderr = [‘docker: Error response from daemon: Conflict. The container name “/embyserver” is already in use by container “4659bd2636c98c0601a0c94ebab8a121a5891f0a46b78e9921fd25dd93562419”. You have to remove (or rename) that container to be able to reuse that name.’, “See ‘docker run --help’.”, ‘’]
[02/Dec/2021 15:46:38] ERROR [storageadmin.views.network:98] Unknown ctype: bridge config: {}