Can't even mount with degraded,ro

qubes · July 8, 2022, 3:32pm

Rockstor 4.1.0.0 (Linux: 5.3.18-150300.59.76-default)
Extra mount options: [degraded,ro] for the only pool I have called “poolname”, but even after page refresh as well as reboot the share is still unmounted (Active mount options / Status: unmounted).
If I try Balances I get:
Error running a command. cmd = /usr/bin/mount /dev/disk/by-label/poolname /mnt2/poolname -o degraded,ro,compress=no. rc = 32. stdout = [’’]. stderr = [‘mount: /mnt2/poolname: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error.’, ‘’]

sde is a normal working disk. sdb was corrupted so I’ve got one “detached” in the list of disks. Another disk (which is still visible under the disks menu) suddenly disappeared from the disk view unter “poolname”.
I am not trying to recover any data - just wanting to remove and add disks again, but it seems I need to be able to mount the pool first.

Thanks for any help

phillxnet · July 8, 2022, 7:07pm

@qubes Welcome to the Rockstor community.

Could you share some more info on the raid level used. As you mention potentially loosing 2 disks:

and:

The following command output, run as the ‘root’ user, may be able to given folks here more of an idea of you systems status.

btrfs fi show

which will also hopefully help here.

Only the less production ready btrfs raid 6 can handle a 2 disk failure per pool.

Yes, a filesystem can not generally be altered (meaningfully) unless it is mounted. And yes the degraded option is the key here. You may also want to try a reboot (although you mention having done this): in such situations as this it can sometimes free up the mount by label that can get confused and we use this, but we also try to fail over to by dev mounts.

Take a look at the following newly rewritten doc section. It may have some helpers.

“Data Loss-prevention and Recovery in Rockstor”:
https://rockstor.com/docs/data_loss.html

More prep than repair orientated but may have some useful info in there.

The normal ‘method’ for disk removal ‘missing’ or otherwise, is via our ReSize/ReRaid. But again a mount is required. Also note our:
“Redundancy profiles”: doc section
https://rockstor.com/docs/interface/storage/pools-btrfs.html#redundancyprofiles
in case this is a too many failed devices thing.

Lets see that command output and know a little more of your setup before suggesting anything more drastic; like for example:
“Installing the Stable Kernel Backport”: https://rockstor.com/docs/howtos/stable_kernel_backport.html

Hope that helps.

qubes · July 11, 2022, 11:44am

Thank you for the reply. I’m sorry I forgot to mention that I’ve used six disks in Single profile for the pool.
As I said one disk had a failure and got lost, but a second one is missing in the pool even though the disk is listed under “disks”.
I tried to refresh the SMART details without success:

Error running a command. cmd = /usr/sbin/smartctl -c /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAH01E9L. rc = 4. stdout = [‘smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.3.18-150300.59.76-default] (SUSE RPM)’, ‘Copyright © 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org’, ‘’, ‘Read SMART Data failed: scsi error badly formed scsi parameters’, ‘’, ‘=== START OF READ SMART DATA SECTION ===’, ‘’]. stderr = [’’]

Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/disk_smart.py”, line 161, in post
return self._info(disk)
File “/opt/rockstor/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py”, line 145, in inner
return func(*args, **kwargs)
File “/opt/rockstor/src/rockstor/storageadmin/views/disk_smart.py”, line 76, in _info
cap = capabilities(disk.name, disk.smart_options)
File “/opt/rockstor/src/rockstor/system/smart.py”, line 144, in capabilities
o, e, rc = run_command([SMART, “-c”] + get_dev_options(device, custom_options))
File “/opt/rockstor/src/rockstor/system/osi.py”, line 224, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/sbin/smartctl -c /dev/disk/by-id/ata-WDC_WD80EFAX-68KNBN0_VAH01E9L. rc = 4. stdout = [‘smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.3.18-150300.59.76-default] (SUSE RPM)’, ‘Copyright © 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org’, ‘’, ‘Read SMART Data failed: scsi error badly formed scsi parameters’, ‘’, ‘=== START OF READ SMART DATA SECTION ===’, ‘’]. stderr = [’’]

Here is the output of btrfs fi show:

btrfs fi show
Label: ‘ROOT’ uuid: 4ac51b0f-afeb-4946-aad1-975a2a26c941
Total devices 1 FS bytes used 8.08GiB
devid 1 size 12.85GiB used 12.80GiB path /dev/sdc4

warning, device 6 is missing
warning, device 6 is missing
bad tree block 22069248, bytenr mismatch, want=22069248, have=0
ERROR: cannot read chunk root
Label: ‘poolname’ uuid: 41335e82-9f2e-47a0-bb2d-e28c8280f636
Total devices 6 FS bytes used 28.61TiB
devid 1 size 5.46TiB used 4.53TiB path /dev/sdf
devid 2 size 5.46TiB used 4.52TiB path /dev/sda
devid 3 size 5.46TiB used 4.54TiB path /dev/sde
devid 5 size 5.46TiB used 4.53TiB path /dev/sdg
*** Some devices missing

Is there a way to forcefully delete a pool and start all over without having to mount the pool?

phillxnet · July 11, 2022, 3:42pm

@qubes Thanks for the additional info:
Re:

I’m afraid that the single profile has no ability to loose any members at all. See:
https://rockstor.com/docs/interface/storage/pools-btrfs.html#redundancyprofiles

"
Single: This profile offers no redundancy.
A single disk failure will result in the entire Pool being lost. It is the only valid option for creating a Pool with a single disk drive. It is also recommended if you have multiple disks of very different sizes, yielding higher total capacity compared to Raid0 in this setting.
"

Essentially single is all about getting maximum space. All members are strung together with zero data redundancy, there is a dup metadata but that is just duplication of metadata, not like in raid1 where that duplication is across devices.

In short yes. See below.

I’ve approximated your situation here by removing one of 3 virtio disks that formed a btrfs-single pool in a KVM instance. And then rebooted. The pool obviously failed to mount given it’s now broken status.

Unfortunately we focus more on the redundancy profiles which are far more popular and so still offer up advise in the pool details page Maintenance required section - likely what has mislead you in your experience here (my apologies). I’ve created the following issue to note this advice bug within the Web-UI:

github.com/rockstor/rockstor-core

Improve 'Maintenance required' advice for btrfs-single/btrfs-raid0

opened 01:11PM - 11 Jul 22 UTC

closed 02:13PM - 03 Mar 23 UTC

phillxnet

In a btrfs-single degraded scenario (missing member) we still offer the generic …advice of: > **Maintenance required** > Missing disk removal requires degraded,rw mount options. Pool is currently (unmounted). > Consider degraded,ro to refresh backups first. > Reload page to refresh active mount options. in our Pool details page. However btrfs-raid single (like btrfs-raid0) has no redundancy capability. If a disk is missing (detached in Rockstor speak) the only recourse is to either reattach the missing device and scrub the pool, or wipe, recreate, and restore from backup. As per our doc section: "Data Recovery: btrfs-raid0 Pools": https://rockstor.com/docs/data_loss.html#data-recovery-btrfs-raid0-pools (N.B we are missing a sister btrfs-single doc section - issue to follow). It is proposed that we contextually change the above **Maintenance required** advice to be wise to this brtrfs-single/btrfs-raid0 lack of redundancy. We might also link to the recently updated Data Recovery doc section.

And given btrfs-single/btrfs-raid0 offer no degraded mount capability given their zero tolerance for missing members we end up being in a corner in their cases. I’ve also crated the following issue as a result of this interaction:

github.com/rockstor/rockstor-core

Allow delete of un-mounted/un-mountable pool

opened 01:24PM - 11 Jul 22 UTC

phillxnet

Thanks to forum member qubes for helping to highlight this usability short-fall.… In certain instances such as when a non redundant pool has a missing member, the recovery scenario is to delete the pool and start over from backups; see: "Data Recovery: btrfs-raid0 and btrfs-single Pools": https://rockstor.com/docs/data_loss.html#data-recovery-btrfs-raid0-and-btrfs-single-pools However we currently fail on the Web-UI side by paradoxically dis-allowing this action due to the pool not being mounted initially!! > Error running a command. cmd = /usr/bin/umount -l /mnt2/test-pool-balance. rc = 32. stdout = ['']. stderr = ['umount: /mnt2/test-pool-balance: not mounted.', ''] > Traceback (most recent call last): > File "/opt/rockstor-dev/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception > yield > File "/opt/rockstor-dev/src/rockstor/storageadmin/views/pool.py", line 800, in delete > umount_root(pool_path) > File "/opt/rockstor-dev/src/rockstor/fs/btrfs.py", line 600, in umount_root > raise ce > CommandException: Error running a command. cmd = /usr/bin/umount -l /mnt2/test-pool-balance. rc = 32. stdout = ['']. stderr = ['umount: /mnt2/test-pool-balance: not mounted.', ''] This is a poor user experience and we already warn of all data shares etc being removed and should thus be able to proceed as a user might expect and allow for the pool delete in this case. To avoid the paradoxical mount requirement prior to pool delete (unmount). The reasoning here is to first wipe all contents but for a broken pool that is obviously not possible.

So to your question on how to start-over:

Shutdown.
Detach all pool members for the un-mountable/defunct pool.
Power-on
In the associated Pool’s details page select “Resize/ReRaid Pool”
Select “Remove disks” and tick one of the detached disks.

N.B. if you were to select more than one detached disk you would receive the following message
“Detached disk selection is limited to a single device. If all Pool members are detached all will be removed and their pool automatically deleted there after.”

Note you may have to browser-refresh the Pools page afterwards for the Pool delete to take-hold in the Web-UI.
Delete the detached prior pool members - Not strictly necessary if you are going to re-attach them.
Power down. (Assuming you don’t have eSATA capability, i.e. live plugging hardware)
Reattach the disks if they are to be re-used. (If done live then press “Rescan” on disk page)
Power-on and wipe the now defunct prior pools members via the disks overview page.

Warning: if you live plug hardware that is not compatible, you will likely irreparably damage it.

As you see this is rather a corner case for us, we tend to focus on the redundant profiles. But doable. And with the later issue addressed this would be far more intuitive/easy.

Hope that helps.

qubes · July 12, 2022, 10:02am

This solved my problem! Thank you so much for your support.

I can completely understand that most users choose a RAID system with redundancy and I’ve been running Rockstor 3.x for years with RAID10. But now I’ve got three NAS Servers (1x OpenMediaVault because of an ARM Board using RAID5, 2x Rockstor 4.1 without redundancy) and they all sync their files. That way the data is pretty safe, but I did not waste much space for high availablity.

I’m sure your github issues will cover any similar problems in the future. Like having a text “the pool is corrupted and cannot be repaired. Click here to remove all disks and delete the pool.” or simply adding your step-by-step guide to the docs.

Thanks again for your help and all of your work in general.