Unable to mount degraded pool with rw flag

miversen33 · June 15, 2024, 4:31pm

Brief description of the problem

A handful of days ago I noticed that nfs would occasionally lock up on my mounting client. After digging into this, I found out that one of my drives in the pool supplying the nfs share was bad. I attempted to remove the drive from the pool and after some time the remove will fail with an “Input/Output” error.

At this point I began researching options on how to approach this. The pool is a raid5 pool and as I understand it, BTRFS currently does not support removing a degraded disk from raid5. As such, I powered down the machine and manually pulled the disk from it. Upon restart, the pool was unmounted as it recognized it was missing a disk. I was prompted to add mount options of either degraded,ro or degraded,rw in order to remount the pool.

Mistakenly I entered degraded,ro and now I am unable to rectify this. Without making the pool rw, I cannot have it remove the missing drive and rebalance itself. I am stuck currently.

When I try to change the mount options via the UI, I get the follow error

I don’t really know what to do here. I am a bit of a novice at BTRFS (though I have plenty of linux experience in general). I figured I could unfuck this via the cli but I am struggling to figure out how to execute this mount myself via cli. A bit of guidance would be fantastic

Error Traceback provided on the Web-UI


            NoneType: None

phillxnet · June 15, 2024, 5:34pm

@miversen33 Welcome to the Rockstor community forum.
Re:

What happens when you try to change the custom mount options to degraded,rw?

If a pool has a missing disk, as you explain yours does, it will refuse to mount without a degraded option. That is normal. But what may have happened here is a partial removal during:

Btrfs does input/output error if it can’t find a ‘good’ copy: or it may be reporting an actual drive input-output error. Either way it’s now poorly. And in which case you may have more limited options. If degraded,ro mount works; it gives you a change to refresh your backups if need be.

Also of relevance here is the age of your system. What version of Rockstor are your running, and what base OS is that version using: top right of Web-UI should tell you all these things.

That would be a degraded pool, and the parity btrfs-raids of 5&6 do allow for drive removal. But are known to have less robust repair capabilities. Hence me asking about the base OS and rockstor version your are running. Leap 15.6 carries, by default, newer kernel and user land for btrfs than our prior Leap bases. And can do parity raid rw by default. But if you are already using say TW or stable kernel backports then you already have the newer btrfs capabilities.

You best option, if you can get degraded,ro mount is to move the data off the flaky pool, and then recreate a pool with know good drives and restore. As per our prior warnings against the parity based btrfs-raid levels they are weaker in repair scenarios. But at least our upstream (openSUSE) have now removed their (and consequently our) default ro (default rw disabled) capability. So that bodes well for improved confidence in the parity raids. But if you end up re-doing, consider using one of the now available mixed raid so metadata is not using parity raid, that way you get the space advantage of the parity raid level for data, but have a more robust raid1c3 or raid1c4 for medadata. We have a slightly outdated doc entry at the end of the following for this:

https://rockstor.com/docs/howtos/stable_kernel_backport.html

We do now support within the Web-UI, in more recent versions, mixed raid levels.

Also, for some CLI hints as to repair options, see our:

https://rockstor.com/docs/data_loss.html

And it’s always worth doing a reboot before attempting to change custom mount options. That is what the message you received was attempting to avoid re stopping and starting services: a reboot does the same. Sometimes btrfs mount options can be sticky, and only free-up over an unmount rescan and mount.

Hope that helps, any my apologies for now having more time to detail your optinos. But those doc references should help, and first-things-first you must use what you have already (degraded,ro) if that is working, to get data off if need be. Otherwise things could be worse. The parity btrfs-raid levels were a little flaky and your pool is probably form that time. So a freshly recreated pool will inherit the newer improvements if you first have newer OS in place before creating the pool again: if that is what you end up doing.

miversen33 · June 15, 2024, 6:22pm

What happens when you try to change the custom mount options to degraded,rw?

The error I screenshotted is what I see when I try to change to degraded,rw

Also of relevance here is the age of your system. What version of Rockstor are your running, and what base OS is that version using: top right of Web-UI should tell you all these things.

I am running version 5.0.9-0 currently.

That would be a degraded pool, and the parity btrfs-raids of 5&6 do allow for drive removal. But are known to have less robust repair capabilities. Hence me asking about the base OS and rockstor version your are running. Leap 15.6 carries, by default, newer kernel and user land for btrfs than our prior Leap bases. And can do parity raid rw by default. But if you are already using say TW or stable kernel backports then you already have the newer btrfs capabilities.

I have not modified the system in anyway since install so currently I am running whatever the base OpenSUSE version is that comes with Rockstor. From the looks of it, I am running Leap 15.6 with kernel 6.4.0-150600.21-default

You best option, if you can get degraded,ro mount is to move the data off the flaky pool, and then recreate a pool with know good drives and restore. As per our prior warnings against the parity based btrfs-raid levels they are weaker in repair scenarios. But at least our upstream (openSUSE) have now removed their (and consequently our) default ro (default rw disabled) capability. So that bodes well for improved confidence in the parity raids. But if you end up re-doing, consider using one of the now available mixed raid so metadata is not using parity raid, that way you get the space advantage of the parity raid level for data, but have a more robust raid1c3 or raid1c4 for medadata. We have a slightly outdated doc entry at the end of the following for this:

I’m honestly not concerned about the data on here, I just need stability and speed which is why I accepted running Raid5. The data will populate again if it is removed, but it cannot do that if NFS randomly drops off (in this case likely due to a dying drive). I haven’t played with the Mixed Raid stuff yet, I am interested so I might spend some time next week doing so.

I did attempt a reboot as the suggestion states but because Rockstor is unable to modify the mount options now, rebooting simply put me back in degraded,ro state.

No need to apologize though! I appreciate the help

miversen33 · June 15, 2024, 6:31pm

A quick update since I had a thought.

I am able to remove the custom mount options. So I am not stuck in RO mode, I simply cannot get it to enter RW mode so I can remove the missing disk.

Edit: Success! By simply removing the ro setting, restarting rockstor (systemctl stop rockstor, umount $POOL, systemctl start rockstor) and then refreshing the page a few times, I am able to see that it is mounted in rw mode now and I can remove the missing disk

miversen33 · June 15, 2024, 6:47pm

Well another update, not so much success lol. When I ran the disk removal it failed with seemingly meaningless Nonetype: None error

When I tried to remove the disk manually with btrfs device remove missing /mnt2/mediapool, it is quite for a bit and then drops the following error

ERROR: error removing device ‘missing’: Input/output error

I’m at a lose, how do I get the pool to accept that this disk no longer exists and to rebalance itself?

phillxnet · June 16, 2024, 7:53am

@miversen33 Thanks for the updates.
Re:

You have likely encountered the type of failure that has lead to pool to no longer be able to repair itself. That has been a long standing weakness in the parity btrfs-raid levels.

Ok, so that simplifies things. For speed you are better off with a btrfs-raid1 or btrfs-raid10 arrangement, but then available space takes quite the hit. Consider starting over, with known good drives, and memory (see below), with say a mixed raid level of raid5-1 or raid5-1c3. This way you get the space use of advantage of a parity raid, but the maturity (for repair & stability) of mirroring for the metadata; with the latter side stepping a known weakness of the btrfs-raid parity levels.

Any use of btrfs-raid parity levels is not currently associated with stability I’m afraid. Our Upstream (openSUSE)) has recently in 15.6 removed their ro default - imposed to encourage folk away from it’s use at that time, but it is still less mature than the other raid levels and does still have less robustness. Hence the suggestion to use a mixed raid setup as having the metadata use raid1 (raid5-1) or raid1c3 (raid5-1c3) helps regarding robustness - and speed most likely. Or go all in on say btrfs-raid10 where you get speed and robustness but at the cost of space.

The raid5-1 references are are own terms for, in this case, raid5 data with raid1 metadata. Btrfs can have different raid levels for data and metadata and we have selected a subset that make sense generally regarding drive numbers re minimum drive count (min_dev_count) & maximum missing (max_dev_missing) capabilities.

“and memory”

Be sure to check your memory, given your experience here. Bad memory will fail btrfs’s checksums as they are first calculated in memory and then committed to disk. If memory corrupts them first they are toast and future retrievals of that same chunk of data or metadata will fail with input/output as the supposedly known good checksum was actually corrupt. Take a look at our following howto:

Pre-Install Best Practice (PBP) — Rockstor documentation

As bad memory will always undermine filesystem robustness, and can lead to the experience you have reported here. I.e. Input/output error as no known-good data/metadata (according to the checksum) can be returned. So none is returned and an Input/output error can result: a key part in your report here. We have the sticky mount nature, which you were able to manage via the unmount etc, but from the initial report you have had this Input/output error.

In short - I suspect the Input/output error is blocking repair, device remove missing in this case, as it cannot ensure data integrity, and so cannot proceed. And likely reverts to ro again to protect the remaining data. Given your stated stance on the data - just try one of the suggested mixed raid profiles on a fresh pool: but only after a thorough memory/RAM check.

Re:

Exactly as you have attempted (initially): but we are snookered by the pool being unable to return know-good data: hence the memory check to chase down why this might have happened. Or you may just have a 2 drive failure when raid5 has only a single drive failure capability. I.e. a remaining copy of data/metadata is also corrupted by disk (or via the ram). Btrfs’s main remit is to return good data, or nothing at all (Input/output error).

Hope that helps, at least for some context. And do check your memory as the parity raid levels are much better these days: especially given your use of the recommended Leap 15.6 (newer kernel) base OS. But they are not of the same quality, and do not fully (yet) adhere to the hard-line COW, of say the non parity (and redundant) btrfs-raid levels. But this can be partially side stepped by not using a parity raid for metadata (with consequent performance improvements).