Btrfs Raid 5/6 warning

phillxnet · July 6, 2016, 11:48am

@FREE_NORWAY Welcome to the Rockstor community. Yes there have been a number or concerns of the production readiness of btrfs raid 5/6 and also a number of discussions on this forum where a few participant have expressed a degree of satisfaction with these parity variants of btrfs; bar the performance issues with scrubbing of course which have been a long standing issue. And prior to the indicated thread the recent communications re raid 5/6 on the linux-btrfs mailing list were edging towards it being considered stable; this is evident in the changes that happened on the btrfs wiki entry on the 24th June as a result of that thread, in brief:

the subtraction of "...should be usable for most purposes..."

https://btrfs.wiki.kernel.org/index.php?title=RAID56&diff=next&oldid=29769

and the addition of:

“The first two of these problems mean that the parity RAID code is not suitable for any system which might encounter unplanned shutdowns (power failure, kernel lock-up), and it should not be considered production-ready.”

https://btrfs.wiki.kernel.org/index.php?title=RAID56&curid=5631&diff=30429&oldid=30425

and:

“It is recommended that parity RAID be used ‘’‘only’‘’ for testing purposes.”

We also have the addition on the main page relating to Parity btrfs which relates to raid5/6 variants:

** Single and Dual Parity implementations (experimental, not production-ready)
with the bold bit added on the same day as the above changes.

https://btrfs.wiki.kernel.org/index.php?title=Main_Page&curid=1&diff=30427&oldid=30343

I’m afraid I dropped the ball on this one as I had these changes lined up ready to post in one of our ongoing forum threads on btrfs raid 5/6 but alas failed to do so in a timely manner by currently running a few days late. There are now a couple of issues open to address this within the docs and the Web-UI that I hope to make some progress on shortly.

Thanks for starting a newer and fresher thread more specifically on this recent issue and do please consider others experience and statements on these parity levels in the linked forum thread, and as @KarstenV kindly points out in this thread the general more cautious recommendation (ie for production) is to not use the btrfs parity variants of raid 5/6.

That thread has in turn been linked to from a number of other less focused threads.

As it goes the Rockstor docs did recently receive and merge and rapidly published a generous and very frank addition relating to bugs in btrfs raid 5/6 and their impact on the raid 5/6 recovery process by @Fantastitech via issue:

github.com/rockstor/rockstor-doc

Raid5/6 recovery documentation is incorrect and/or out of date

opened 07:36PM - 18 Jun 16 UTC

closed 02:16PM - 22 Jun 16 UTC

Fantastitech

I've just finished a couple of pretty major recovery operations and I noticed, a…fter much research that the documentation for data recovery on RAID5/6 arrays had inaccuracies, out of date information, and unnecessary redundancies. ## Mount command Step 1 and step 12 instruct to mount the pool with: ``` # mount /dev/sda /mnt2/mypool ``` However BTRFS will not mount an array with a missing, failed, or otherwise unmountable drive in a RAID5/6 array without passing the `degraded` mount option. A user with an unmountable drive would not be able to get past step 1 and step 2 does not make any sense in context as it could be run before step 1 to determine if a device is missing or not in which case step 1 would not work. ## Double rebalancing Step 3 says to delete the failed device, which in a RAID5/6 array will trigger a rebalance to rewrite the parity information for the new amount of drives. In the case of this document, that brings the array down to 3 drives. In step 13 and 14 the user is instructed to add the new device then manually trigger a second rebalance. This puts additional wear on the drives, increases the chance of catastrophic failure, doubles the length of the entire procedure, and if there is not enough space on the array to accommodate the new parity data it will fail. A much more practical procedure would be to add the new drive to the pool, then delete the failed/missing device. Since failed devices can cause kernel panics, the most sane solution is to remove the failed device from the system completely, mount degraded, add the new device, then delete missing devices which will trigger a single rebalance rebuilding parity information with the new drive. ## Redundant RAID5 and RAID6 sections. There is a section for RAID5 data recovery, then a second section for RAID6 which just describes that it's the same procedure as RAID6. It would be much more clear, clean, and concise to include information about RAID5 and RAID6 in the same section. --- I've rewritten the RAID5/6 recovery documentation to correct these issues, clarify some details, and added an extra troubleshooting step for unmountable arrays in the event of superblock issues that prevent a mount command from finding existing drives.

where a couple of known bugs in raid5/6 can hamper the recovery process.

It can be very disconcerting looking as closely as the developers do at the underlying technologies of many modern projects, given their complexity, and Rockstor is as all non trivial technologies are, dependant on very many projects doing the same with their technologies. It is also worth noting that even ext4 has regular bugs found but they are often very much corner cases, and this is what btrfs is definitely moving towards.

Thanks for the juggling prompt here; this thread should be updated with progress on the referenced issues.

Hope that helps.