Failed to make Storage Sahare. RAID6 on btrfs -Pool creates perfectly but Disk Share won't

[phillxnet (Philip Guyton)]

@roberto0610 Hello again.

@roberto0610

Let use hardware raid even if people around this forum don’t recommend this…

This advice, given to you from me in the following forum thread:

@phillxnet

Note that putting hardware raid underneath btrfs in not desirable and will undermine btrfs’s ability to self repair. It can also cover up the type of harddrive failure I describe above. We have recently added this advice to our:

Minimum system requirements: Quick start — Rockstor documentation

Rockstor is a complete Linux distribution “Built on openSUSE” intended for direct hardware installation. Virtual Machine installs can work but are not recommended without full drive or preferably whole drive controller pass-through. Hardware raid underneath btrfs, Rockstor’s chosen filesystem, will weaken data integrity assurances. Raid controllers, if used, should thus be configured to HBA / JBOD operation.

And a recent btrfs mailing list quote from Zygo Blaxell 4 hours ago has:

• Use btrfs raid1 instead of hardware RAID1, i.e. expose each disk
  separately through the RAID interface to btrfs. This will enable btrfs
  to correct errors and isolate faults if one of your drives goes bad.
  You can also use iostat to see if one of the drives is running much
  slower than the other, which might be an early indication of failure
  (and it might be the only indication of failure you get, if your drive’s
  firmware doesn’t support SCTERC and hides failures).

btrfs with huge numbers of hardlinks is extremely slow

was, as you can see as it was referenced at the time, from the btrfs mailing list. And Zygo Blaxell as far as I know is not, at least yet, on this forum. The source is good and not hear-say or held by only those on this forum. It’s science :). Hardware raid doesn’t know which copy is correct, that might be important!

Your performance knowledge re hardare raid is not relevant to that of btrfs’s chunk based, rather than drive based raid. Take a look at for example:

Benchmarks Of Btrfs RAID On Four Samsung 970 EVO NVMe SSDs (Linux 4.18 + MSI...

for comparisons of the raid performance levels in btrfs. It’s not neccessarily what you might think. That article is a little old but I’m pretty sure there was a newer on form the same source. I’ll post when I get the time to find it but others can do likewise as it would be good to build up some references actually. As I explain in my support email, btrfs raid borrows from hardware raid concepts but is not the same; one can’t always draw direct parallels. Putting hardware raid under btrfs will void btrfs’s ability to correct data and basically also void guarantees of returning what was requested to be saved. It’s why it was invented.

@roberto0610

-RAID10 really good too, Parity and striping “Super”

There is no parity in btrfs raid 10. Only btrfs 5 & 6 use parity.

The documentation links I gave in my support email to you, that you requested of me, then you will have the canonical references for this info.

@roberto0610

But for certain scenarios it really good, like in multimedia creation aplicación. When most of the demand is in reading thousands of times the same video data or photos. That gets written into the drives only one time but it’s been read many, many times.

For this you want lots of RAM, irrespective of the underlying file system. Or even better a bcache write throught (so only caches reads as much safer) using one or more ssd / nvem devices. I.e. see the following technical wiki entry on how we implemented in in our older CentOS base:

Bcache - developers notes

This document represents the current and pending state of bcache in Rockstor and is not an endorsement of it’s use or support status but is simply intended to share the progress and development made to date. It is intended as a developers document so any changes made should be in accordance with keeping this text up to date with the current code state, including in the case pending pr’s. If you wish to comment on bcache support in general, as opposed to the specifics of what is actually in play…

It would be nice to have this document updated as the version of bcache in our new OS base may well need only the udev role to work.

There are a whole host of technologies to assist in performance, but very few that do what btrfs does. That is because it’s a difficult task that has been attempted, and failed, many times. Hence the caution taken with the btrfs parity raid levels. I’m keen to follow upstream on this as I’m not the btrfs expert; they are.

Lets take care to keep an open mind on the knowledge and diligence of others here on the forum. We are all trying to help one another out here. You do find yourself in a tricky stop as you presumably required a single large pool that requires many drives. That a sticky point when btrfs can currently only offer a single drive failure. But with local fast backups the risk can be reduced. Or you opt for using the approach I referenced earlier of employing a newer stable kernel. All food for thought.

Apologies if this communication has an unintended tone. I’m just trying to advise as best as I see things from my own limited knowledge of all relevant things. And I’m not feeling:

@roberto0610

… Nice there is no Support for RAID5 or 6 currently…

this tone to be appropriate. This was an upstream move from the experts at the time. Neither you nor I are btrfs experts so we kind of have to go with the flow if we are to take advantage of what is passed down to us by way of our only COW file system option that can be included within the kernel and has the block pointer re-write capability.

Hope that helps and let us know how your adventures go. And you might want to follow up on some of the software vs hardware raid info out there on the internet so that you are informed on the balance of risk. It may be that you can find a way to mitigate the second drive failure during repair risk. I.e. if a single disk fails and you are say using btrfs raid 10, (4 disk minimum) and you have 6 disks, then the failed drive can be removed from the pool via Resize/ReRaid fairly quickly while you are still using the pool. The resulting pool will still have minimum +1 drives left. Another drive can then be added on-line without the data going off-line. Assuming you hardware supports hot plugging.

Anyway all foof for thought and many things to consider.

Thanks again for you long term support and interest by the way. You may well be interested in our GlusterFS support that I hope to get to once we have all our technical dept sorted in the next year. Another option for providing fail-over when things go down in the bad way :). And again entirely doable via command line today: on top of what Rockstor currently enables by way of magic made easier.