Hello, I have a new setup and have 6 x 14tb drives (SATA 6gb/s) I’d like to run in a dual-parity pool (was planning RAID 6 before I saw the notice and looked deeper). I see the notice about RAID 5/6 being unstable so of course I don’t want to play with fire. I see there is a raid6-1c4 level. Reading the documentation it speaks about metadata in c3 or preferably c4 - does this mean that you can use RAID6 for the data and RAID1c4 for the metadata and “all is good”? I am looking for two drive redundancy, which should be around 52TB raw on normal RAID6 or ZFS type (usable around 50TB since each drive is only 12.7TB mounted). I do have an extra 1tb SSD (not part of planned pool) that I can use for cache if that’s an option. When testing I didn’t play with this stuff because I didn’t have the actual system setup with all the drives. Now they’re in, I’m learning a bit more.
What can I do? What do you guys suggest? Reading the BTRFS site (mkfs.btrfs(8) — BTRFS documentation) and it still indicates RAID56 is not good, but doesn’t mention anything about raid6-1c4. I’m not opposed to going to latest kernel. I’m running Leap 15.6 with kernel 6.4.0 (default) right now.
I’ve used RAID5 before without any issues, but then moved over to the RAID10c3 for convenience. The scrubbing took way too long, and because I was concerned about potential impacts of the write-hole I felt the need to run scrubs maybe more often than usual (you also should read up on how the scrub be executed by device to avoid further delays, which I didn’t know at the time).
On the other hand, a lot depends on your use case, too. If, e.g. the majority of the time the system used to server up files, rather than have data written to it, then the risk of data loss due to unclean shutdowns can be conceivably lower. Of course, having an UPS connected further reduces that risk (but that’s an additional expense).
In the 8 or 9 years I’ve been running Rockstor, I have had only 1 drive outright fail, a couple started to show issues and were replaced. Only recently had I one quickly going out and a second one (from the same production month) starting to show some issues. Again, the risks of dual failure will (not exclusively) be a function of your use case.
And I would be remiss not to state that any RAID config is not a backup, so I assume you’ve put some thought into that on how to protect your data in addition to running it on Rockstor.
Here are some resources that discuss some more in-depth challenges (and some workaround) on RAID5/6 (granted they’re a couple of years old, so might not be all valid anymore):
In any case, if you’re going to do it, I’d say raid6 for data and raid1c4 for metadata, since you can separate the data and metadata. That likely gives you the best compromise of risk vs. space utilization.
@phillxnet recommended this in another thread some time ago as well:
and it’s something I’ve been contemplating at some point given my use case of mostly read-from storage, but not got around to investing in.
My Rockstor is initially planned as my backup server for my Synology NAS. My NAS is 56TB (not full) and my current backup system is two different places. 1/2 of my media and all of my data / dockers / photo’s go to an external USB attached directly to my NAS, and the other half of my media and a second copy of all my data / dockers / photo’s go to another computer running Duplicati. I do not have 56TB of backup capacity and with versioning I’m running out of space so my plan is to keep my existing backup schedule and have my Rockstor as a full LIVE copy (not in a backup file I need software to parse but a file / folder copy available online). I am not 100% sure the best way to do it, and honestly since my new machine has newer drives and more storage I may make it the “primary” server and use my Synology for backup once I’m comfortable with things.
I do have a UPS and will be using it to shutdown my machine in the event of a power failure. Sounds like RAID1 or 10 are best with BTRFS but man I don’t want to eat up 50% of my drives, I was willing to do 33% for 2-drive redundancy.
I’m curious what Synology does with SHR-1 and SHR-2. They use BTRFS and have equivalent to RAID5 / RAID6 that is reliable. Is that similar to the RAID5-1c3 and RAID6-1c3? Does anyone even know?
Lastly, you mention speed. I found a few post that talk about it. What kind of speed are we talking about? Is it really that bad? There’s only a couple of us using it, biggest use will be Plex media server maybe a couple instances streaming files. Will that be a problem? Will I be able to saturate the network or is it that slow?
I was referring to scrubs taking a long time (when I had RAID5 five or so years ago). The read speed for streaming 4K movies and the likes from Plex is plenty, even multi-stream, especially if the streaming happens hard-wired.
Since you have the UPS I think you can probably manage the risk with RAID6-1c4. And if you set up your smart warnings (or check frequently enough) the risk of losing one or two disks is certainly low. My last ones have lasted over 5 year with the server running 24/7 (though had disk idling enabled via the WebUI).
On the SHR-1/2 my understanding is that you still have the option to use ext4 or btrfs, so I would assume, it might be similar but not quite the same (aka proprietary on how they do the striping/redundancy setup).
Does the RAID6-1c4 have the same “not stable” caveat as the plain RAID6? That’s the only reason I’m even questioning it because of that warning but it’s not clear if it applies to the -1c3/-1c4 variants.
It’s a moot point, I already made the pool but I haven’t started copying data over so remaking it is simple. That’s another whole task I have, the best way to keep a replicated copy of my Synology NAS on here.
I’m pretty sure that Synology do not use btrfs for their raid arrangements. But are instead using the older technology of mdraid. They then put btrfs over the top of this. However this would need confirming.
The “raid6-1c4” is our short-hand for data-metadata raid levels. In upstream speak this would be, as you have surmised I realise:
btrfs-raid6 data with btrfs-raid1c4 metadata
I chose to shorten this to our data-metadata reference used within the Web-UI.
We document this in our Pools subsection Btrtfs mixed raid profiles here:
And note that raid1c3 is 3 copies on 3 different drives, rather than the vanilla raid1 2 copies.
With raid1c4 referencing 4 copies on 4 different drives.
Where btrfs-raid1, the base of both these extra copies variants is generally well regarded reliability wise.
Not necessarily, where-as mdraid cannot change it’s raid level without a re-format btrfs-raid can change raid profile, data, metadata, or both, while still serving up the data. That’s the magic/flexibility we get by using btrfs as our drive management also. Plus when there is a non-btrfs raid mechanism under a btrfs volume (Pool), there is reduced data security against corruption. Btrfs checksums everything, mdraid does not. Ergo mdraid will return either copy stored, btrfs-raid will only return a good checksummed copy. Or non at all - to avoid returning what was not saved. Not nocking mdraid - it’s awsome, but btrfs raid is the next generation of software raid. And living within the filesystem itself, rather than under it, enables far more flexibility.
@Hooverdan Thanks, yes that last article from synology themselves indicates the use of at least:
/proc/mdstat
and
mdadmin status
so definitely mdraid. And if they are also using LVM (I’d forgotten about that) and we can assume partitions they have:
partitions
– mdraid
— LVM
---- btrfs
Or some other arrangement of the same layers, as LVM has some multi-drive capability also. But the second article you reference confirms at least the above layered arrangement. And note that LVM has itself some layers within it:
We have, as we strongly favour whole drive (no partitions) use:
btrfs
Avoiding technology layers and interfaces between them - was a primary directive of sorts. It also meant we had only to deal with btrfs - which has presented its challenges on its own .
Curious. I’ve got a chart somewhere I created of the various partition schemes also: lets just say they don’t all agree on things. Ergo no partitions and whole drive as our default.
Cheers - brougth back some memories here: I once used LVM multi-dive capability on a desktop to use two ssd’s as an interlaced single drive for performance + space (increased risk). It was fast but complicated.
Thanks for the education. I’m glad to read that the SHR1/SHR2 is generally considered solid since that is my “truth” for all my data. Home base (for now) is the Synology and all my backups are scattered on external drives and a Windows 11 workstation with a couple 12tb drives and Duplicati My goal is to eventually get comfortable with this new machine and make it the “truth” and the Synology a backup.
.
My goal is to eventually get comfortable with this new machine and make it the “truth” and the Synology a backup.