Raid Speed Tests For Beginners et al... 8-21-24

First of all, the reason I like BTRFS so much is because it uses the Copy-On-Write technique rather than a journaling file system:

Best Explanation Quote From COW vs. Ext4

"When data is overwritten in an ext4 filesystem, the new data is written on top of the existing data on the storage device, destroying the old copy. Btrfs, instead, will move overwritten blocks elsewhere in the filesystem and write the new data there, leaving the older copy of the data in place.

The COW mode of operation brings some significant advantages. Since old data is not overwritten, recovery from crashes and power failures should be more straightforward; if a transaction has not completed, the previous state of the data (and metadata) will be where it always was. So, among other things, a COW filesystem does not need to implement a separate journal to provide crash resistance."

The test setup consists of a Winderz 11 5950X setup with NVME M.2 drives thru a 10Gbps Fiber Optic LAN to a Rockstor test setup with 16G ECC RAM and 3.3GHz CPU and 8-port HBA with 4x 4TB drives attached. Nominal usable drive space is 3.6TB for each drive.

There is a large “TEST” directory on one NVME drive 117GB large containing about 50% very large video files, 25% medium sized files and 25% small files.

The REASON for these tests is to help beginners figure out which NAS setup and Raid configuration may best suit their purposes. By looking at the graphs, one can guess what a small file transfer either way will do whilst also seeing what a huge file transfer would do (such as in a huge backup). The NVME drive and fast LAN allow the Rockstor test setup to run at as fast as it can. Indeed, in many cases, it saturates the 10Gbps LAN over 950MB/Sec in transfer speeds for a brief time.

I’ve attempted to organize this as easily as possible realizing a spreadsheet would probably work better.

****************** ALL TESTS BELOW USE 4x 4TB HDD’s *****************************************

SINGLE
Usable File Space: 14.55TB

Write- Single
Read- SingleR

SINGLE-DUP
Usable File Space: 14.55TB

Write- SingleDup
Read- SingleDupR

Raid-0
Usable File Space: 14.55TB

Write- Raid0
Read- Raid0R

Raid-1
Usable File Space: 7.28TB

Write- Raid1
Read- Raid1R

Raid-1C3
Usable File Space: 4.85TB

Write- Raid1C3
Read- Raid1C3R

Raid-1C4
Usable File Space: 3.64TB

Write- Raid1C4
Read- Raid1C4R

Raid-10
Usable File Space: 7.28TB

Write- Raid10
Read- Raid10R

Raid1-1C3
Usable File Space: 7.28TB

Write- Raid1-1C3
Read- Raid1-1C3R

Raid1-1C4
Usable File Space: 7.28TB

Write- Raid1-1C4
Read- Raid1-1C4R

Raid10-1C3
Usable File Space: 7.28TB

Write- Raid10-1C3
Read- Raid10-1C3R

Raid10-1C4
Usable File Space: 7.28TB

Write- Raid10-1C4
Read- Raid10-1C4R

Raid5
Usable File Space: 10.92TB

Write- Raid5
Read- Raid5R

Raid5-1
Usable File Space: 10.92TB

Write- Raid5-1

Read- Raid5-1R

Raid5-1C3
Usable File Space: 10.92TB

Write- Raid5-1C3
Read- Raid5-1C3R

Raid6
Usable File Space: 7.28TB

Write- Raid6
Read- Raid6R

Raid6-1C3
Usable File Space: 7.28TB

Write- Raid6-1C3
Read- Raid6-1C3R

Raid6-1C4
Usable File Space: 7.28TB

Write- Raid6-1C4
Read- Raid6-1C4R

******************************************** FINIS **********************************************

As a final note, users with a 1Gbps LAN may not be concerned about most speeds since the system
can respond fast enough. But for faster LAN performance and large backups/replications, this data may be helpful.

Feel free to ask for a specific kind of test and I will be happy to do it if I have the hardware available.

:sunglasses:

PS: Any booboo’s or whoopsies are entirely the fault of my keyboard, but happily corrected!

7 Likes

Very cool! Thank you so much for spending the time to test all these variations. Puts some real-world numbers to the assumptions that are made about the various RAID levels. This should be very helpful even for more experienced users. The speed vs fallback/repair risks comparison comes to mind.

1 Like

I would like to do that as well, but I don’t know “ALL” the recovery options for all the Raid levels as of yet. It will have to wait for a bit as I need to make a trip to Tennessee to work on a 5 system setup for some folks for about a week. Then I will certainly look into it as well as test it!

(Hint: I already tested Raid5 in a 3 disk setup successfully because I learned that the last known bug was supposedly corrected earlier this year!)

:sunglasses:

3 Likes

Thank you for sharing your investigation results :blush:

That’s also great to hear, as I am follwoing the BTRFS Raid 5/6 issue for years now. Starting in 2018 I would not have thought that we are still talking about this issue in 2024 :joy:

Nevertheles, do you have any links regarding this “all known bugs are fixe”?

All the best
Simon

2 Likes

https://www.reddit.com/r/btrfs/comments/127zmsx/what_do_you_think_about_the_kernel_62_btrfs/

Many things in Google search, but bottom line is 6.2 seems to have fixed the “write hole” bug??? Other “features” maybe still not implemented?? (I’m a dummy on that sort of stuff)

Hope OpenSUSE uses 6.2 or something…

:sunglasses:

PS: I leave for Tennessee in a few hours… I’ll check in but all the setups will be powered down at home.

3 Likes

In case you need some ideas for more speed tests to run, I just had a thought for you:

Currently the “share size enforcement” is disabled in Rockstor, and with it the “quota” of btrfs.
I found this short discussion about re-enabling the quotas at some point soon, due to progress in btrfs kernel:

I actually have the quota enabled partially on my system (on a SSD) for a while now and everything seems fine and without a noticeable speed penalty.

In case you have some time to spare and would like to do this, feel free to make an explicit test with quotas disabled & enabled and check whether there is a measurable difference.

Cheers
Simon

PS and thank you for the link regarding BTRFS Raid 5/6 @Tex1954

1 Like

Still in Tennessee for another week or so. On my laptop seeing how things going. I too am interested in several features like Quotas and such. Hope to dive in again we I return!

This is like a vacation for me and I am enjoying it!

:sunglasses: