Multi device BTRFS filesystem with disk of different size

phillxnet · April 19, 2019, 8:05pm

@shocker Hello again.

Yes btrfs can mix and match drive sizes, but it’s efficiency from a drive space usage point of view may vary depending on the btrfs raid level. It works by doing raid at a chunk level, ie a chunk of the drive space, usually 1GB for data, is paired with a chunk on another drive in for example raid1. Thus raid at chunk level and so different sized devices can take part in a raid1 pool as long as they have unallocated space (so appropriately new chunks can be made) or existing raid1 chunks (with free space) active in the pool.

I can also chip in with a potentially relevant but now fixed in stable Rockstor bug that may affect systems with 27 or more drives: See the referenced issue, originally reported by @kingwavy in the following forum thread:

Issue:

github.com/rockstor/rockstor-core

disk serial=none or fake-serial re sda[a-z] dev names

opened 08:59AM - 16 May 18 UTC

closed 12:16PM - 16 Jul 18 UTC

phillxnet

Thanks to forum member kingwavy in the following form thread for highlighting th…is bug. On initial inspection of the reported data, drive names of the form sdab, sdac, when the same system has a system drive of the more common sda# type name, are being incorrectly identified as system drives. This is currently thought to be the trigger for an incompatible serial=none response from scan_disks() that is then surfaced by scan_disks() caller _update_disk_state(): ``` File “/opt/rockstor/src/rockstor/storageadmin/views/disk.py”, line 342, in _update_disk_state if (re.match(‘fake-serial-’, do.serial) is not None) or ( File “/usr/lib64/python2.7/re.py”, line 137, in match return _compile(pattern, flags).match(string) TypeError: expected string or buffer ``` With kingwavy's assistance the incorrect labelling of root was identified via debug logging thus: ``` m = Disk(name='sdaf', model='PERC H710', serial='fake-serial-0b606a7b-c7fa-4cf0-9e7a-c8dcc70d4034', size=976748544, transport=None, vendor='DELL', hctl='0:2:0:0', type='disk', fstype='btrfs', label='SCRATCH', uuid='a90e6787-1c45-46d6-a2ba-41017a17c1d5', parted=False, root=True, partitions={}) [14/May/2018 13:21:21] DEBUG [system.osi:478] disks item = Disk(name='sdag', model=None, serial='fake-serial-2a01338a-d494-40ef-80d9-ba2888bfde5f', size=976748544, transport=None, vendor=None, hctl=None, type='disk', fstype='btrfs', label='SCRATCH', uuid='a90e6787-1c45-46d6-a2ba-41017a17c1d5', parted=False, root=True, partitions={}) [14/May/2018 13:21:21] DEBUG [system.osi:478] disks item = Disk(name='sdae', model=None, serial='fake-serial-288f98c5-e108-4ca4-b1e0-fd4a791e7ea5', size=234461593, transport=None, vendor=None, hctl=None, type='disk', fstype='btrfs', label='INTEL_SSD', uuid='a504bf03-0299-4648-8a95-c91aba291de8', parted=False, root=True, partitions={}) ``` ie multiple additional "root=True" attributions and a suspected consequent erroneous 'fake-serial-...' attribution: ie relevant lsblk: NAME="sdae" MODEL="INTEL SSDSC2KW24" SERIAL="CVLT6153072G240CGN" SIZE="223.6G" TRAN="sas" VENDOR="ATA " HCTL="1:0:17:0" TYPE="disk" FSTYPE="btrfs" LABEL="INTEL_SSD" UUID="a504bf03-0299-4648-8a95-c91aba291de8" However the system drive in this trigger system is still identified correctly: ``` [14/May/2018 13:21:21] DEBUG [system.osi:478] disks item = Disk(name='sda3', model='PERC H710', serial='6848f690e936450018b7c3a11330997b', size=277558067, transport=None, vendor='DELL', hctl='0:2:0:0', type='part', fstype='btrfs', label='rockstor_rockstor', uuid='7f7acdd7-493e-4bb5-b801-b7b7dc289535', parted=True, root=True, partitions={}) ``` It is noteworthy that the 'sdab' drive (presumably list order related) is the victim of _update_disk_state() bug with "serial=none": ``` [14/May/2018 13:21:21] DEBUG [system.osi:478] disks item = Disk(name='sdab', model=None, serial=None, size=976748544, transport=None, vendor=None, hctl=None, type='disk', fstype='btrfs', label='SCRATCH', uuid='a90e6787-1c45-46d6-a2ba-41017a17c1d5', parted=False, root=True, partitions={}) ``` when the serial is again plainly available (via default and initial lsblk call): ``` NAME="sdab" MODEL="ST91000640SS " SERIAL="5000c50063041947" SIZE="931.5G" TRAN="sas" VENDOR="SEAGATE " HCTL="1:0:14:0" TYPE="disk" FSTYPE="btrfs" LABEL="SCRATCH" UUID="a90e6787-1c45-46d6-a2ba-41017a17c1d5" ``` Also tangentially relevant is root_disk()'s inflexibility re matching only sda and not sdab type drive names, though this is not currently thought to be a trigger in this issue. Please reference / update the following forum thread with this issues progress / resolution: https://forum.rockstor.com/t/disk-scan-errors-expected-string-or-buffer/4783

and the pull request that fixed it:

github.com/rockstor/rockstor-core

disk serial=none or fake-serial re sda[a-z] dev names. Fixes #1925

rockstor:master ← phillxnet:1925_disk_serial=none_or_fake-serial_re_sda(a-z)_dev_names

opened 06:32PM - 11 Jul 18 UTC

phillxnet

+1088 -18

On systems with 27 or more disks named sd* where the system/root is also install…ed on sda ie sda3 and where the 27th and subsequent disks are named sdaa, sdab etc and are also btrfs formatted; a regex based serial propagation bug (from base device to partition) in scan_disks() resulted in the first listed (by lsblk) sda[a-z] device receiving a serial=none attribution and the second and all subsequent listed sda[a-z] devices receiving an erroneous fake-serial attribution. All affected devices would also have empty (or incorrectly inherited from sda) model, transport, vendor, and hctl attributions. This resulted in an inability for the system to update the disk state as the abstracted disk info from the corrupt parsing of attached disk state caused an exception: "TypeError: expected string or buffer". Note that all affected disks (sda[a-z]) were also miss attributed as root=True in the erroneous abstract data produced by scan_disks() Summary 1. Add scan_disks() test mode to enable repeatable fake-serial- tests. 2. Establish unit tests to reproduce the above issue, as reported (36 disk system) and as a minimum artificial drive subset. 3. Establish unit tests to identify current correct behaviour for system disk on LUKS, mdraid, and bios mdraid; as well as btrfs in partition and data disk on bios mdraid behaviour. 4. Fix overly broad regex to identify system disk partitions, if any: ie improve system disk partition match mechanism. 5. Fix system disk LUKS, mdraid, bios mdraid regressions as a result of (4). 6. Address pre-existing TODO: re mdraid info via model info on non system disks; as part of the fix for (5). Fixes #1925 and, by way of duplication: Fixes #1834 ("disk serial is null") @schakrava Ready for review. Testing: The reported issue was reproduced via unit testing based on the reporter's 36 disk system. An abstract of the minimum drive count thought to be able to reproduce the issue was also produced. Further unit tests were constructed for regression testing during the fix development. ``` ./bin/test --settings=test-settings -v 3 -p test_osi* ``` with the relevant scan_disks() related results, post pr, as follows: ``` ... test_scan_disks_27_plus_disks_regression_issue (system.tests.test_osi.OSITests) ... ok test_scan_disks_btrfs_in_partition (system.tests.test_osi.OSITests) ... ok test_scan_disks_dell_perk_h710_md1220_36_disks (system.tests.test_osi.OSITests) ... ok test_scan_disks_intel_bios_raid_data_disk (system.tests.test_osi.OSITests) ... ok test_scan_disks_intel_bios_raid_sys_disk (system.tests.test_osi.OSITests) ... ok test_scan_disks_luks_on_bcache (system.tests.test_osi.OSITests) ... ok test_scan_disks_luks_sys_disk (system.tests.test_osi.OSITests) ... ok test_scan_disks_mdraid_sys_disk (system.tests.test_osi.OSITests) ... ok ---------------------------------------------------------------------- Ran 11 tests in 0.020s OK ``` The Intel bios raid related unit tests were developed using appropriately configured hardware. And all bcache related test data (LUKS on bcache) assumed the udev rules detailed at: https://forum.rockstor.com/t/bcache-developers-notes/2762 Thanks to forum member kingwavy for their cooperation and their contributing the report and the command outputs used to construct the initial unit test reproducer.

And from the pull request we have the following summary of the issue:

“On systems with 27 or more disks named sd* where the system/root is also installed on sda ie sda3 and where the 27th and subsequent disks are named sdaa, sdab etc and are also btrfs formatted; a regex based serial propagation bug (from base device to partition) in scan_disks() resulted in the first listed (by lsblk) sda[a-z] device receiving a serial=none attribution and the second and all subsequent listed sda[a-z] devices receiving an erroneous fake-serial attribution.”

The fix landed in stable channel updates version 3.9.2-31 around July 2018.

I would say not really as it’s raid6, but as long as you are using a non parity raid (ie non 5 or 6) then you should be good. I.e. Raid 1 or 10 would be a better choice, but we have quite a few reports of ‘working’ parity raid systems but I’d make sure to have a UPS arrangement and one that has been tested to shut the system down in the even of a power loss. But this of course does not protect the pool from a kernel issue. You will also probably find that you have to disable quotas for reasonable performance, again this is currently a stable channel only feature.

If you get this setup done some pics might be nice

Hope that helps.