RAID 10 disk removal/rebalance

1

Hi all.

I currently have a 9 disk RAID 10 setup:

btrfs filesystem show

Label: 'MainPool'  uuid: 8c10c7f7-3b1e-4f77-b9da-e1045701a0b0
	Total devices 9 FS bytes used 5.78TiB
	devid    1 size 931.51GiB used 741.75GiB path /dev/sde
	devid    2 size 2.73TiB used 1.45TiB path /dev/sdk
	devid    3 size 1.82TiB used 1.45TiB path /dev/sdc
	devid    4 size 931.51GiB used 741.28GiB path /dev/sdd
	devid    5 size 1.82TiB used 1.45TiB path /dev/sdf
	devid    6 size 1.82TiB used 1.45TiB path /dev/sdi
	devid    7 size 3.64TiB used 1.45TiB path /dev/sdg
	devid    8 size 3.64TiB used 1.45TiB path /dev/sdh
	devid    9 size 2.73TiB used 1.45TiB path /dev/sdj

I just now got a SMART failure on /dev/sdc. I’ve done the math, and if I remove the drive, I will still have enough space remaining for all my data. What I want to know is can I just remove the bad disk, and will btrfs auto-rebalance the data that remains? If I remove the drive, my data should still be ok, because it was a RIA 10 setup? I know that I’ll be in the danger zone until the rebalance finishes. But before I do anything, I wanted to double check the best operations to perform.

2

@kupan787 A belated welcome to the Rockstor community forum.

No: as this would put the pool into a degraded state. However you can, given the assumption that the drive is working but due for replacement’ resize the pool. That is:

Storage - Pools - click on Pool name - Resize - Remove Disks - Tick box by poorly drive.

This should first do a quick calculation, as you have done, on if the operation is viable, space wise. Then it kick off a disk removal process which in turn does the re-balance / redistribution of the data currently on the disk to be removed.

In the above scenario Rockstor internally executes a btrfs device delete/remove ie:

The state of the balance (which can take quite a while) can be seen by browser refreshing the ‘Balances’ tab on the Pool’s page.

Best to make sure you select the correct drive, ie note the serial number of the disk in question as it appears in the by-id names used by Rockstor’s UI.

Yes as btrfs raid 10 can only handle one drive failure not the occasionally associated 2.[quote=“kupan787, post:1, topic:3251”]
I wanted to double check the best operations to perform.
[/quote]

If the drive is very poorly, as apposed to just a bad sector appearing that was also successfully relocated, then you are better of using a command line option that will restrict that particular drive to a read-only role within the pool, but I think that is only available on btrfs device replace.

If the data is not backed up then you would of course be best advised to do this first.

Hope that helps.

3

So, I am hoping I am not totally SOL…

My power went out this morning, in the middle of the re-balance. I turned on my server, and the rockstor site isn’t loading locally. I then SSHed into the box, and if I do:

[root@rocknas ~]# btrfs fi show

    Label: 'MainPool'  uuid: 8c10c7f7-3b1e-4f77-b9da-e1045701a0b0
	Total devices 9 FS bytes used 5.78TiB
	devid    1 size 931.51GiB used 803.00GiB path /dev/sdf
	devid    2 size 2.73TiB used 1.45TiB path /dev/sdl
	devid    3 size 1.82TiB used 1.45TiB path /dev/sdd
	devid    4 size 931.51GiB used 802.27GiB path /dev/sde
	devid    5 size 1.82TiB used 1.45TiB path /dev/sdg
	devid    6 size 1.82TiB used 1.45TiB path /dev/sdj
	devid    7 size 3.64TiB used 1.45TiB path /dev/sdh
	devid    8 size 3.64TiB used 1.45TiB path /dev/sdi
	devid    9 size 2.73TiB used 1.45TiB path /dev/sdk

It looks like it is there. However, if I do:

[root@rocknas ~]# btrfs fi df /mnt2/MainPool
Data, single: total=3.01GiB, used=2.41GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=1.00GiB, used=91.70MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=16.00MiB, used=0.00B

That is really not correct. If I try and go to the mount point, /mnt2/MainPool, I see nothing is there. I then tried doing a:

[root@rocknas ~]# sudo mount -o degraded /dev/sdf /mnt2/MainPool
mount: wrong fs type, bad option, bad superblock on /dev/sdf,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

[root@rocknas ~]# dmesg | tail
[  507.303607] BTRFS info (device sde): allowing degraded mounts
[  507.303611] BTRFS info (device sde): disk space caching is enabled
[  507.303613] BTRFS info (device sde): has skinny extents
[  507.305792] BTRFS error (device sde): failed to read chunk root
[  507.314459] BTRFS error (device sde): open_ctree failed
[ 1062.453473] BTRFS info (device sde): allowing degraded mounts
[ 1062.453479] BTRFS info (device sde): disk space caching is enabled
[ 1062.453481] BTRFS info (device sde): has skinny extents
[ 1062.457164] BTRFS error (device sde): failed to read chunk root
[ 1062.462832] BTRFS error (device sde): open_ctree failed

Is everything lost? I only lost one drive, so I’d think I should be able to recover (this was a RAID 10). Or did the power going out in the middle of the rebalance just trash it all?

4

Phew!

So all was not lost. I don’t know which part did the trick, but I’ve got it mounted, and am copying data off now (most important first). I had a backup, but it wasn’t a full (only 2 TB of the 5TB). At any rate, some lessons learned :slight_smile:

So here are the commands I ran. Again, not sure which did the trick, and sorry I didn’t capture all of the output.

btrfs rescue super-recover -v /dev/sde
btrfs check /dev/sde
btrfs-zero-log /dev/sde
btrfs rescue zero-log /dev/sde
btrfs device scan

The super-recover found some bad superblocks, and fixed them. The check reported a bunch of “parent transid verify failed” messages. If run of btrfs-zero-log said it was deprecated, and to run rescue zero-log instead. I tried doing mounts at each point, and it kept crapping on me.

Finally after the btrfs device scan, I tried a mount again, and it spun for awhile (1 minute or so), and then came back saying that the mount point was busy. I checked, and sure enough the drive was remounted.