Need help replacing a failed disk in a RAID10

Troubleshooting
1

Hi, not sure if I should put this under Troubleshooting or Support.

I’m running a RAID10 with 4 disks of the same size.
Now one disk has failed causing the pool failing. I’ve ordered the same type of disk again to replace the failing one.

ls -la /dev/sd*
brw-rw---- 1 root disk 8, 0 Dec 4 10:37 /dev/sda
brw-rw---- 1 root disk 8, 1 Dec 4 10:37 /dev/sda1
brw-rw---- 1 root disk 8, 2 Dec 4 10:37 /dev/sda2
brw-rw---- 1 root disk 8, 3 Dec 4 10:37 /dev/sda3
brw-rw---- 1 root disk 8, 16 Dec 4 10:37 /dev/sdb
brw-rw---- 1 root disk 8, 32 Dec 4 10:37 /dev/sdc
brw-rw---- 1 root disk 8, 48 Dec 4 10:37 /dev/sdd
brw-rw---- 1 root disk 8, 64 Dec 4 10:37 /dev/sde

fdisk -l

Disk /dev/sda: 16.1 GB, 16106127360 bytes, 31457280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000cb919

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 3123199 1048576 82 Linux swap / Solaris
/dev/sda3 3123200 20971519 8924160 83 Linux

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk /dev/sde: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

btrfs filesystem show:

Label: ‘rockstor_rockstor’ uuid: 36a00428-512d-4353-931b-d1af4fae2ba7
Total devices 1 FS bytes used 6.37GiB
devid 1 size 8.51GiB used 8.51GiB path /dev/sda3

warning, device 2 is missing
Label: ‘StoragePool’ uuid: 7253a3b0-2ddc-41da-a2b8-0b3113f8007b
Total devices 4 FS bytes used 4.40TiB
devid 1 size 2.73TiB used 2.21TiB path /dev/sdb
devid 3 size 2.73TiB used 2.21TiB path /dev/sdd
devid 4 size 2.73TiB used 2.21TiB path /dev/sde
*** Some devices missing

/dev/sdc is missing.

Reading the documentation at http://rockstor.com/docs/data_loss.html#data-loss-prevention-and-recovery-in-raid10-pools
the example mentions /dev/sda to be missing and then to mount in degraded mode the device /dev/sdb
This means I should mount the still working disk?
When reading the btrfs wiki the example and text more points to mounting the failing disk in degraded mode.

So I’m unsure on what to do. So my question: what should the commands be for a RAID10?
Do I need to mount all 3 remaining devices in degraded mode, mount the failing disk or the still working paired disk (how do I find out which one it would be then)?

mount -o degraded /dev/sdX /mnt2/mypool

since btrfs fi show does not show the devid of the failing disk, I would assume it to be “2” based on the printout:

btrfs replace start 2 /dev/sdc /mnt2/mypool

Anyone can help me out here with the commands?

Thanks in advance.

2

@XiangXiangSD Welcome to the Rockstor community.

It is now possible to add the degraded mount option to Rockstor via the custom mount options. That way the Web-UI can handle the mount operation and allow you to at least refresh your disks. Modern versions of Rockstor, i.e the Stable branch or our Release candidate “Built on openSUSE” variant have prompts to guide your through this process.

Basically you need ensure you are running a modern Rockstor and then add “degraded” and “ro” as custom mount options. This way you can refresh your backups before removing the ro to allow adding the replacement disk.

As to which drive you mount, you can mount a btrfs pool from any member of that pool. With missing disks it’s more tricky to know their id and they are usually referenced as “missing” but you can just use any one of the remaining disks to reference the pool.

Hope that helps.

Let us know how you get on. And if you specify the version of Rockstor you are running it can help folks on the forum help you more effectively.

yum info rockstor

should give the required info on which version you are running.

3

Hi phillxnet,

Thanks for the welcome and your reply.

I totally forgot to mention the version, I’m ashamed to admit that I’m running on a very old version :stuck_out_tongue: : 3.9.1

So I would like some help with the command line. Since I’m running it in a VM, I was planning to create a new VM with latest Rockstor and import the disks. But I would need to fix the RAID10 first and have the disks balanced before I do that. I did receive the new disk today.

I think I could do with the the following commands:

mount -o degraded /dev/sdb /mnt2/StoragePool

btrfs replace start 2 /dev/sdc /mnt2/StoragePool

Can you please verify if this is correct?

Also, do I need to do the mount -o degraded before I physically replace the failed disk?

4

So I went ahead and replaced the disk, but running into an issue.
What I did:

mount -o degraded /dev/sdb /mnt2/StoragePool

Powered down the node and replaced the failed disk.

Boot up the server.

fdisk shows the new disk and smartctl is happy with it:

# smartctl -H /dev/sdc
smartctl 6.2 2017-02-27 r4394 [x86_64-linux-4.12.4-1.el7.elrepo.x86_64] (local build)
Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Step 10 of http://rockstor.com/docs/data_loss.html#data-loss-prevention-and-recovery-in-raid1-pools tells me to make sure the pool is mounted.
Which I tried, but mounting fails:

# mount -o degraded /dev/sdb /mnt2/StoragePool
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

dmesg:

[ 2.718270] Adding 1048572k swap on /dev/sda2. Priority:-1 extents:1 across:1048572k FS
[ 2.739576] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[ 2.755422] input: PC Speaker as /devices/platform/pcspkr/input/input5
[ 2.770420] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 2.981112] ppdev: user-space parallel port driver
[ 3.306928] IPv6: ADDRCONF(NETDEV_UP): ens18: link is not ready
[ 3.310827] IPv6: ADDRCONF(NETDEV_UP): ens18: link is not ready
[ 3.311361] e1000: ens18 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 3.311659] IPv6: ADDRCONF(NETDEV_CHANGE): ens18: link becomes ready
[ 3.759910] NFSD: starting 90-second grace period (net ffffffff81dd5400)
[ 12.467516] BTRFS: device label StoragePool devid 4 transid 79474 /dev/sde
[ 12.467762] BTRFS: device label StoragePool devid 3 transid 79474 /dev/sdd
[ 12.468006] BTRFS: device label StoragePool devid 1 transid 79474 /dev/sdb
[ 15.895521] device-mapper: uevent: version 1.0.3
[ 15.895579] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com
[ 16.244979] BTRFS info (device sdb): use lzo compression
[ 16.244981] BTRFS info (device sdb): disk space caching is enabled
[ 16.244982] BTRFS info (device sdb): has skinny extents
[ 16.245422] BTRFS error (device sdb): failed to read the system array: -5
[ 16.270226] BTRFS error (device sdb): open_ctree failed
[ 2614.133181] BTRFS info (device sdb): allowing degraded mounts
[ 2614.133183] BTRFS info (device sdb): disk space caching is enabled
[ 2614.133183] BTRFS info (device sdb): has skinny extents
[ 2614.133533] BTRFS warning (device sdb): devid 2 uuid deda5911-3947-422e-820f-04493c6d0e5b is missing
[ 2614.222850] BTRFS info (device sdb): bdev (null) errs: wr 0, rd 2244, flush 0, corrupt 0, gen 0
[ 2621.904871] BTRFS warning (device sdb): missing devices (1) exceeds the limit (0), writeable mount is not allowed
[ 2621.937170] BTRFS error (device sdb): open_ctree failed

Any help is appreciated.

EDIT:

# blkid
/dev/sda1: UUID=“281e26bc-11dd-4761-9fca-5f901881354f” TYPE=“ext4”
/dev/sda2: UUID=“2e763dfe-839b-45ee-ad3b-d7aa5df3a918” TYPE=“swap”
/dev/sda3: LABEL=“rockstor_rockstor” UUID=“36a00428-512d-4353-931b-d1af4fae2ba7” UUID_SUB=“136897d3-0a7c-4fc8-b002-455363486f40” TYPE=“btrfs”
/dev/sdb: LABEL=“StoragePool” UUID=“7253a3b0-2ddc-41da-a2b8-0b3113f8007b” UUID_SUB=“cbeee571-96e0-4397-b860-b1b3d320337e” TYPE=“btrfs”
/dev/sdd: LABEL=“StoragePool” UUID=“7253a3b0-2ddc-41da-a2b8-0b3113f8007b” UUID_SUB=“078bfaec-9c74-4c11-977f-67243429faf9” TYPE=“btrfs”
/dev/sde: LABEL=“StoragePool” UUID=“7253a3b0-2ddc-41da-a2b8-0b3113f8007b” UUID_SUB=“586569cf-301f-47d0-8e3f-2a30312a6ef9” TYPE=“btrfs”

5

So I read that since I’ve already mounted my multi-device disk as degraded once, I cannot do it for a 2nd time.
Is there anyway to fix this? Or do I have to end up recreating my entire raid10 and putting back a backup?

6

That can be the case but it’s not that you can’t re-mount degraded, just that you only get one chance, with the older kernels particularly, to mount rw. You could try mounting degraded and ro. That way you can hopefully retrieve / refresh your backups and take things from there. The newer kernels in our 4 variant may well help with these constraints but I think there is still a limited rw mount capability in some circumstances.

So you are looking to use degraded and read only mount options simultaneously to at least get to your existing data.

Hope that helps and let us know how this goes. Plus you may well have better results doing all this with a newer 4 variant due to it’s extended kernel capabilities via the newer btrfs stack. Just a thought. Although you will most likely still need to mount degraded,ro via the command line first before being able to do the initial import as we don’t yet have that capability from the Web-UI. But once it’s already mounted it should import OK. Assuming you can achieve a degraded,ro mount in the first place.

Apologies for not catching your:

and:

discrepancy earlier.

You can usually mount a btrfs volume (Pool in Rockstor speak) by any one of it’s members. You can also specify all members if you like. But be careful when using ids as they can change with various disk operations.

1 Like