Problems with disk replacement

maxhq · January 10, 2017, 3:42pm

I had three disks (A,B,C) running in RAID1 configuration for a pool “data” (Rockstor 3.8.16-2). Disk C seemed to fail soon (disconnects, click noise) and I wanted to replace it.

Side note: as the UI does not provide support for disk replacement (yet), it would be really helpful if the UI showed the disk name (/dev/sdx) and BTRFS id of each disk in a pool. Otherwise it takes more steps on the command line to find out which disk is the right one.

I did this:

Try #1

power off, remove disk C, added new disk D.
Rockstor then recognizes missing disk C and I tried to remove it via trash icon in the UI but that failed.
see #1 here: https://www.icloud.com/sharedalbum/de-de/#B0s532ODWxuaiq

Try #2

power off, inserted disk C again
power on, command line:
btrfs replace start 3 /dev/sdd /mnt2/data
replacement successful, but Rockstor UI does not see that disk D belongs to pool “data” (“Rescan” in Disks menu does not change this)
btrfs fi sh /mnt2/data says:

Label: 'data'  uuid: 52fde21a-5b20-4cb9-9f25-1f3eac61704f
	Total devices 3 FS bytes used 613.49GiB
	devid    1 size 931.51GiB used 412.06GiB path /dev/sda
	devid    2 size 931.51GiB used 412.03GiB path /dev/sdb
	devid    3 size 931.51GiB used 412.03GiB path /dev/sde

How can I make Rockstor see that disk D belongs to the pool?

phillxnet · January 10, 2017, 5:06pm

@maxhq Hello again. The ability to display current boot temp short names (sdX) in Disks as an option has been discussed and the disk model does maintain them but they are currently held just for this pending UI tweak. We use the by-id names for boot to boot stability but yes in some settings it would be nice to be able to switch to showing the temp short names. No good from one boot to the next though is the problem hence the new default of by-id which are stable.

At a guess I would say you ran into the “cosmetic” issue here:

Re the removal via bin icon as it’s still relying on a previous mechanism that is now at odds with other code. So it requires a resent rescan in order that the ‘fake’ device name be recent enough to access the db entry. We need to move this to a serial based search really, that way it will be in sync with the more recent code. I’ve added this forum thread to that issue to help encourage it’s fix along a tad.

Your report has brought more to the for, at least for me, how this issue is rather more important that I at least had previously thought. I’ll try and get to it myself soon as in this particular scenario it’s a bit of a pain and really lends itself to making thing worse at a delicate time. Oh well, bit by bit.

Just seen the edit with the link to the disk page images. Is that how things look now (in the #2 pic)?

Also can you paste the output of:

lsblk -P -o 'NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID'

and

mount | grep btrfs

We definitely have some work to do in this area though.

Incidentally, as a side note, have you tried “-d sat” as custom smart options on that sandisk ultra?

maxhq · January 10, 2017, 6:42pm

Hi @phillxnet, you are great! Thanks for the detailed answer.

To your points:

Disk names: if you don’t want to show the temporary /dev/sdx names, the UUID would also be fine to match the disks in the Rockstor UI with the command line.

Trash icon: You might be right then with the missing rescan before I tried removal via trash icon. Good to hear that it might be fixed soon (where “soon” means < 1 year ).

Current state: yes #2 is how it looks now, I did do anything since then so we can test this a bit.

$ lsblk -P -o 'NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID'
NAME="sdf" MODEL="Ultra Fit       " SERIAL="4C530001191128105421" SIZE="57,9G" TRAN="usb" VENDOR="SanDisk " HCTL="8:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="usbsticks" UUID="2a71f9de-ff1c-4aa3-b8f0-608666a8ed0d"
NAME="sdd" MODEL="Ultra Fit       " SERIAL="4C531001521209118043" SIZE="57,9G" TRAN="usb" VENDOR="SanDisk " HCTL="7:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="usbsticks" UUID="2a71f9de-ff1c-4aa3-b8f0-608666a8ed0d"
NAME="sdb" MODEL="ST1000DM003-1ER1" SERIAL="Z4Y2C9MX" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="1:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="data" UUID="52fde21a-5b20-4cb9-9f25-1f3eac61704f"
NAME="sr0" MODEL="DVD RAM UJ8E2   " SERIAL="H086_004119" SIZE="1024M" TRAN="sata" VENDOR="hp      " HCTL="4:0:0:0" TYPE="rom" FSTYPE="" LABEL="" UUID=""
NAME="sdg" MODEL="LUN 00 Media 0  " SERIAL="000002660A01" SIZE="256M" TRAN="usb" VENDOR="HP iLO  " HCTL="9:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""
NAME="sdg1" MODEL="" SERIAL="" SIZE="251M" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="vfat" LABEL="VID" UUID="5417-C92E"
NAME="sde" MODEL="WDC WD10EZEX-08W" SERIAL="WD-WCC6Y4VECZYV" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="2:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""
NAME="sdc" MODEL="Ultra           " SERIAL="4C530123131108122192" SIZE="29G" TRAN="usb" VENDOR="SanDisk " HCTL="6:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""
NAME="sdc2" MODEL="" SERIAL="" SIZE="2,9G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="swap" LABEL="" UUID="daa70678-e5f7-45f1-bc9b-ca6847466c42"
NAME="sdc3" MODEL="" SERIAL="" SIZE="25,6G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="btrfs" LABEL="rockstor_rockstor" UUID="7474b7ee-466f-4e77-84c3-325187e19531"
NAME="sdc1" MODEL="" SERIAL="" SIZE="500M" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="ext4" LABEL="" UUID="6be814d4-3300-4d19-bac1-2b4c51a0bfb6"
NAME="sda" MODEL="ST1000DM003-1ER1" SERIAL="Z4Y2C9SM" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="0:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="data" UUID="52fde21a-5b20-4cb9-9f25-1f3eac61704f"
NAME="sdh" MODEL="WDC WD10EARS-00Y" SERIAL="WD-WCAV5D118259" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="3:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""

$ mount | grep btrfs
/dev/sdc3 on / type btrfs (rw,noatime,space_cache,subvolid=258,subvol=/root)
/dev/sdc3 on /home type btrfs (rw,noatime,space_cache,subvolid=257,subvol=/home)
/dev/sdd on /mnt2/usbsticks type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
/dev/sdc3 on /mnt2/rockstor_rockstor type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
/dev/sda on /mnt2/data type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
/dev/sda on /mnt2/data_max type btrfs (rw,relatime,space_cache,subvolid=532,subvol=/data_max)
/dev/sda on /mnt2/data_backup_obnam type btrfs (rw,relatime,space_cache,subvolid=535,subvol=/data_backup_obnam)
/dev/sda on /mnt2/rockstor_addons type btrfs (rw,relatime,space_cache,subvolid=456,subvol=/rockstor_addons)
/dev/sda on /mnt2/rockstor_addon_plex type btrfs (rw,relatime,space_cache,subvolid=457,subvol=/rockstor_addon_plex)
/dev/sda on /mnt2/rockstor_addon_gogs_conf type btrfs (rw,relatime,space_cache,subvolid=5762,subvol=/rockstor_addon_gogs_conf)
/dev/sda on /mnt2/data_software type btrfs (rw,relatime,space_cache,subvolid=931,subvol=/data_software)
/dev/sda on /mnt2/carl type btrfs (rw,relatime,space_cache,subvolid=6292,subvol=/carl)
/dev/sda on /mnt2/rockstor_addon_gogs_repos type btrfs (rw,relatime,space_cache,subvolid=4347,subvol=/rockstor_addon_gogs_repos)
/dev/sda on /mnt2/data_pictures type btrfs (rw,relatime,space_cache,subvolid=277,subvol=/data_pictures)
/dev/sda on /mnt2/data_documents type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/data_documents)
/dev/sda on /mnt2/data_media type btrfs (rw,relatime,space_cache,subvolid=444,subvol=/data_media)
/dev/sda on /mnt2/data_backup type btrfs (rw,relatime,space_cache,subvolid=367,subvol=/data_backup)
/dev/sdd on /mnt2/usbsticks_camera_sync type btrfs (rw,relatime,space_cache,subvolid=282,subvol=/usbsticks_camera_sync)
/dev/sdd on /mnt2/usbsticks_addon_syncthing_config type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/usbsticks_addon_syncthing_config)
/dev/sdd on /mnt2/usbsticks_addon_syncthing_data type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/usbsticks_addon_syncthing_data)
/dev/sdc3 on /mnt2/home type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/home)
/dev/sdc3 on /mnt2/root type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/root)
/dev/sda on /mnt2/data_max/.BTRFS_SNAPSHOT_201701101400 type btrfs (rw,relatime,space_cache,subvolid=8419,subvol=/.snapshots/data_max/BTRFS_SNAPSHOT_201701101400)
/dev/sda on /mnt2/data_documents/.BTRFS_SNAPSHOT_201701101400 type btrfs (rw,relatime,space_cache,subvolid=8420,subvol=/.snapshots/data_documents/BTRFS_SNAPSHOT_201701101400)

SMART: This was a hint into the right direction: option “-d scsi” worked. Would be nice if Rockstor showed the error in the UI

phillxnet · January 10, 2017, 7:26pm

@maxhq Thanks.

So 364 days left on that issue then. Loads of time.

A table option to flip between by-id and sdX type names I think was the consensus / potential plan from memory and quite a way back now

As for the old flaky ticking drive:

OK, yes just wanted to make sure the old disk was cleared of btrfs as it appears to have been.

But looking at the new replacement disk we have:

Which is not what I would expect ie there is no FSTYPE=“btrfs” and no LABEL=“data”.

If you could check I have the disks identified correctly in above, just in case.

I’m assuming here that the replace command has now finished. Does the output of the following command suggest this:

btrfs replace status /mnt2/data

If it has finished and your outputs for the above drives are the same as you posted could you try executing the following command:

udevadm trigger

Then Rescan your disks via the button on the Disk page.

Try not to do anything else for the time being as I think I’ve seen this before and it would be useful to know if that command helps.

Also another lsblk command like the first after you execute the udevadm command would be good.

Cheers and thanks for helping with looking into this.

(Re side note/line custom smart options - Yes the -d scsi was going to be my next suggestion)

maxhq · January 10, 2017, 7:58pm

@phillxnet I am impressed!
Indeed the btrfs replace was finished and you also got the disks right.
Aaaaand: udevadm trigger helped! I immediately refreshed the UI disks page before and after executing it and now the new disk appears as part of the “data” pool.

By the way btrfs fi label /dev/sde showed “data” even before the udevadm command, so the label was just not known to the kernel (?).

$ lsblk -P -o 'NAME,MODEL,SERIAL,SIZE,TRAN,VENDOR,HCTL,TYPE,FSTYPE,LABEL,UUID'
NAME="sdf" MODEL="Ultra Fit       " SERIAL="4C530001191128105421" SIZE="57,9G" TRAN="usb" VENDOR="SanDisk " HCTL="8:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="usbsticks" UUID="2a71f9de-ff1c-4aa3-b8f0-608666a8ed0d"
NAME="sdd" MODEL="Ultra Fit       " SERIAL="4C531001521209118043" SIZE="57,9G" TRAN="usb" VENDOR="SanDisk " HCTL="7:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="usbsticks" UUID="2a71f9de-ff1c-4aa3-b8f0-608666a8ed0d"
NAME="sdb" MODEL="ST1000DM003-1ER1" SERIAL="Z4Y2C9MX" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="1:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="data" UUID="52fde21a-5b20-4cb9-9f25-1f3eac61704f"
NAME="sr0" MODEL="DVD RAM UJ8E2   " SERIAL="H086_004119" SIZE="1024M" TRAN="sata" VENDOR="hp      " HCTL="4:0:0:0" TYPE="rom" FSTYPE="" LABEL="" UUID=""
NAME="sdg" MODEL="LUN 00 Media 0  " SERIAL="000002660A01" SIZE="256M" TRAN="usb" VENDOR="HP iLO  " HCTL="9:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""
NAME="sdg1" MODEL="" SERIAL="" SIZE="251M" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="vfat" LABEL="VID" UUID="5417-C92E"
NAME="sde" MODEL="WDC WD10EZEX-08W" SERIAL="WD-WCC6Y4VECZYV" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="2:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="data" UUID="52fde21a-5b20-4cb9-9f25-1f3eac61704f"
NAME="sdc" MODEL="Ultra           " SERIAL="4C530123131108122192" SIZE="29G" TRAN="usb" VENDOR="SanDisk " HCTL="6:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""
NAME="sdc2" MODEL="" SERIAL="" SIZE="2,9G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="swap" LABEL="" UUID="daa70678-e5f7-45f1-bc9b-ca6847466c42"
NAME="sdc3" MODEL="" SERIAL="" SIZE="25,6G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="btrfs" LABEL="rockstor_rockstor" UUID="7474b7ee-466f-4e77-84c3-325187e19531"
NAME="sdc1" MODEL="" SERIAL="" SIZE="500M" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="ext4" LABEL="" UUID="6be814d4-3300-4d19-bac1-2b4c51a0bfb6"
NAME="sda" MODEL="ST1000DM003-1ER1" SERIAL="Z4Y2C9SM" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="0:0:0:0" TYPE="disk" FSTYPE="btrfs" LABEL="data" UUID="52fde21a-5b20-4cb9-9f25-1f3eac61704f"
NAME="sdh" MODEL="WDC WD10EARS-00Y" SERIAL="WD-WCAV5D118259" SIZE="931,5G" TRAN="sata" VENDOR="ATA     " HCTL="3:0:0:0" TYPE="disk" FSTYPE="" LABEL="" UUID=""

After the trigger I got the following message (/var/log/messages):
systemd-logind: Watching system buttons on /dev/input/event0 (Power Button)
If you plan to include udevadm trigger into the code you should consider restricting it to certain actions or devices:

The action argument is of utmost importance. Without it, the kernel will tell udev to treat all devices as NEW. This will have lots of side-effects you are probably not expecting.
“change” is completely safe. It tells udev just to refresh devices, and make sure everything’s as it should be.

(from a post in 2009, don’t know if this is still valid)

Thank you very much!

phillxnet · January 10, 2017, 8:45pm

@maxhq Thanks for confirming that and I’m chuffed it got thing sorted for you.

As it goes your report is quite timely and you will be please to know that the following issue addresses this, at least from the disk add point of view which is all we have at the moment:

and that issue is addressed by the following pending pull requrest:

Yes I had considered this but I thought that the overhead of parsing this and that to work out relevant restrictions would probably out weight the overhead of the global udevadm trigger. Wanted to get it in ‘as is’ for now and maybe in the future we can pass the existing wrapper a list of devices added. But it may be that we don’t actually know the list of devices. Anyway , at least a bridge or two ahead of us there and I think for now the global refresh, overkill though it is, should serve. Also my efforts to reproduce this enough times for proof that lead to the issue and patch took me a few days to get enough examples of this occurring that I only had the chance to test a couple of commands (the other is listed in the issue and is very specific), we would need a repeatable and reliable example scenario to test any other udev directive and I’m hoping that you have just found one. But it will need testing as in many many of my trials the command was completely unnecessary so it may well be that the “change” directive doesn’t work.

Thanks again for you input here and I have added appropriate links back to this forum thread from the sighted issue and patch. Also earlier I opened the following:

This might take a while but at least it’s there now. Currently Rockstor wise I think the advise would be to first Pool - Resize add a new drive and then once that’s done, Pool - Resize - remove the old disk.

Bit by bit and,
Your Welcome.

maxhq · January 11, 2017, 12:16am

@phillxnet Thank you for all your help. I am glad to see that you already opened issues and have a PR ready.
I see that it is quite difficult to test the problem and the variations of udevadm trigger that might work. Maybe I can do some tests with my USB sticks, the data on them is not as important as the rest…

Btw.: is there a UI way to replace a failed disk? I guess in that case a pool resize would not work, or does it?