Added Disk - Rebalance doesn't seem to be happening

[HarryHUK (Matthew Harrison)]

Brief description of the problem

Hi again. I’m replacing another disk (4TB) with a larger one (8TB) as it has become available.
Over 90 minutes after the rebalance started, nothing seems to be happening. I know it’s going to take a while, because I’ve already replaced one.

Anyway, nearly 2 hours after starting, 0% progress, and the new drive is still showing 1GB allocated. The new pool size is showing correctly though.

Detailed step by step instructions to reproduce the problem

1. While shutdown, added ZA1FZPBY physically to /dev/sdc (having moved the 4TB drive it will replace (Z306JZ39 ) to /dev/sde.
2. Wiped the new disk, as last time, but this time on the Rockstor box, not my ubuntu desktop. (fdisk to remove the partitions, then wipe the disk in the Assign role page, from Disks)
3. In Pools, Resize/Re-raid, Add disk, select the new drive and Submit

(The last time, I added the 8TB (WKD391VC) to /dev/sde first, and after removing the 4TB it replaced, and rebalanced, shutdown and moved the new drive to the old drive’s position. From this I figured that the drives are identified by their serial number, not which file in /dev they are mapped to… maybe I was wrong… but, everything started fine before adding the disk to the pool.)

Web-UI screenshots

See both the existing 8TB drive (WKD391VC, not the one being replaced, but the previous one added) and the new one ( ZA1FZPBY ) are showing unknown power status.

New disk still only has 1 GB allocated.

Balance is running, still 0% progress

See how quiet the disks are. I would expect a flurry of writing going on.

ssh’d in. btrfs fi us shows the new drive, /dev/sdc, has no data, and 1GB of metadata, (the other 8TB drive, /dev/sdd, has 8GB of metadata)
Existing data seems not to have budged a bit.

Error Traceback provided on the Web-UI

What’s the best way to deal with a hung re-balance?
(Fortunately I have backup copies of the data…)

I’ve tried a btrfs balance pause, and that just hung.

[HarryHUK (Matthew Harrison)]

Update

As btrfs balance pause, and btrfs balance cancel both hung, I tried shutting down, but that seemed to be hung on the balance that wasn’t progressing. So…
power button with extreme prejudice.

Restarted, and the logs show that it noticed things weren’t right, and seems to be fixing sda at the moment. And data seems to be moving onto the new disk.
I’m going to leave it overnight.

[HarryHUK (Matthew Harrison)]

Things always look better in the morning

See the difference in disk IO and CPU.

I haven’t found anything in the logs yet for the failed balance start cmd, but I’m positive things were stalled last night. I really tried to look for signs of life.

I did find this thread on reddit. One user says a single block can take a long time, and to wait up to an hour after cancel. OP waited several.

Just given me an idea for a dashboard widget to show task progress (e.g. balance, scrub) and maybe a ‘top’ widget.

Anyway, not much to go on, really, but now that it’s moving, I’m happy, so we can leave it.
Just sharing my experience as a new user (who is learning).

Question

Would it be worth upgrading to the latest 4.0 RC? Is it at least as stable as 3.9.2-57 on 4.12 kernel?

[phillxnet (Philip Guyton)]

@HarryHUK Thanks for the details report and status update.
Re:

@HarryHUK

Would it be worth upgrading to the latest 4.0 RC? Is it at least as stable as 3.9.2-57 on 4.12 kernel?

Take a look at my recent response to a similar question from @Noggin recently:

System unresponsive

@Noggin Sorry I’ve not got much time on this one but re: Quite possibly. You still need to scrub the pool, as you tried but I suspect you have either bad PSU or bad ram. Investigate those before you try anything else at the OS level as with bad hardware all OS/fs stuff is essentially doomed. Not necessarily. I would say if you don’t need AD then definitely go with the openSUSE as it’s btrfs core and kernel are years newer and given btrfs has seen some significant imporovements in this time I…

@phillxnet

Am I correct in assuming that CentOS is more stable than OpenSuse?

Not necessarily. I would say if you don’t need AD then definitely go with the openSUSE as it’s btrfs core and kernel are years newer and given btrfs has seen some significant imporovements in this time I would definitely go for the openSUSE variant via the DIY installer build method:

And the following forum thread references the changes made from 3.9.2-57 to 4.0.4:

BETA "Built on openSUSE" Testing Channel Changelog

I am delighted, and somewhat relieved, to finally announce after more than 2 years of transition from our CentOS base, our first BETA release within our “Built on openSUSE” only testing channel. We are, as yet, not quite at feature parity with our prior CentOS variant (Stable Channel) but getting very close. The intention is to finally offer a Stable Channel rpm for our “Built on openSUSE” rpms once we have this final beta testing phase completed successfully. As always there are wrinkles and o…

Hope that helps.

[HarryHUK (Matthew Harrison)]

Many thanks for your reply. The disk add is nearly done now,
and after that I’ll remove the 4TB disk. Will I still need to scrub, given I will have done 2 re-balances?

(I hope my issue wasn’t hardware… it hasn’t crashed )

Will have a look at 4.0 soon.

[Noggin (Henry McDonald)]

FYI, I did a several rebalances when swapping out disks and when changing from RAID10 to RAID6. Disabling quotas made everything go much faster.