Errors on Balance operation after adding new disk

[bhsramz]

Brief description of the problem

After adding a 4th 2TB disk to my NAS appliance, I added it to my RAID0 pool. This kicked off a Balance operation, which ran for about 24 hours before the device was restarted through the Web UI with the balance about 15% complete. When Rockstor (3.9.1-0 with recent fresh reinstall) rebooted, an error appeared and the status showed it was 100% finished. In the screenshot below, this is Balance Id 1.

I started a new Balance operation, Id 2, on November 17th, just after the failure. I just returned from a week away, and the Balance operation was showing 17% complete, but the Status field showed “finished”. I thought the Percent finished field may have not been updated, so I reenabled my shares and tried to kick off a Time Machine backup. My MBP was still unable to connect to the share, so I checked back on the Balance tab on the Pool, and saw the screenshot below: 6% finished and an Error message about an input/output error.

Any thoughts on how I can verify that the Balance is still running, or what diagnostics need to be performed? I need to get backups resumed soon, but would it be better to remove the disk from the pool and retry adding it? Most importantly, did the restart during the Balance operation cause any issues? I could not find any information that suggested a soft restart would be an issue while the Balance was in progress, so I hope that didn’t corrupt my Pool.

Web-UI screenshot

Error Traceback provided on the Web-UI

N/A

[phillxnet (Philip Guyton)]

@bhsramz Hello again.

@bhsramz

3.9.1-0 with recent fresh reinstall

This is a rather old version of Rockstor and since previous forum thread indicated a stable channel version of 3.9.2-30:

Shares are Visible in Web UI but Cannot Connect

Brief description of the problem I have not been backing up regularly lately, and over the last couple of weeks my Rockstor appliance updated (I am on Stable updates) to 3.9.2-30. I can no longer connect to my backup shares, and am getting errors in the Web UI while navigating. Detailed step by step instructions to reproduce the problem Tried to use time machine to backup my MBP, and got an error saying the disk was unavailable. When navigating the Web UI to investigate, I am getting the erro…

But you also state re-installing on a suspect system disk so maybe that is your motivation for not pushing your luck with a drive intensive OS update (which a Rockstor update would bring with it).

I would advise moving back to the Stable release, especially given the following new capability on the pool details page added in 3.9.2-35 (assuming you system disk is not suspect):

(The above raid1 pool was able to repair itself in a single subsequent scrub operation, suspected cause: interconnects or controller hw/driver hang)

to address issue:
https://github.com/rockstor/rockstor-core/issues/1532

Which should help to identify which drive is causing your io errors.

That UI aspect gets it’s info from the output of the following command (with your homeshare pool name):

btrfs dev stats /mnt2/homeshare

You could paste that output here along with an:

ls -la /dev/disk/by-id

as your current pool details page is unable to display the temp canonical to by-id name mapping.

Your use of Raid0, which has no redundancy, is trading increased risk (with each disk added) for the gained space as if any one disk dies it can take the entire pool with it. So with 4 disks in raid0 you are running 4 * the risk associated with a single disk failure, but for the whole pool. Raid1 is a better bet all around, unless of course the primary concern is available space given the above.

@bhsramz

… status showed it was 100% finished. In the screenshot below, this is Balance Id 1.

Yes this can happen with our current percentage reporting, but it may have been improved from that version of Rockstor.

@bhsramz

I thought the Percent finished field may have not been updated,

Potentially. We definitely need more work in that area.

Balance times can be drastically improved by disabling quotas but again only stable channel can handle that.

@bhsramz

My MBP was still unable to connect to the share,

3.9.1-0 was very weak on displaying errors / unmounted / degraded pool states. You may very well have issues that would be made clearer on a newer version given the above referenced issues and a number of others that have also been addressed.

@bhsramz

Any thoughts on how I can verify that the Balance is still running

Yes execute the following command via ssh and run as the root user, it’s what Rockstor uses internally:

btrfs balance status /mnt2/homeshare

If as I suspect you have a dodgy drive, potentially indicted by the io (In/Out) errors, then your pool is in a rather precarious situation and with raid0 it has no redundancy with with to repair itself.

I can’t spend much time on this but the output of the following commands may help others chip in:

btrfs fi show

and

btrfs fi usage /mnt2/homeshare

and

btrfs dev usage /mnt2/homeshare

The last of which I’m actually in the process of adding to Rockstor currently.

@bhsramz

… but would it be better to remove the disk from the pool and retry adding it?

That would be a very risky practice and would likely cause more problem as I suspect you have a flaky disk and it would first be wise to identify which one it is. And if it is the newly added one then the situation is all the more precarious especially given raid0. Raid0 is not a good choice in most scenarios except where the pool is disposable.

@bhsramz

I need to get backups resumed soon …

Then I would move to identify which is the problematic drive, from the above commands, and by looking to their S.M.A.R.T info, see our S.M.A.R.T howto. And by running the drives self test and checking their smart error log entries there after. Then once you have identified the problematic drive, physically remove it. At this point your pool is toast as raid0 has no redundancy so you will have to wipe the pool and all it’s remaining members and start over by creating a new one with the remaining members. That way you are up and running the quickest.

@bhsramz

I could not find any information that suggested a soft restart would be an issue while the Balance was in progress

That’s right, a balance, once initiated, should resume where it left off during shutdown/reboot upon the next boot. But in your case it looks very much like you have an filesystem issue and or a hardware issue. Again command outputs (or newer rockstor version) should help to pin this down.

Hope that helps and thanks for helping to support Rockstor via you stable channel subscription, but I would advise using that subscription as the stable channel updates are a significant improvement on non updated iso installs: especially when things go wrong ie drive / pool errors which in turn can lead to unmounted pools or pools going read only.

[bhsramz]

@phillxnet Hello, and thanks for your assistance, again. Concurrent development and support are hard to balance, so kudos to you and the rest of the Rockstor team. I am a member a a 5 person analytics team, so I can relate. Hopefully I can give enough information to help you or other forum members point me in the right direction here.

You are correct, I did have a suspected failing system disk. I replaced the system disk, and upgraded the processor in my NAS box before adding the 4th disk to my RAID0 pool. I realize that RAID0 is much less safe than RAID1, and my intent was to move to RAID1 now that I have enough storage capacity to trade for redundancy. These Hitachi drives are NAS rated, and my home NAS is put through very light workloads, so I would be surprised if I was suffering a disk failure in the Pool.

When I installed the new system disk, I did a fresh install from ISO. However, when I enabled stable updates and got up to the most recent release, I started having the same symptoms as before: very slow response and GET/POST errors in the UI when loading any page or performing any operation. I can try to update again, but I wanted to get the appliance back up and running before dealing with the update issues. After a reinstall I was able to get backups completed from all of my devices, so now I am in the process of expanding the Pool and moving to RAID1. Do you still recommend updating Rockstor before anything else?

Here is some output to chew on:

[root@rockstor ~]# btrfs dev stats /mnt2/homeshare
[/dev/sda].write_io_errs 0
[/dev/sda].read_io_errs 0
[/dev/sda].flush_io_errs 0
[/dev/sda].corruption_errs 0
[/dev/sda].generation_errs 0
[/dev/sdb].write_io_errs 0
[/dev/sdb].read_io_errs 0
[/dev/sdb].flush_io_errs 0
[/dev/sdb].corruption_errs 0
[/dev/sdb].generation_errs 0
[/dev/sdc].write_io_errs 0
[/dev/sdc].read_io_errs 0
[/dev/sdc].flush_io_errs 0
[/dev/sdc].corruption_errs 0
[/dev/sdc].generation_errs 0
[/dev/sdd].write_io_errs 0
[/dev/sdd].read_io_errs 0
[/dev/sdd].flush_io_errs 0
[/dev/sdd].corruption_errs 0
[/dev/sdd].generation_errs 0

[root@rockstor ~]# ls -la /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 280 Nov 17 16:55 .
drwxr-xr-x 6 root root 120 Nov 17 16:55 …
lrwxrwxrwx 1 root root 9 Nov 17 16:55 ata-Hitachi_HUA723020ALA641_YFGRZTHA -> …/…/sda
lrwxrwxrwx 1 root root 9 Nov 17 16:55 ata-Hitachi_HUA723020ALA641_YGG3KZEA -> …/…/sdc
lrwxrwxrwx 1 root root 9 Nov 17 16:55 ata-Hitachi_HUA723020ALA641_YGHH5YGA -> …/…/sdb
lrwxrwxrwx 1 root root 9 Nov 17 16:55 ata-Hitachi_HUA723020ALA641_YGJ0DGSA -> …/…/sdd
lrwxrwxrwx 1 root root 9 Nov 17 16:55 ata-Maxtor_6L160P0_L327KL3G -> …/…/sde
lrwxrwxrwx 1 root root 10 Nov 17 16:55 ata-Maxtor_6L160P0_L327KL3G-part1 -> …/…/sde1
lrwxrwxrwx 1 root root 10 Nov 17 16:55 ata-Maxtor_6L160P0_L327KL3G-part2 -> …/…/sde2
lrwxrwxrwx 1 root root 10 Nov 17 16:55 ata-Maxtor_6L160P0_L327KL3G-part3 -> …/…/sde3
lrwxrwxrwx 1 root root 9 Nov 17 16:55 wwn-0x5000cca223ca73bf -> …/…/sda
lrwxrwxrwx 1 root root 9 Nov 17 16:55 wwn-0x5000cca224c1a09d -> …/…/sdc
lrwxrwxrwx 1 root root 9 Nov 17 16:55 wwn-0x5000cca224d4ff68 -> …/…/sdb
lrwxrwxrwx 1 root root 9 Nov 17 16:55 wwn-0x5000cca224dc5dd7 -> …/…/sdd

[root@rockstor ~]# btrfs balance status /mnt2/homeshare
Balance on ‘/mnt2/homeshare’ is running
85 out of about 890 chunks balanced (86 considered), 90% left

[root@rockstor ~]# btrfs fi show
Label: ‘rockstor_rockstor’ uuid: 3ee753c6-4ba9-4831-a0e2-05cbb009598a
Total devices 1 FS bytes used 2.60GiB
devid 1 size 144.43GiB used 5.04GiB path /dev/sde3

Label: ‘homeshare’ uuid: 67ca3f52-4b5a-4d7c-894f-8564066e9d4a
Total devices 4 FS bytes used 2.87TiB
devid 1 size 1.82TiB used 887.22GiB path /dev/sda
devid 2 size 1.82TiB used 887.22GiB path /dev/sdb
devid 3 size 1.82TiB used 887.22GiB path /dev/sdc
devid 4 size 1.82TiB used 287.50GiB path /dev/sdd

[root@rockstor ~]# btrfs fi usage /mnt2/homeshare
Overall:
Device size: 7.28TiB
Device allocated: 2.88TiB
Device unallocated: 4.40TiB
Device missing: 0.00B
Used: 2.87TiB
Free (estimated): 4.40TiB (min: 4.40TiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 8.09MiB)

Data,RAID0: Size:2.87TiB, Used:2.87TiB
/dev/sda 885.00GiB
/dev/sdb 885.00GiB
/dev/sdc 885.00GiB
/dev/sdd 287.00GiB

Metadata,RAID0: Size:7.06GiB, Used:5.88GiB
/dev/sda 2.19GiB
/dev/sdb 2.19GiB
/dev/sdc 2.19GiB
/dev/sdd 512.00MiB

System,RAID0: Size:96.00MiB, Used:208.00KiB
/dev/sda 32.00MiB
/dev/sdb 32.00MiB
/dev/sdc 32.00MiB

Unallocated:
/dev/sda 975.80GiB
/dev/sdb 975.80GiB
/dev/sdc 975.80GiB
/dev/sdd 1.54TiB

[root@rockstor ~]# btrfs dev usage /mnt2/homeshare
/dev/sda, ID: 1
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 287.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 975.80GiB

/dev/sdb, ID: 2
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 287.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 975.80GiB

/dev/sdc, ID: 3
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 287.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 975.80GiB

/dev/sdd, ID: 4
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 287.00GiB
Metadata,RAID0: 512.00MiB
Unallocated: 1.54TiB

A couple of observations on the command line output above: 1) system datetime appears to be off…not sure how 2) all the disks seem to be error-free, so that is good 3) it appears the balance operation is ongoing and 10% complete.

Given these observations, should I just let it keep running and hope it completes successfully this time, or should I go ahead and try to get updated to the latest stable release and then complete the balance? Thanks in advance for any thoughts!

[phillxnet (Philip Guyton)]

@bhsramz Thanks for the update and thorough response.

Given these observations, should I just let it keep running and hope it completes successfully this time …

I would say yes. It’s currently doing what you asked, at the btrfs level at least, and given you can observe it’s progress via successive:

btrfs balance status /mnt2/homeshare

You can keep an eye on it.

It would go much faster with quotas disabled but this would break Rockstor’s Web-UI at the non stable version. Lets see how it goes and take it from there. If it ends up failing again you can then update to stable, disable quotas from the Web-UI, and give it another go. Also of note here is that both update channels will provide a moderate kernel and btrfs progs update from 3.9.1-0 (iso) which is quite important, so there’s that element to either update channel. But still lets leave it be for the time being as you would have to update to get that kernel active anyway, which would again interrupt this balance.

I started having the same symptoms as before: very slow response and GET/POST errors in the UI when loading any page or performing any operation.

Lets circle back around to this one (it may just be quota related again as later versions try and keep better track of these; but only if they are enabled). I vote to open this in another thread when we get to it to keep this thread on original topic.

Thanks for your exposition of your choices, nicely put. And yes it does appear that your pool is healthier than I feared: which is good news and a relief.

Hope that helps.

[bhsramz]

Well, the balance operation appears to have completed successfully! It definitely did not take as long as I expected given that it showed 90% remaining earlier today. The Web UI now shows 100% complete, and I am now able to mount to shares and backup, which I am doing now.

From the command line, no balance running:

[root@rockstor ~]# btrfs balance status /mnt2/homeshare

No balance found on ‘/mnt2/homeshare’

I do wonder though, why are the disks still not “balanced”? My original 3 disks have 2 RAID0 data records, for a total of 884GB each, while the new disk only has a single 286GB RAID0 data record. Is this expected behavior after adding a disk to an existing Pool? If I do decide to move forward and convert this Pool to RAID1, should I expect any issues with what I see as unbalanced disks? I realize this is probably exactly what should happen in this scenario, and my ignorance of btrfs is just causing some heartburn, but I want to be sure before I proceed to update Rockstor and convert the Pool to RAID1.

Pool usage after the balance completed:

[root@rockstor ~]# btrfs dev usage /mnt2/homeshare
/dev/sda, ID: 1
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 286.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 976.80GiB

/dev/sdb, ID: 2
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 286.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 976.80GiB

/dev/sdc, ID: 3
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 598.00GiB
Data,RAID0: 286.00GiB
Metadata,RAID0: 1.69GiB
Metadata,RAID0: 512.00MiB
System,RAID0: 32.00MiB
Unallocated: 976.80GiB

/dev/sdd, ID: 4
Device size: 1.82TiB
Device slack: 0.00B
Data,RAID0: 286.00GiB
Metadata,RAID0: 512.00MiB
Unallocated: 1.54TiB

[root@rockstor ~]# btrfs fi usage /mnt2/homeshare
Overall:
Device size: 7.28TiB
Device allocated: 2.88TiB
Device unallocated: 4.40TiB
Device missing: 0.00B
Used: 2.87TiB
Free (estimated): 4.41TiB (min: 4.41TiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)

Data,RAID0: Size:2.87TiB, Used:2.86TiB
/dev/sda 884.00GiB
/dev/sdb 884.00GiB
/dev/sdc 884.00GiB
/dev/sdd 286.00GiB

Metadata,RAID0: Size:7.06GiB, Used:5.88GiB
/dev/sda 2.19GiB
/dev/sdb 2.19GiB
/dev/sdc 2.19GiB
/dev/sdd 512.00MiB

System,RAID0: Size:96.00MiB, Used:208.00KiB
/dev/sda 32.00MiB
/dev/sdb 32.00MiB
/dev/sdc 32.00MiB

Unallocated:
/dev/sda 976.80GiB
/dev/sdb 976.80GiB
/dev/sdc 976.80GiB
/dev/sdd 1.54TiB

Again, thanks @phillxnet for the support! Looks like I am out of the woods now, so I will finish backing up devices and plan next steps. As you suggest, I will open another thread to troubleshoot issues with updates if they emerge again. Let me know if there is anything special I should do to get back on stable updates.

[phillxnet (Philip Guyton)]

@bhsramz

Well, the balance operation appears to have completed successfully!

That’s great.

I do wonder though, why are the disks still not “balanced”? My original 3 disks have 2 RAID0 data records, for a total of 884GB each, while the new disk only has a single 286GB RAID0 data record.

I wouldn’t worry about that. If the balance succeed and the drive is not included in the pool’s tally then you are pretty much set. It may be that a subsequent balance would ‘do more’ but not really worth it as the system will favour the more empty drives anyway so should even itself up more as it’s used. I have more experience with btrfs raid1 than raid0 as it goes but it look OK to me.

If I do decide to move forward and convert this Pool to RAID1, should I expect any issues with what I see as unbalanced disks?

I don’t expect so. And with raid1 balances I have seen them pretty much end up even as one would expect. We are also looking at older btrfs here so things are improving all the time. We are working on moving our kernel and btrfs-progs maintenance / updates to our base distro (by moving distros) so in time we should inherit current version of them both.

I’d also check again the output of:

btrfs dev stats /mnt2/homeshare

just to be sure.

And out of curiosity I’d also look to the error referenced in the Web-UI report of that balance. I.e take a look at the system log, either via System - Logs Manager and or via

journalctl

again just to be aware of it’s cause. It may pertain more to the way Rockstor tracks the job, and the reboot during that task upsetting that element. I.e. if Rockstor finds a balance running that it has no current task open for then it simply updates the last known job with the status of the currently running balance: as that is the truth of the matter. And this allows for us to report on balances that are command line initialised. Although that’s a little from memory and I’m not sure when that was released but it was ages ago. This would account for the id 2 job being updated in it’s progress when the reboot may have upset at least Rockstor’s task tracking, with it failing over to simply updating the last known job, and possibly not in turn updating the message (again it’s pretty old code).

You outputs here are helpfull for a pending feature I’m currently considering, the surfacing in the pool details page devices table of the individual disk usage.

Let me know if there is anything special I should do to get back on stable updates.

As you have expressed / experienced performance issues with latest stable I would say it’s worth disabling quotas in case that it the cause in your case. Once you have updated of course. Another option, in case you are caught between really old code (3.9.1-0 iso) and newest (latest stable) being unworkable (at least until resolve the why) on your system then it’s still very much worth trying last released testing channel. That is still pretty old but is much improved from 3.9.1-0 (by around 14 fixes) and equates to (from memory) 3.9.2-0 release. But latest stable is then around 44 fixes and or features on top of that.

As to upgrading caveats if one is already on testing channel updates and fully updated, and then changes to stable there is a bug (fixed in an update you can’t install via Web-UI due to this bug) which can end up showing available version as installed. So to test for sure do

yum info rockstor

and if the Web-UI is wrong about installed version then just do

yum update rockstor

Not necessary if going straight to stable but affects testing to stable transition. Just so you know.
You can always ask again here if need be. Bit if a chicken and egg one that and come our next iso will be gone.

Also worth noting that any Rockstor update, brings with it all the CentOS updates: which for our current iso are pretty massive. So you must be patient and the Web-UI will disappear for a bit while this is happening. Just don’t reboot while this is happening and be patient, otherwise you may end up creating further difficulties.

Oh and run an extended S.M.A.R.T self test on your system disk before the upgrade. Just to get the results from that.

Hope that helps and chuffed you appear to be out of the woods / weeds.

[phillxnet (Philip Guyton)]

@bhsramz Re (again):

However, when I enabled stable updates and got up to the most recent release, I started having the same symptoms as before: very slow response

I’ve been thinking about this comment. We should, as planned, address in a new thread but just to let you know that it has prompted me to look into this and I’ve found something that can be done much more efficiently. I’m rolling the improvement into my current issue so hopefully that should help things some in time: assuming it passes review. No promises it was your ‘slow down’ issue but at least it’s something.

Thanks for bringing this to my attention, it is the case that we simply do more in later Rockstor versions but I’m keen that we keep thing workable on reasonable hardware obviously so will try and whittle away at performance issues as I go. We are also due to move to Python 3 from legacy Python which should in turn allow us to update a bunch of other things so I’m hopeful we will get a boost across the board in time from those updates.

[iecs]

I m running a rockstor 3.9.1-16 on esxi vm, with the system on a vmfs disk and other hhds all pass-through directly to vm
when I re-balance a pool after adding two new disks, with the raid level remaining the same
I got the error cannot start banlance

"BTRFS error (device sda3 unable to start balance with target data profile 128) "

But, sda3 is a partition of the root virtual disk as below

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 14448639 6711296 82 Linux swap / Solaris
/dev/sda3 14448640 134217727 59884544 83 Linux

and the instance got down

then I restart the instance and logged in got the error from the balance UI

Error running a command. cmd = btrfs balance start -mconvert=raid5 -dconvert=raid5 /mnt2/st5000danpan. rc = 1. stdout = [’’]. stderr = [“ERROR: error during balancing ‘/mnt2/st5000danpan’: Invalid argument”, ‘There may be more info in syslog - try dmesg | tail’, ‘’]

yeh, raid 5 is not production yet but I had it running for about one month till I switch from Raid 1

and the newly added two disks are both 8T

while the old 3 disks of the pool are all 5T

when I tried dmesg | tail, got:

[ 1531.737675] BTRFS info (device sdc): relocating block group 14331865464832 flags data|raid5
[ 1563.305257] BTRFS info (device sdc): found 226 extents
[ 1575.943863] BTRFS info (device sdc): found 226 extents
[ 1576.878599] BTRFS info (device sdc): relocating block group 14327570497536 flags data|raid5
[ 1609.942144] BTRFS info (device sdc): found 235 extents
[ 1621.991289] BTRFS info (device sdc): found 235 extents
[ 1622.709741] BTRFS info (device sdc): relocating block group 14323275530240 flags data|raid5
[ 1655.922741] BTRFS info (device sdc): found 426 extents
[ 1672.241578] BTRFS info (device sdc): found 426 extents
[ 1673.097908] BTRFS info (device sdc): relocating block group 14318980562944 flags data|raid5

so the problem may be related to disk sdc which is one of the newly added 8T hdd

when writing this post I start a new balance force to the pool

will report later

about the extents

BTRFS info (device dm-1): found 1 extents

so maybe just reboot the system is ok?

[phillxnet (Philip Guyton)]

@iecs Welcome to the Rockstor community.

"BTRFS error (device sda3 unable to start balance with target data profile 128) "

But, sda3 is a partition of the root virtual disk as below

Yes this is very strange, but:

then I restart the instance and logged in got the error from the balance UI

Error running a command. cmd = btrfs balance start -mconvert=raid5 -dconvert=raid5 /mnt2/st5000danpan. rc = 1. stdout = [’’]. stderr = [“ERROR: error during balancing ‘/mnt2/st5000danpan’: Invalid argument”, ‘There may be more info in syslog - try dmesg | tail’, ‘’]

This could just be btrfs’s answer to not starting a balance when one is ongoing. I’m assuming here you haven’t removed a drive. As in that case there is an internal balance that is kicked off that doesn’t show up in the usual balance status commands. But all the same will block ongoing balances.

and the newly added two disks are both 8T

Did you add the disks via Rockstor’s Web-UI.

when I tried dmesg | tail, got:

[ 1531.737675] BTRFS info (device sdc): relocating block group 14331865464832 flags data|raid5
[ 1563.305257] BTRFS info (device sdc): found 226 extents
[ 1575.943863] BTRFS info (device sdc): found 226 extents
[ 1576.878599] BTRFS info (device sdc): relocating block group 14327570497536 flags data|raid5
[ 1609.942144] BTRFS info (device sdc): found 235 extents

This just looks like a regular balance is in play. So just wait out this balance. But note that as you have already rebooted at least once during what looks like an ongoing balance it has resumed upon reboot. But their is a recently fixed bug in btrfs, not in our CentOS offerings, where if a balance is resumed it will skip a section of the volume, i.e. some of the data/metadata will not be moved over to the new raid levels. But you haven’t changed them but you are running the weaker side of btrfs, ie the parity raid levels of 5 or 6. These are less well proven in their ability to repair / modify.

In short your balance looks to be in play. Incidentally we do now have monitoring of a sort within our Stable releases for this within our Pools details section. But the stable release should still be able to report a non internal balance (adding disks) via the balance tab within the pools details section.

so the problem may be related to disk sdc which is one of the newly added 8T hdd

This doesn’t look currently like a problem though, just a balance in progress. And note that btrfs will reference one of the pool members when it can sometimes mean the pool at large.

… when writing this post I start a new balance force to the pool

This may have complicated things still further but likely just delayed the ultimate finish of a balance. Once the log finishes with this normal balance output try doing another balance from the Web-UI and be patient while that finishes. Incidentally if you turn quotas off these balances can go very much quicker but balance disabled in no supported in our older testing channel.

so maybe just reboot the system is ok?

Give it time to finish what it is going. Balances, especially with quotas enabled and even more so with the parity btrfs raid levels of 5/6 can take a very long time. All depends on the amount of data you have.

Hope that helps and let us know who it proceeds.

The sda3 issue is likely a bug in our much earlier device management code in the CentOS testing channel. More on this in your other post.

[iecs]

all my operations are through web-ui

and the raid now works fine, but I cannot remove disks from it as I created a new issue Unknown internal error doing a PUT to /api/pools/2/remove.

[phillxnet (Philip Guyton)]

@iecs Hello again.

@iecs

but I cannot remove disks from it as I created a new issue Unknown internal error doing a PUT to /api/pools/2/remove.

Yes, this was a known limitation within the Web-UI of Rockstor to report / recognise disk removal progress if long lived, which they often are. You initial disk removal request is most likely in the process of executing, hence the log report, it’s just that while it’s running the Web-UI is not aware of this state until it is asked to do the same again. It’s now fixed in the stable channel as details in my reply to your specific thread.

Thanks for the report and for detailing your findings in a seperate report, much appreciated.

Glad you otherwise seem to be OK, and hopefully we should have more appropriate testing offerings soon.

Hope that helps and thanks again for your engagement.