Drives not adding to pool after a failed or uncompleted Balance

soulwise · August 14, 2016, 7:01pm

Hi there.

I am new, very new in all this btrfs stuff, so please excuse me if this info is not new or not important.
I installed latest Rockstor on a virtual machine, then updated it using the unstable train.
After that, wanting to see what this OS can handle in terms of user abuse i began to do all kind of nasty stuff.
The Rockstor OS handled itself beautifully becoming my prime candidate in replacing my current FreeNAS Box, but…
After a few rapid changes of raid level and adding or subtracting disks the balances associated with few of the events never finished. Their status is 0% and i do not think they will ever change.
Now, when in want to add a few extra drives, the OS keeps telling me

Traceback (most recent call last):
File “/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py”, line 34, in run
client, addr = self.socket.accept()
File “/usr/lib64/python2.7/socket.py”, line 202, in accept
sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable

The current status of the balance jobs is:

6 finished August 14th 2016, 3:35:02 pm 100
5 started August 14th 2016, 3:34:48 pm 0
4 started August 14th 2016, 3:33:29 pm 0
3 finished August 14th 2016, 2:07:57 pm 100
2 finished August 14th 2016, 1:36:35 pm 100

You can clearly see that 4 and 5 are at 0% and the percentage never changes even after reboot, new balance job, or scrub.

Also no new balance can’t be initiated even if i try the force option:

Traceback (most recent call last):
File “/opt/rockstor/eggs/gunicorn-0.16.1-py2.7.egg/gunicorn/workers/sync.py”, line 34, in run
client, addr = self.socket.accept()
File “/usr/lib64/python2.7/socket.py”, line 202, in accept
sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable

Hope it helps and sorry for my poor english.
Best Wishes.

phillxnet · August 15, 2016, 10:58am

@soulwise Welcome to the Rockstor community.

Yes the balance feedback in the Web-UI is definitely due to get an upgrade as currently it fails to report in-progress balance tasks. However there is a pending code change that should make things a little better in this department, although there is more to be done here in the future but at least it’s a start.
The pull request reference is:

github.com/rockstor/rockstor-core

balance not updated during execution plus fs unittests (initial commit) #1405 #1022

rockstor:master ← phillxnet:schakrava-1022_btrfs_unittests

opened 04:52PM - 05 Aug 16 UTC

phillxnet

+946 -30

Fixes #1405 - When a Web-UI initiated balance is in progress and the Pool page i…s refreshed there is no indication of the percentage of progress until the balance task initiated has reached 100%. Given balance tasks can be very long lived processes it is a better user experience to have an indication of the current status of percent complete. Prior behaviour is to only indicate status as started or complete. These alterations allow for in progress 'percentage finished' updates via 'Pools' page refresh. They also extend the balance status reporting to include running, pausing, paused, cancelling, and cancelled, however these status updates are only available via a page update. But this at least paves the way for future improvements on balance status reporting. Also note that with the addition of these changes it will be easier to accommodate and report accordingly the status intervention affected by a balance pause/resume button and a balance cancel button, see #1413. Also partially addresses elements of #862. Initial commit on my side for #1022: initially the focus of this pr but due to changing circumstances the above issue also became enrolled, hence the dual personality of the pull request. All commits in this request that do not pertain to the above #1405 issue uniquely relate to furthering unit test coverage and do not affect user run code. But given we are now receiving pull requests on some of what is already covered by unit tests withing this pr (if sparsely as yet) it may be appropriate for these to be merged sooner rather than later so that they might serve to aid in these recent pull requests, most notably now closed #1408 and as yet pre-review pr #1415. The test for the latter will have to be modified to accommodate the scope change introduced by #1415 (if merged). @schakrava Apologies for making this far more messy than either of us would like (dual target pr) but given the fs/btrfs.py related changes are going to be progressively more difficult to merge with recent and ongoing pr's on this code, I though it best to merge sooner rather than later. I can there after revert to a more single issue focused pr model and enact on your feedback re unit test advise. Most btrfs.py changes pertain to comment additions where they are not related to the resolution of balance reporting #1405. I am aware that it is almost completely 'happy path' testing stuff and will extend to failure cases upon next engaging with these tests. ~~Ready for review.~~ EDIT: See below commends on progress with splitting this pr into 2 more focused pr's as more recently requested.

and the issue that lead to this first attempt to ‘upgrade’ this behaviour is:

github.com/rockstor/rockstor-core

Web-UI initiated balance status not updated during execution

opened 05:47PM - 23 Jul 16 UTC

closed 03:42AM - 30 Sep 16 UTC

phillxnet

enhancement

Thanks to @grebnek in the following forum thread for bringing attention to this …matter. In an effort to reproduce @grebnek's raid level change update issue I noticed that the balance status fails to update in the Web-UI. https://forum.rockstor.com/t/finally-build-my-own-nas/1811 Steps to reproduce (Rockstor version 3.8-14.07):- Had a single disk pool with single raid level. Resize Pool - Add Disk - Yes to change RAID level - selected raid1 from drop down and ticked additional disk - Next Refreshed Web-UI to find Pools - "rock-pool" - Balances tab to read: ID=2, Status=started, Start Time=July 23rd 2016, 5:36:39 pm, Percent finished=0, Errors=(blank) Expected it to reflect a status of "running" and the relevant percentage done. The command we use to track this progress executed on the command line gave: ``` btrfs balance status /mnt2/rock-pool/ Balance on '/mnt2/rock-pool/' is running 73 out of about 117 chunks balanced (74 considered), 38% left ``` Once completed the Web-UI correctly indicates status=finished, Percent finished=100. Related function fs/btrfs.py balance_status(pool) Note that this issue differs from #1062 in that this balance was initiated from the Web-UI, not manually from the command line.

Could you get the current status of balance from the command line:

btrfs balance status /mnt2/pool-name-here

And paste the result in this thread.

That way we can confirm the current state.

As for prior balances, only one balance operation can take place on each pool so the readings that show prior balance task info as started with 0% complete are bogus and can safely be ignored. This area is definitely due for improvement and a start has been made here with possible additional future enhancements noted within the above pull request.

Hope that helps to at least settle you mind on this buggy balance reporting.

I’m afraid I don’t yet see why a new balance can’t be initiated but lets see what the above command returns first and maybe I or others can pin it down.

You could also try a balance from the command line but note that it may take quite some time, depending on the size and configuration of the pool and is not reported in the Web-UI but may help to clarify what’s going on.

btrfs balance start /mnt2/pool-name-here

See the btrfs wiki btrfs-balance page for other balance commands. There are plans to reflect command line initiated balance within the UI in the future and to include pause/resume and cancel functionality within the Web-UI also.

soulwise · August 15, 2016, 11:31am

The result of the command is:

ERROR: balance status on ‘/mnt2/Soulshare’ failed: Operation not permitted

No changes from last time. I can not add drives, i can not initiate new balances.
As i said, i am very new and i think i am not very well prepared to answer your questions on the software side so i think thai it will be in our best interest to give you access to the VM i am running on my machine to see for yourself. If you agree send me the details. I use Teamviewer.

Thanks!

phillxnet · August 15, 2016, 11:37am

I’m afraid I can’t enter into such an endeavour right now but could you post the result of:

btrfs fi show

That way we will all have more info to work with. Thanks.

soulwise · August 15, 2016, 11:45am

btrfs fi show returns:

ERROR: cannot open /dev/sda3: Permision denied
ERROR: cannot open /dev/sdc: Permision denied
ERROR: cannot open /dev/sdd: Permision denied

phillxnet · August 15, 2016, 11:50am

@soulwise You will have to use the root user for these commands as they require root’s permissions.

Hope that helps, sorry I should have mentioned this, my mistake.

soulwise · August 18, 2016, 8:17am

Shortly after the events above, my VM crashed, no error , nothing. So, i wanted to investigate some more. I now have a “real” machine setup with the stable Rockstor Release available form the ISO image installed onto an SSD and three disks 1 TB each running for two days by now in Raid 1.

Remarcably stable despite my best efforts so i think today i will start to incorporate the box in my production workflow ( i shoot and edit videos for a living ) an see how it behaves.

Thank you.