One disk failed -> Rockstor messed up

Troubleshooting
1

Sorry for the provocative title. I am running into so many issues with btrfs, I feel like posting under a different username so as not to appear incompetent…

So, here is my story:
I had 4 disks (2, 1.5 ,1.5 ,0.5 TB), that I wanted to use for a raid1 or maybe raid5 setup as home-nas, braswell based. Since two of these disks (each 1.5 TB) were used in an existing raid1 with all my important data, I wanted to do a step-by-step build of the btrfs array. I have some experience on btrfs, but only with single devices and non crucial data- so far without any problems.

I started to use the 2 and 0.5 TB in single mode, copied all my data over and then proceeded to integrate the first 1.5 TB disk into the array - the second one I kept as fallback if anything should go wrong.

  1. Issue - I ran into the extent_tree problem, causing my server to crash a few times (I assume that was the reason), So I think my balance might have never completed after converting this now 3-disk-array to raid1- technically, that shouldn’t be a serious problem for my

  2. Issue - my freshly added 1.5 TB disk appears to have died. Rockstor didn’t report anything, but silently failed to mount the array. Maybe it was because of a few hard resets I/Linux had to do, maybe it was a random failure, but my array just wouldn’t mount. After a while I noticed one disk emitting clicking noises in certain conditions, fdisk -l also sometimes didn’t recognize that device. (I restarted a couple of times to narrow down the issue) Removing the disk and putting it in an external case just told me the disk was not responding reliably so a SMART test wasn’t really indicated.

In order to make use of btrfs famed recovery abilities and to keep this as hassle free as possible, I decided to simply back up my important data, and adding my fourth, working drive to the array. Only I couldn’t, because now I ran into the

3. issue ;tldr; - Apparently, under certain circumstances when a disk goes missing one cannot mount a btrfs in rw mode, so you cannot even add a new disk, even though there’s no data missing. (There shouldn’t be anything missing in my case and I can see and copy my files in ro mode) So, I didn’t lose any data- nice. But I can’t do anything to progress from here, because of the ro-mode since. Reading up a little, this might be caused by my converting and balance-‘pausing’ earlier as well

  1. issue - I am unable to remove the faulty filesystem from rockstor because of snapshots on some shares - read-only->so no snapshot deletion-> no share deletion-> no pool deletion. I could do with simply nuking the remaining filesystem, and rebuild the array from scratch, but all the names, especially my share names are then still in the Rockstor database and “taken” and I’d have to reconfigure the clients I already set up. I’d rather reinstall rockstor.

So, many of my errors are simply btrfs or hardware related, and I guess we all now what we’re in for using an “unstable” filesystem - I still think it’s a good solution and I also like rockstor, but I am looking for ways out of here - I’d lbe content with the possibility to remove the shares from within rockstor - maybe an option to delete an entire pool. I’d like to avoid reinstalling rockstor from scratch…

Thanks for reading :stuck_out_tongue_winking_eye:

2

If you manually delete subvolumes(Shares), snapshots and filesystem(Pool), and then refresh the UI, the db should get updated. It may take a few seconds, but the db state is checked with system state and it’s synced in cases like this. Give it a try if you can.

3

Thanks for your post and all the detail. You appear plenty competent to me. We have an open issue to implement a new feature for alerts, where failures like drive drops, mount failures etc… are properly notified to the user.

Was your Pool still in Single raid profile or did you start having issues after successfully converting to RAID1?

4

Well, the last I checked, I think it still had to balance about 90.0/1000 GB, so most of the data should have been raid1, parts were still in single, but they should have been on the still working disk.

Since btrfs would continue the balance at every mount, and I experienced crashes because of that, rockstor wouldn’t run for more than a few minutes at last. The extent_tree issue was while balancing, the broken disk presumably after being almost done, the ro-mount issue while parts of the data should have been in 9% single-mode on a working disk and in 91% raid1 spread over 3 disks, one of which was broken.

5

This sounds too good to be true, but I am not sure we understand each other completely here: Since the filesystem is read-only, and I have no way of mounting it any other way (mount -o recovery,degraded doesn’t work) I can’t remove anything, not even from the console - if by “delete!” you mean to just dd/mkfs over the old filesystem, yes I can certainly do that, but I doubt this will clean up the mess in the UI/db EDIT: Especially remembering

I’d expect the same result when dd’ing over an existing pool

6

despite my doubts I went ahead and dd’d the first few sectors of the two remaining disks and rebuild the data I had since copied over to the third. Not sure if that was a smart move, but hey, you were partly correct (unfortunately, so was I):

Rockstor indeed removed the remaining disk from the pool, leaving me with just the pool-name in the ui/db. I could then reattach the two disks to a new array, basically repeating the steps above. So far no errors, about 150/1000GB balanced.

Well, the pool’s name is still taken, the shares still show and are undeletable
(“Error! Share(Personal) cannot be deleted as it has snapshots. Delete snapshots and try again” or
"Error! Failed to delete the Share(Backups). Error from the OS: ‘NoneType’ object has no attribute ‘name’")
as are said snapshots (“Error! ‘NoneType’ object has no attribute ‘name’”)
I’ll add that info to the bugtracker, as it is related.

7

I just thought of something: If I were to rename my new btrfs to its old name on the commandline, maybe even recreate the subvolumes, would rockstor accept that as it’s old pool? Or does it work with IDs internally?
That might also be a workaround for the aforementioned bug

8

This is why I am a bit wary of building out the rockstor setup. Btrfs has a bug when using raw devices. If the first device in a pool fails the whole pool could be lost. It does not happen if using pools on partitioned devices. I had replicated the issue several times in vm and physical setup, after loosing almost 2tb on drive failure.

9

Hm, do you have any reference on that? For the record, I didn’t lose any data. I had trouble converting the array, but the redundancy worked as it should (I think), just the rebuilding part had a major bump on the road.

10

Not sure what you mean by reference.
I had lost the data, and I was able to reproduce the issue several times on vm setup and real pc.
The issue is not 100% consistent. I can not repriduce it every time, but it’s exist.
Basically when you build a pool with raw devices,
In my case I had tested with 3 and 4 devices using raid1 level, if the first device in the pool fails, when I say first device I mean device that was used first in the list for mkfs command, the whole pool becomes un usable. Can’t read it, can’t mounted, can recover it. Nothing. All data is gone.
When using partitions to build out a pool, it works as expected. I could not fail a pool build on partitioned devices , not even once. Any device fails the pool is mountable and readable, I can replace the faild device no problem.

11

Answer: yes it does. I am not sure about all the consequences, but I just renamed the pool via “btrfs fi label” and was then able to delete my old snapshot and shares (they were all gone from the fs) and recreate them.

12

@Vl1969 If you could find a way to replicate the failure you describe re whole devices vs partitions then that would certainly be of great value to the btrfs devs over at the linux-btrfs mailing list and by extension to the whole btrfs community. If nothing else then there would at least be a reference of this having been reported / found.

Oh and I see you have made quite a few posts but have yet to be welcomed so Welcome to the Rockstor community.

1 Like
13

Thanks phillxnet, actually I have been welcomed when I created an account, but thank you anyway.
I did submit an issue couple of month back to the btrfs bug tracker.
Not sure what become of it, but it is there.
Aa for replicating the bug, unfortunately it is not 100% consistent. I have tried it on vms and got the failure 3 out of 10/12 times. When using raw virtual disks. I do not have a lot of real hardware that I can use for testing, but when I had a setup I replicate the issue 2 out of 5 times I tried.
Basically, when building a pool with mkfs.btrfs command using 2+ devices, sda, sdb, sdc
If sda fails, you can loose the whole pool and data on it. It’s like the rest of the disks are empty all of a sudden. No mount, nothing. Loose Any other device in he pool and it’s ok, you can still mount the pool and replace the disk and re balance as expected. When using partitioned devices, as in sda1, sdb1 etc. All is working as expected for any drive in the pool.

14

In these cases, have you ever tried to mount the array in ro-mode? As I mentioned in my post, I tried mounting in degraded mode, but was stopped by a message complaining about a bad fs and missing codepage- it basically looked like there was no fs at all.

15

Yes I have tried all possible recovery options I could find. It was like the disks were totally empty. No fs , no data, nothing. I could never replicate the effect on partitioned disks.