"Failed to import any pool on this device", following upgrade/downgrade

Support
1

Hi

I was running 3.8.15 in a VM. I have 4 data disks passed through (i.e. not virtualised). I upgraded to 3.9.0 and got the error below. I have since wiped the OS disk and reinstalled both 3.8.15 and 3.8.16 but still get the same error when I try to auto import pools/shares.

Error

Failed to import any pool on this device(scsi-1ATA_WDC_WD30EFRX-68EUZN0_WD-WCC4N0PYDZT7). Error: Error running a command. cmd = [‘/bin/mount’, ‘/dev/disk/by-label/btrPOOL’, ‘/mnt2/btrPOOL’]. rc = 32. stdout = [‘’]. stderr = [‘mount: wrong fs type, bad option, bad superblock on /dev/sdb,’, ’ missing codepage or helper program, or other error’, ‘’, ’ In some cases useful info is found in syslog - try’, ’ dmesg | tail or so.', ‘’]

Web-UI screenshot

image.png1600×890 107 KB

…and in the console:

image.jpg1620×350 78.8 KB

Error Traceback provided on the Web-UI

Traceback (most recent call last): File "/opt/rockstor/src/rockstor/storageadmin/views/disk.py", line 380, in _btrfs_disk_import mount_root(po) File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 236, in mount_root run_command(mnt_cmd) File "/opt/rockstor/src/rockstor/system/osi.py", line 104, in run_command raise CommandException(cmd, out, err, rc) CommandException: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/btrPOOL', '/mnt2/btrPOOL']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/sdb,', ' missing codepage or helper program, or other error', '', ' In some cases useful info is found in syslog - try', ' dmesg | tail or so.', '']

###Also
Via SSH I tried:
mount -o rw,degraded,recovery /dev/sdb /mnt2/Files
…but got:
mount: mount point /mnt2/Files does not exist

The I tried:

mount -o rw,degraded,recovery, /dev/sdb /mnt2/
…but got (what I guess is the original error):
mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

So then I remembered that I had the noatime and autodefrag options on the pool, so I tried:
mount -o rw,degraded,recovery,noatime,autodefrag /dev/sdb /mnt2/
…but got the same:

mount: wrong fs type, bad option, bad superblock on /dev/sdb,
missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

###Any ideas
Does anyone have any ideas on how to recover my pool / data?

2

###And then
Looking through the forums here I have tried:
btrfs check /dev/sdb

…which produces:

parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
Ignoring transid failure
leaf parent key incorrect 1131794464768
Checking filesystem on /dev/sdb
UUID: 7aec0f3f-399d-4579-b3b9-a2acad00bac7
parent transid verify failed on 1131794071552 wanted 71645 found 70951
…
many many lines
…
Ignoring transid failure
parent transid verify failed on 1131790761984 wanted 71645 found 70951
Ignoring transid failure
Segmentation fault

###And then and then
And then I tired:
btrfs restore -D /dev/sdb /mnt2/
…which produced:

parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
Ignoring transid failure
leaf parent key incorrect 1131794464768
This is a dry-run, no files are going to be restored
We have looped trying to restore files in /Pictures/2 2006-2015 (also in Photos)/2007-11 Mexico too many times to be making progress, stopping
We have looped trying to restore files in /Pictures/2 2006-2015 (also in Photos)/2009-10-03 Africa too many times to be making progress, stopping
…
etc etc

3

@f2fbf60b Hello again. As a quick observation I would say you should not use “/mnt2” as your mount point as it is used to mount other filesystems within Rockstor. First create a fresh directory for this purpose (ie /mnt2/RETRIEVE): don’t use the /mnt2 point ‘raw’ as it will obscure the existing mount points already based there.

Also care must be take in this scenario and your best bet is to first attempt to retrieve the data, if important, and if you are unable to mount, even with the degraded option (which in combination with the rw option is often a one shot deal!), then your only option is the restore method and it required a mounted filesystem to dump whatever it finds (which /mnt2 is not).

https://btrfs.wiki.kernel.org/index.php/Restore

Hope that helps.

4

Thanks for the quick reply. I’ll look into the Restore. May have to buy an massive external HDD for the data dump.

5

@phillxnet I created a /mnt2/RETRIEVE folder and again tried
mount -o rw,degraded,recovery, /dev/sdb /mnt2/RETRIEVE

…but no luck. I get the same wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error error as before.

Could the fact that my pool has lzo compression and noatime + autodfrag options be causing this “bad option” error?

Re a btrfs restore
Looking back above I tried restoring to /mnt2/ which wasn’t mounted. I think this might be what was causing the loops described in the errors.
To use btrfs restore properly do you think it would be best to plug in a massive external HDD, create a share through the GUI and then run the restore command? Or might it be better to plug the 4 disks from my broken raid10 appliance (or maybe just 2) into my other Rockstor appliance and run the restore command to dump the data onto one of its shares?

6

Also there’s some banter on the net about btrfs-zero-log. Is this the lastist of last resorts. The wiki says you generally don’t need it. But when do you say to heck with it and potentially risk data loss?

7

For info I also did btrfs-find-root /dev/sdb giving:

parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
parent transid verify failed on 1131794464768 wanted 71645 found 70951
Ignoring transid failure
leaf parent key incorrect 1131794464768
Superblock thinks the generation is 71648
Superblock thinks the level is 1
Found tree root at 1131765989376 gen 71648 level 1
Well block 1131765088256(gen: 71647 level: 1) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
Well block 1131763793920(gen: 71646 level: 1) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
Well block 654934016(gen: 71644 level: 1) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
Well block 650543104(gen: 71643 level: 1) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
Well block 645890048(gen: 71642 level: 1) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
Well block 660471808(gen: 71638 level: 0) seems good, but generation/level doesn’t match, want gen: 71648 level: 1
…

So that seems promising.

8

@f2fbf60b Hello again:

Yes that sound like a plan. Pretty sure the destination can be anything that holds files so doesn’t have to be btrfs so could be what ever you already have and as long as you mount it (and not over the top of anything else’s mount point) then you should be able to use it as a destination/target for restore (unfortunately named really) to dump what ever it can find for your.

Don’t think so as what ever mount options you use will be the current options going forward, ie during that mount any future date will not be compressed for example. Do be carefull with that rw mount option along with the degraded though as it’s a one shot deal. Best not use that until you really have to as otherwise you may wast an oportunity later on to repair things if that is possible. Best first get stuff off with restore then try mounting degraded with ro and see if that works before you go the whole hog with rw,degraded.

You originally state:

Yet all your examples are attempting to mount via a single drive. If they are all members of the same pool then you can mount the pool by specifying any one of them. And I believe in some circumstances it can help to specify them all.

For context and to help forum members help you could you paste the output of:

btrfs fi show

Hope that helps.

9

@phillxnet Just finished the btrfs Restore. The process lasted about 3 days.

It kept pausing for “We seem to be looping a lot on /mnt2/DUMP/yourfile, do you want to keep going on ? (y/N/a):” a lot. So I have mostly been hitting y. Some file locations did actually look looped so I went with N, and it carried on.

After (mis)reading this page I did a btrfs-zero-log before I did the restore. I probably should have just tried the restore first. What worried me was that all the dry-run restores just returned loop errors, but I didn’t actually try a proper restore, so I don’t know if clearing the log helped, hindered or made no difference.

Anyway all my files seems to be restored. I have now destroyed the faulty pool, recreated and am now copying (cp -R) all my files back to their original locations.

Thanks for all your help.

10

@f2fbf60b

That’s great new, and your are welcome. Yes I think when in doubt ‘backup wise’ it is always best to go straight to the restore method as any further intervention after an error state can go either way with regard to helping or hindering. Best get the data off and then try stuff.

Thanks for the update and it definitely looks like things are improve in this regard going forward. I believe multi disk btrfs functionality is one of the focus points this year over at facebook and there are long pending improvements re disk failure in the pipeline also.