Escalating btrfs issue

Hey guys,

I’m having several issues with my Rockstor server at home. My setup:
1 HD (rockstor OS; 2"5 250gb)
1 HD (Shares/download drive 2.5" 500gb)
2x HD (DATA pool, 2 drives of 2TB each, setup in raid1 mirror)

It starts with an btrfs issue. This could be the main issue, causing all other issues so lets begin:

1: when the btrfs issue raises, I’m seeing an error message on screen starting with " BTRFS error (device sdc) parent transid verify failed on 986… wanted 183… found 183…", see below for the first screenshot:

I reinstalled Rockstor hoping this would fix it, though I didn’t have much luck with that. After getting Rockstor back up and running it kept failing. The weird thing is that the whole server crashes in a kernel panic, while the issue resides in the DATA pool. I would expect Rockstor to raise an error about the btrfs issue, but keep running, but it crashes all together, locks up and will not get back up and running unless I do a hard reset (switch power off without shutting down; ouch). This caused my Media Server database to become corrupted. Luckily I was able to restore this issue with some googling and an automatic backup.

I had a feeling that the btrfs issue was being caused by the attempt to access a file which is corrupted (not the media server database) during a file scan / database scan, so I let my Media server scan for the whole library while keeping an eye on my screen for error messages (none popped up). During testing I had every plugin running, except for one: Sonarr, so I turned my Media server off (precaution) and turned on Sonarr.

Surprising enough, each time after rockstor kernel panics, I’m unable to load Sonarr’s page, or do anything else for that matter. To get it up and running, I first have to disconnect the SATA cables from my 2 drives (from DATA pool), boot up Rockstor, switch off Sonarr plugin, shutdown Rockstor, connect 2 SATA cables, boot up Rockstor, uninstall Sonarr, delete Sonarr shares and do several attempts to reinstall Sonarr (I haven’t figured out the ‘sweet setup’ settings to get it working at first try; alternating with port number, virgin shares (aka completely empty shares, not used by/linked to other plugins), UID/GID numbers, or rebooting RockOn service because not all necessary shares are prompted during install).

Once I get a completely fresh install of Sonarr completed, add my media, and run a database scan/library scan, thats when I start getting messages. Oddly enough, now that I’m logging each step while typing this topic I only got 1 btrfs error message in the process: “BTRFS error (device sdc) parent transid verify failed on 986… wanted 183… found 183…” (btw: dev/sdc is one of the 2 TB HD), perhaps because I’ve got my Media sever turned off as rockstor crashing caused a corrupt database.
edit: turned my media server back on, along with Sonarr, ran library scans in both plugins, and no more error messages (other than the one btrfs error). Ill keep you posted about this one.

2: I cant update Rockstor to latest version (from 3.9.1-0), even though I activated my license. Once I start the update process I get a countdown for 5:00 min, but when it runs out and boot back up, the version is still stuck on 3.9.1-0 instead of the current latest version 3.9.2-10.

3: After attempting to update Rockstor for a couple times, Samba has no desire to turn back on. I worked after the first reinstall, I was able to connect with all my pc’s in my network to the folders in the DATA pool, but now when I try to switch the Samba service on, it “turns on” (turns green), but when I refresh the page, or navigate to another page and get back to the Samba page, its turned itself back off.
Attempting to turn Samba on under the Services tab results in the same issue.
Attempting to turn Samba on through ssh client returns the following message:
Failed to start samba.service: Unit not found.
Removing all samba shares, switching Samba on and adding shares results in the same result (samba stays turned off)
(reboot doesn’t work either)

Anyone have any idea about these issues?


Edit: I have a backup of all data in my DATA pool, so I also tried to run:

[root@rockstor samba]# btrfs check --repair DATA
enabling repair mode
ERROR: mount check: cannot open DATA: No such file or directory
ERROR: could not check mount status: No such file or directory

I also tried:
[root@rockstor samba]# btrfs check --repair /dev/sdc
enabling repair mode
ERROR: /dev/sdc is currently mounted, aborting

Though I don’t know how I can unmount this data pool…

Edit2: (~3h later)

I’m starting to suspect I made a mistake in choosing the wrong pool when (consistently) creating the sonarr share (I realized it was still in pool DATA instead of the pool RockOn-ConfigData) though that doesn’t make any sense since I re-created the sonarr share several times before realizing I was creating it in the DATA pool. Anyway, almost 4h later and no new btrfs error message on screen.

@NuclearFury Welcome to the Rockstor community.

I’d like to at least start with this one if I may, also if you are using the system without having upgraded to the improved 4.12 kernel that comes with either update channel then you don’t have all the interim improvements within the kernel on the btrfs, these could help with your stated and very reasonable:

So getting to the much improved 4.12, via either update channel subscription should help.

What is the output of:

yum info rockstor

Also you samba issues could be related to an interrupted update, see the following thread:

Could the 2.5" 250GB OS drive be flaky at all. Ie. try running a short and then a long smart test via the instructions as: S.M.A.R.T just in case.

Also an precis of your systems current state / function would be good.

Hope that helps.

Hi Philip,

The output of yum info rockstor is:

[root@rockstor ~]# yum info rockstor
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
 * base:
 * epel:
 * extras:
 * updates:
Installed Packages
Name        : rockstor
Arch        : x86_64
Version     : 3.9.1
Release     : 0
Size        : 84 M
Repo        : installed
From repo   : anaconda
Summary     : RockStor -- Store Smartly
License     : GPL
Description : RockStor -- Store Smartly

Available Packages
Name        : rockstor
Arch        : x86_64
Version     : 3.9.2
Release     : 10
Size        : 15 M
Repo        : Rockstor-Stable
Summary     : RockStor -- Store Smartly
License     : GPL
Description : RockStor -- Store Smartly

You are right about the kernel version, currently I’m running on kernel version 4.10. I’m seeing a blinking “RSS” logo (that’s what I confused it with) near the kernel version indicating an update, though when I run this and update the (long) list of updates, I don’t see any change in version numbers after running the updater. Could this be because of a failed previous update that’s blocking the packet manager to update?
Running yum upgrade from ssh looks like it’s installing new updates, but finishes with an error in multilib:

Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem.

About the 2.5" OS drive, it’s true this disk is old, probably came out of an old laptop and reused for this purpose. Replacing it might indeed not be a bad idea. I ran the short and long S.M.A.R.T test, but no issues were reported. I’m planning to replace this drive sometime soon though as this is a drive I’m relying on quite a bit

I’m not quite sure what you meant with current state/function of systems. Do you mean like logs?

Thanks for your help!

I’m planning on reinstalling Rockstor on a new hard drive. I just ordered one that’s coming in tomorrow. That’ll give me a chance to rule out the hard drive, and reorganize some partitions before installing.