Hey, I’m not sure what the root cause is, but I seem to have been running into one thing after another. I added a couple of drives to my pool, rebalanced, disabled quotas to make the rebalancing go faster, rebooted to work around the “too many files open” errors, deleted the docker network config so that it could start again, and then noticed that my main share appeared to be empty.
I know my data is still intact, because I can see it under /mnt2/<pool>/<share> but /mnt2/<share> is either empty or default config files for various rockons for each of my shares. Another reboot didn’t seem to help.
I looked around and found a number of forum threads with similar issues, but it was hard to figure out what still applied and what would be the best course of action. Can anyone here provide some guidance?
I’m on RockStor 3.9.1-0 / Linux: 4.10.6-1.el7.elrepo.x86_64 for reference.
@nfriedly The “… too many open files …” issue is a bad one as it can randomly break any number of things across the entire system related to the volume in question. Not just Rockstor related. Also of note is that disabling quotas is very invasive and can lead to undetermined behaviour. There have, however, been a number of fixes in the current testing channel that may be relevant. Most notable of these were both merged a month ago in:
With the most relevant, from a functionally perspective, being:
while the other in that release is a useful addition that reports all current pool and share mount options in all relevant tables and pool/share detail views.
Due to the relatively frequent stable releases, which contain all ‘to-date’ at the time of release testing channel fixes, testing channel release fixes are very rarely ‘back ported’. And the testing channel is mostly assumed when assisting with development.
Hope that helps and there is also a pending pr that improves subvol mount robustness re degraded pool scenarios.
However, given all of that, I would first check your pool health as if it has gone Read-only that can also break subvol (share) mounts: at least for released Rockstor code anyway as that was the subject of my pr today. But I do appreciate that you have a chicken and egg scenario here given your system is affected by the “… too many open files …” scenario. Tricky.
I tried re-enabling quotas and rebooting, but that didn’t seem to fix anything
So, I tried manually mounting all of my subvolumes and that appears to work for now. Hopefully I won’t need to reboot very often before the next release.
@nfriedly Cheers for the update. Yes if you look at @Dragon2611 linked issue in the above referenced pull request, I’ll link for ease here:
They state that heir work around involved the same manual mount, and later confirmed their system was fixed by the update referenced.
Their issue was marked as a duplicate of the one used as the focus of the indicated fix pull request.
Upon review just now it looks like they also turned off quotas at some point:
“I suspect it’s because I’ve had to turn quotas off in the past, or something didn’t upgrade correctly”
So hopefully your system should be sorted similarly when the next stable comes out with that fix in.
Not really but from the time frame it looks like we are on the homeward run.
Sorry for the slow response and glad you got up and running again.
as that was the subject of my pr today. But I do appreciate that you have a chicken and egg scenario here given your system is affected by the “… too many open files …” scenario. Tricky.