Ran out of space, now rockstor won't mount drives

[phillxnet (Philip Guyton)]

@Michael_Arthur

@anon52054362_Arthur

I’ve paid for stable updates and have been running the same version for a long time now,

Thanks for helping to support Rockstor development. Well worth double checking your version exactly as if you went from testing to stable we had the following bug:

version and date incorrectly reported re update info

and it’s associated pull request:

version and date incorrectly reported re update info. Fixes #1870

Also note that very early 3.9.2.# versions shared this quotas disabled failure; as per my last posts GitHub reverence so best check and report here the version so we can rule that one out also:

So do make sure via:

yum info rockstor

Which should show the installed and available versions.

Would be good to get to the source of your issue here as you have reported non rockstor systemd tasks failing to start atd (the at deamon) which makes me think you have a low level problem with your system disk (ssd) or it’s attachment (cable) / driver / controller. And of course Rockstor sits on top of a bunch of other such services. Maybe take a look at the btrfs report for errors on the system disk for example via (in your case):

btrfs dev stats /mnt2/rockstor_rockstor00

Re:

@anon52054362_Arthur

May 10 18:06:13 lazarus docker-wrapper[28140]: system.exceptions.CommandException: Error running a command. cmd = /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --data-root /mnt2/rockons. rc = 1. stdout = ['']. stderr = ['chmod /mnt2/rockons: operation not permitted', '']

you may have a stuck immutable bit or that subvol has gone read only, or is usually on data pool but if not mounted at that time has fallen through to system pool directory. For an instance we had a while ago with immutable flag bugs see @Haioken and @Rene_Castberg contributions in the following thread:

SOLVED: BTRFS Subvolume create/delete fail

Problem has been resolved, though I’m unsure why it had happened. Immutable flag was set on /mnt2/tempraid, meaning that the contents of /mnt2/tempraid was unmodifiable (though decendants could be altered). I know I didn’t set the flag, and according to BTRFS IRC, scrubs do not set the flag in the event of error. Problem was resolved with chattr -i /mnt2/tempraid Should have thought of it before - in fact, I could swear I checked it, but apparently I did not.

Again this was associated from early stable channel releases.

You have quite dispersed errors all from a single system. And seemingly spanning both your pools. Assuming in that case the /mnt2/rockon was on your data drive.

You could also check your systems memory, ie take a look at our Pre-Install Best Practice (PBP) doc section and specifically the Memory Test (memtest86+) subsection.

Just puzzled by so many different and unrelated (bar via OS parts) errors here.

Let us know how you get on. And think about what might have changed from your prior, presumably more stable time to what is happening now. I.e. have you increased the load on the PSU for example, again a common cause of intermittent and unrelated issues. Does the system have adequate ventilation: dust bunnies etc.

Hope that helps.