I noticed that all of my recent scheduled snapshots have been failing. When looking through the logs, I noticed that all were erroring with “not a BTRFS subvolume.”
I found that none of my shares are mounted under mnt2, as they had been previously. There shares were all created before installing rockstor. They are still visible under the pool name, such as: \mnt2\datadisk1\archives
But Rockstor used to mount them directly under mnt2, such as \mnt2\archives. When I look at those directories, though, they are all blank, indicating that the shares aren’t mounted. Because of this, the snapshot command is not finding a subvolume at that path.
I can’t imagine that I did something that would cause Rockstor to stop mounting shares. Anyone know what could cause this to happen, and more importantly, how to get them restored? I don’t actually care about having them mounted this way, but I do need the snapshots to work.
Thanks.
CommandException: Error running a command. cmd = /usr/sbin/btrfs subvolume snapshot -r /mnt2/archives /mnt2/datadisk1/.snapshots/archives/archives-daily_202307170100. rc = 1. stdout = ['']. stderr = ['ERROR: Not a Btrfs subvolume: Invalid argument', '']
[17/Jul/2023 01:00:02] ERROR [scripts.scheduled_tasks.snapshot:142] Failed to create snapshot at shares/20/snapshots/archives-daily_202307170100
[17/Jul/2023 01:00:02] ERROR [scripts.scheduled_tasks.snapshot:143] 500 Server Error: Internal Server Error for url: http://127.0.0.1:8000/api/shares/20/snapshots/archives-daily_202307170100
I’ve tried looking for issues in the Rockstor log, but I’m not finding anything in there. But I am no expert on the logs, so I could be missing something.
The only thing I can see is that it complains when trying to mount a snapshot, which it likely can’t do due to the parent folder not being mounted correctly. Ironically, the subvolume mentioned is one that should be ignored, as it’s mounted under the subvolume that I excluded from Rockstor. I see this same error message repeated many times throughout the log, so it seems to keep trying to mount this particular snapshot.
So I’m a bit confused here. Hoping someone can help me figure out why the shares aren’t mounting, at which point my snapshots should start working again.
CommandException: Error running a command. cmd = /usr/bin/mkdir -p /mnt2/basesnaps/media/media-daily_202307150102. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \xe2\x80\x98/mnt2/basesnaps/media\xe2\x80\x99: Operation not permitted', '']
[17/Jul/2023 17:52:38] ERROR [smart_manager.data_collector:994] Failed to update share state.. exception: ['Internal Server Error: No JSON object could be decoded']
[17/Jul/2023 17:53:40] INFO [fs.btrfs:1235] Mount Point: /mnt2/backup has indeterminate quota status, skipping qgroup show.
Try 'btrfs quota disable /mnt2/backup'.
[17/Jul/2023 17:53:40] ERROR [storageadmin.middleware:33] Exception occurred while processing a request. Path: /api/commands/refresh-share-state method: POST
[17/Jul/2023 17:53:40] ERROR [storageadmin.middleware:34] Error running a command. cmd = /usr/bin/mkdir -p /mnt2/basesnaps/media/media-daily_202307150102. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \xe2\x80\x98/mnt2/basesnaps/media\xe2\x80\x99: Operation not permitted', '']
I saw in another post a recommendation to check the output of journalctl -u rockstor-bootstrap.
It seems that it tries 15 times and times out.
Jul 17 17:43:14 Rockstor systemd[1]: Starting Rockstor bootstrapping tasks…
Jul 17 17:44:41 Rockstor bootstrap[13973]: BTRFS device scan complete
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Exception occured while bootstrapping. This could be because rockstor.service is still starting up. will wait 2 seconds and try again. Exception: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor bootstrap[13973]: Max attempts(15) reached. Connection errors persist. Failed to bootstrap. Error: [‘Internal Server Error: No JSON object could be decoded’]
Jul 17 17:44:41 Rockstor systemd[1]: rockstor-bootstrap.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 17:44:41 Rockstor systemd[1]: rockstor-bootstrap.service: Failed with result ‘exit-code’.
Jul 17 17:44:41 Rockstor systemd[1]: Failed to start Rockstor bootstrapping tasks.
Update: due to a lack of other options, I updated Rockstor to 4.6.1-0 (stable channel), which seems to have taken care of the problem. Once I rebooted after the update, my shares were mounted again.
I have to be honest that this is shaking my faith in Rockstor. I feel like every time I get comfortable with it, it throws me a curve ball. Since it stopped mounting my shares for no reason, I am now paranoid that it will happen again.
I am thinking I will go ahead and keep using it, but if it happens again, I will go back to Ubuntu. No offense to the developers, as I know they are working very hard on it, and I love the relative ease-of-use. My old Ubuntu NAS took more time to setup (although in fairness I then probably spent more time troubleshooting my various Rockstor issues after the install).
Hopefully it was just a one-time glitch, and I can forget all about it.
Entirely understandable, however your setup is not entirely generated by our Web-UI: so to be frank you are kind of custom here. That is not generally the happy path - but being DIY and community orientated we are game, but also restricted on resources to expand our capabilities to all folks wants and needs. You were able to found our code-editing option to ignore subvolumes - but you then set about doing fancy command line re-arrangements and we basically have to, by design, not account (at least yet) for all of what btrfs can do. That is a shame but we are working on it. And we are open source - so the old adage of pull requests welcome always stands. And you have rather caught us at a transitional period where all the main folks are tied up in managing our new stable release (yesterday) and establishing our new testing release as well. That make it problematic for us to tend to custom cases that will likely involve quite the drill-down. As you know there is quite the complexity in dealing with a fractal-like filesystem such as btrfs and we have all our other capabilities to adjust when ever we make any change to how we do stuff. We also try to maintain import and ‘understanding’ of our Pool / share structure compatible consistend across as many older versions as we can. It basically hasn’t changed for years - that is reasuring and important. So if we begin introducing greater capabilities such as subvolume within subvolume (such as you may now have): not counting the top volume (Pool in our speak) as a subvol in this case: that could well be a overreach at this time and introduce more backward compatibility issues.
You may well not be our target audience: we are an appliance by design; and thus limit our base OS necessarily in the interests of ease of use. Where-as you have already, likely for years, had not such constraint. We are expanding our capabilities bit-by-bit but we will likely never encompase (and more to the point make accessible) all of what our underlying OS has to offer. The command line after-all, is unlimited. But I like to think we can more recently cater for the majority of users: but the second one does their-own-thing and customises beyond what can be done at the Web-UI one is basically, by definition off-piste.
So in short you may have spoilt yourself with command line powers - I know the feeling - but I also like to have an easy to use Web-UI and I think most do. It all really depends where you interest lie and the time one has to do what one wants to do at any one time. Your interaction there on the forum has been great so it would be a shame to see you move on - but you never know - you might be back and we may have adapted some more to what folks want as and when we receive input and assistance accordingly. More recently we have had to deal with many dependency updates so our flexibility improvements (features/capabilities) has slowed: but we must take the time to up our dependency game as it only improves our robustness and options at a lower level. And updating dependencies is a must these days.
Likely not if you are cli intervening - and you have in-addition already enabled capabilities only accessible via editing our code as distributed: that’s all fancy and super welcome but is definitely in the realm of our testing channel / developer user base who are the driving force of what we end up being able to do. We are embarking on a new testing any day now so do give us a look-in and maybe contribute as and when - as you have done to date - as all input is welcome. But of course development is not production, our stable channel is aimed at providing a system to folks where (during the last testing release) most stuff is basically settled: But that does not include non Web-UI initiated configurations.
So do keep in touch and many thanks for your excellent reports: such informed input/experimentation is invaluable and it’s been fun: but we may just not be what you need if you required ultimate flexibility. We kind of impose a set structure for our Pools (btrfs volumes), for example, that helps (a lot) to keep things simple for the development and for the user. But it is a restriction that may be a restriction too far for some.
Hope that helps, and maybe we can serve a little in your setup still, or later down the line. And do keep an eye on our next testing phase changelog as it appears here on the forum. But for a while we will be mainly updating our dependencies!!
Thanks much for the detailed reply and kind words, @phillxnet.
I understand 100% what you are saying. I do realize that I am kind of straddling two worlds here, wanting both the ease-of-use of an appliance and the full customization of DIY. I can’t compare my experiences to a pure out-of-the-box scenario.
To better clarify, though, I actually haven’t done that much customization (yet), which is why I was confused about my shares suddenly not mounting. The only unusual thing that I have done to date with BTRFS is to create two dummy subvolumes, one to facilitate easier sharing of multiple shares via Samba and the other to hide some snapshot backups I made across pools.
If either of those broke Rockstor’s ability to mount my regular shares, then I would have understood and realized that I had inadvertently screwed it up. But I believe both of those changes have survived multiple reboots, with no issues seen with mounting shares. (Unfortunately, it’s hard sometimes to say that conclusively, since it took me days to notice that my shares weren’t mounting, so I acknowledge that it’s possible that the subvolume “hiding” trick caused an issue…but then why would it start working after the upgrade?) Then abruptly my shares stopped mounting, and that persisted across at least 6 reboots. Then upgrading Rockstor to the new stable version fixed it, so it seems that Rockstor doesn’t have any issues with this setup. I wish that I could have figured out the root cause, as that would have given me more confidence. The only thing I can remember changing around this time is donating to get access to the stable channel (but not doing any updates). I can’t imagine that causing any issues.
But with that all said, I think Rockstor is terrific, and I certainly would not discourage anyone from using it, including myself. The great thing is that because it uses BTRFS, I know that I can switch to another flavor of Linux (or back) at any time and maintain my subvolumes and snapshots. My next experimentation will probably be to set up Rockstor on a second machine, as I’d like to try replication. If that works well, then I will be able to live with a more “stock” Rockstor.
@orangejulio Hello again,
Yes I’m a little at a loss also, and you obviously appreciate some of the complexities of btrfs: and we definitely have limitation, with some a little too close in (i.e. share/pool renaming for example) but all in good time and we have made a few steps towards such flexibility.
Upgrade so re-write re-assert code that is shared with what it is updating. Maybe that ignore subvolume code change ended up being reverted. I really don’t know as I’m actually not sure of your entire setup any-longer. I know you have detailed it but I’ve also been pretty busy with getting our stable release out and just about to start a new testing phase so all go in the back-ground as it were.
Yes, that would be great. To approach the GitHub issue stage to start proper debugging/code improvement we really need an reproducer. Once that is in-hand we are almost home and dry as then anyone who cares to can reproduce where we may be failing in our intent or failing expectations and it be resolved one way or another.
As you say, we are pretty generic under neath the Web-UI: but we can only handle a subset of all that is possible. It would be great to have a reproducer of what you experienced so if you end up recreating anything the dandy. A confusion here may be that we do our own mounts - it allows us to fully control the mount process/options/etc so if we do mess-up somewhere there can be a mount failure. But the code is pretty straigh-forward and hasn’t really change for a number of years - at least significantly.
Do keep us posted and I’m sorry I’m unable to spend more time digging in here. But we are all open source so it is entirely possible to root out exactly what has happened. But in our new testing phase we are again doing more updates to our dependencies so again we are likely not to have that much time to tend (at least for a bit) to improvements re features/capabilities. But we often get outside contributors tending to some requests - or the reporters themselves of course.
Our main development is done in the testing branch which, as stated, is about to enter a new phase.
Anyway thanks for you kind words and I hope the project can help you in some way as that is what it’s all about. And keep that -looking at the code, bit in mind as it’s all fairly straight-forward: at least when looked at in little bits. :).
