[Solved] How hosed am I? (cleared out the database) [turns out: not *that* hosed]

[doenietzomoeilijk (Max)]

Yesterday, I made a dumb mistake, I think - when troubleshooting a problem with the WebUI not showing up after a reboot (something I blame on me horsing around with nginx to run several sites on the same machine, but on a different IP - different story for a different time), I somehow convinced myself that removing /opt/rockstor/.initrock and running /opt/rockstor/bin/initrock would be a fine idea, to get things running again. Blame it on lack of caffeine. Once initrock mentioned doing things to the database, I somewhat realised what was happening…

So yeah, now my database is empty. No mention of users, shares, rockons, and while I managed to get the WebUI to function again (or it did that itself, at this point I’m not even sure anymore), it greets me with the “let’s get started with creating an admin user” thing.

Let’s not.

At this point in time, services are humming along just fine, all the containers are running. I’m now wondering if there’s a spot where a previous backup might be stored (because I know for certain that I did at some point create a backup), and whether I can restore at least the database. Failing that, I’d have to set it up again, preferably with roughly the same setup I had before. I’m sure that’s a lot more work…

Also, it might be an idea to have the root fs use snapshots as well, at least for dumb people like me.

For what it’s worth: this is a 3-disk, 1-pool, 24-share setup, with about 4 rock-ons running (and some containers outside of Rockstor).

So, where do I start?

tl;dr after getting it solved:

• Don’t clear out your database
• Disks, Pools and Shares are very easy to get back
• It helps if you have a Config Backup, that’ll restore you some settings
• Users, Groups and Samba shares etc take some manual labor
• Really, don’t clear out your database…

[Haioken]

Hi @doenietzomoeilijk,

I don’t have access to my own RS machine right now (I’m at work), however I’d start by checking if you have any snapshots of your root BTRFS disk.

Your root BTRFS root ‘pool’ should (from memory) be called ‘rockstor_rockstor’. In this pool should be a share called ‘root’.

You should be able to browse snapshots by going to Storage -> Shares -> Root -> Snapshots.
Assuming you have one or more snapshots prior to breaking things, these should be listed by snapshot name.

Locate the snapshot you’re looking for and click the ‘clone’ button, give the share a name and your snapshot should be mounted in /mnt2/<sharename>.
You should now be able to stop your Rockstor service, copy back the data, and hopefully successfully restart the Rockstor service:

systemctl stop rockstor
cp -R /mnt2/<sharename>/opt/rockstor/.initrock /opt/rockstor/
systemctl start rockstor

You can give this a shot yourself as time permits (it shouldn’t leave you in any worse state than you’re in now!).

EDIT: Apparently root is not automatically scheduled for periodic snapshots, so unless you’ve manually done it - You’re pretty much hosed. Import existing shares into new DB, hope and prey other things are detected, re-install if they’re not. Unfortunately, some of Rockstor is still somewhat primitive for disaster recovery

[Haioken]

Also, regarding altering the nginx configs, I suggest minimal alteration of the nginx config, and instead adding additional configs.

See my topic here: Using nginx/pi-hole to reverse proxy (the dodgy way)
This gives you a cronjob and script that can be run to inject a custom config location into the Rockstor nginx config.

You can then use this for creating additional ‘server’ blocks, for which I’m adding reverse proxies for all my Rockons - My missus isn’t so good at remembering IP address/port combinations, but ‘sonarr.home’ or ‘tv.home’ work well enough!

Specifically mentioned, the following will give you somewhere to have your non-standard configs:

@Haioken

/etc/crontab

* * * * * root /root/scripts/nginx-proxy.sh

@Haioken

/root/scripts/nginx-proxy.sh

#!/usr/bin/bash

# Check for inclide directory in rockstor nginx config
grep -q conf.d /opt/rockstor/etc/nginx/nginx.conf || {
    # Add config in place, create backup nginx.conf.bak
    perl -plni.bak -e 'print "\tinclude\t\t/root/conf.d/\*;\n" if(/\s+server \{/)' /opt/rockstor/etc/nginx/nginx.conf;

    # Restart the service
    systemctl restart rockstor.service || {
        # Failed! Retrieve backup and restart again
        mv /opt/rockstor/etc/nginx/nginx.conf.bak /opt/rockstor/etc/nginx.conf;
        systemctl restart rockstor.service;
    }
}

Hope this helps!

[phillxnet (Philip Guyton)]

@doenietzomoeilijk

@doenietzomoeilijk

So yeah, now my database is empty. …it greets me with the “let’s get started with creating an admin user” thing.

Yes that does sound like a fresh (empty) database.

@doenietzomoeilijk

(because I know for certain that I did at some point create a backup), and whether I can restore at least the database.

This would help if you are referring to a Configuration back of the Configuration Backup and Restore type. You could check that route out. The file’s contents, if it exists, is documented in that doc entry at the bottom. Note that there is now also a terminal option to do configuration backup added by forum member @dilli with assistance from @Flyer, in the following pull request:

https://github.com/rockstor/rockstor-core/pull/1576

And unfortunately I have dropped the ball in documenting this feature, I’ll get to that as time allows; sorry about that people. And I don’t think it has a restore option as of yet. But note that the config backup files are simply gzipped json so, at a push, are human readable once unzipped.

But all in I’m in agreement with @Haioken on this one, ie import your prior pools; which in turn brings shares and snapshots, and re-create the rest. It’s a drag I know but if you do have that gzipped json you also have the last settings (mostly).

The Rocksons however will be tricky as they now have no db counterpart. I’d go for a fresh rockon-root and once that is established remove the existing one, which by then should have been made redundant. You should still have, and can re-use during each rock-on reinstall, your existing individual rock-on config / data shares. So settings and data should be OK for those.

As per reasserting the admin user within the db I’d:

userdel prior-admin-user

prior to re-entering it on that initial setup screen.

That way it doesn’t fail on a ‘user already exists’ type scenario.
Existing users directly after setup are treated differently to users that are created from within the UI so again it may just be easier to userdel them and re-create so that you have the same control over them from within the UI as is normal for Rockstor created users, though this isn’t mandatory.

If all else fails and this gets too messy you can always re-install and re-import but you are pretty much at this point anyway, bar the users and existing ghost rockon config.

Hope that helps and let us know how it goes.

[phillxnet (Philip Guyton)]

@Haioken

@Haioken

EDIT: Apparently root is not automatically scheduled for periodic snapshots, so unless you’ve manually done it …

also

@doenietzomoeilijk

Also, it might be an idea to have the root fs use snapshots as well, at least for dumb people like me.

Yes me too. And this is underway, and their is a hint / easter egg of the direction we are going with this in a recent image post of mine in this forum: just saying. But unfortunately everything just takes ages longer than all would normally like. Oh well in time we should be able to ‘inherit’ this feature (boot to snapshot) and maybe even integrate it’s capability within the Web-UI via a feature such as the long standing and very patient pr from @sfranzen:

https://github.com/rockstor/rockstor-core/pull/1491

which given our medium term plans will need a further rework but again, all in good time hopefully.

@Haioken

Unfortunately, some of Rockstor is still somewhat primitive for disaster recovery

Agreed, and I think our best move here initially is to stand on the shoulders of a different giant (see above referenced hint). Which I’m (evidently ) already working on.

[doenietzomoeilijk (Max)]

@Haioken: Nope, no snapshots of the root partition, unfortunately. It might be something I’ll just go and do manually from now on, at least before I start making big changes…

As for the nginx part - yes, I kept the original as-is, and just added new configs in /etc/nginx/vhosts. Or at least, that was the plan. Seems to have been reverted at this moment, but that’s something to sort later. I’ll keep your dodgy topic in mind.

@phillxnet yes, that’s the type I was in fact referring to - and I’ve found the backup in the right spot. It’s a couple of months old, but unless I’m misremembering things, there shouldn’t have been too many changes, so this is a help. Too bad there’s no restore script yet.

All in all it looks like there’s a decent amount of handywork involved in getting stuff back in shape - either I do the whole “add things, guided by the backed up JSON file” song and dance, or I try and get my hands dirty with an actual restore script. Seems like it’s a matter of hydrating and storing the relevant models from the look of the JSON file - I could always use the backup script as a reference.

I’ll keep you guys posted - thanks so far!

[phillxnet (Philip Guyton)]

@doenietzomoeilijk

… or I try and get my hands dirty with an actual restore script.

That would be nice, but given you currently have a functioning Web-UI you could use it to restore your settings via the Browse button option (top right) on the System - Config Backups page. That way once selected you can Upload that file as a known config. Note that you may face an issue with it already existing in the default directory. Ie you may have to move the saved config file off to a client machine first, just in case. The Web-UI has both the save and the restore. It’s only the cli that I suspect as having only a save. But even if saved via a cli it is then available in the Web-UI as a file that can be restored, but only while the database remains.

So move that saved config to say /root, then copy it over to a client machine and upload it as above then it should show up as a know config (in the db) that can be re-played.

Hope that helps.

[doenietzomoeilijk (Max)]

@phillxnet ah, misunderstood you there, then. But isn’t setting up through the web UI (and creating the root and admin user etc) going to interfere with whatever I’m going to restore from the JSON file? Or is that going to overwrite those newer settings?

And the RockOns not having their database entries backed up is slightly unfortunate, but I suppose I can work around that - I should be able to gather some info from the running Docker environment, at least, and maybe I’ll just go ahead and keep all the containers “outside” RockStor, and use something like Portainer instead.

For now I’m mostly interested in what sense a restore is going to overwrite anything new I create before restoring, in the process of getting the web UI going again.

[phillxnet (Philip Guyton)]

@doenietzomoeilijk

But isn’t setting up through the web UI (and creating the root and admin user etc) I’m going to restore from the JSON file? Or is that going to overwrite those newer settings?

Sort of but the root account always exist and isn’t restored as far as I can remember and where there is a conflict, ie if a user already exists, I think the restore process just moves onto the next item in the json. Not entirely sure as it was a while ago when I last looked at that code but pretty sure. Best to do a little as possible if you are to restore and then see what’s left to do there after as it’s definitely not complete, as per the doc entry for it.

And the RockOns not having their database entries backed up is slightly unfortunate

Yes, slightly for sure.

and maybe I’ll just go ahead and keep all the containers “outside” RockStor, and use something like Portainer instead.

Always an option but a bit extreme. Might be easier just to stop and delete what images you did have as Rockons and just re-install them. Then at least you are back where you were and not creating more potential friction for yourself. Either way do let us know how you get on. I’ve done a few bug fixes to the restore but it’s non trivial code so is bound to have some surprises left in there.

For now I’m mostly interested in what sense a restore is going to overwrite anything new I create before restoring,

Again best to restore to a freshly installed system, which yours is not as it’s only db wiped and so has a bunch of stuff already setup so it’s difficult to account for that in a restore. But then if you wish to maintain that stuff it may make sense to take it from here and try the restore. It logs everything it does both success and failure (if I remember correctly) so you should be able to see what went wrong. But it has nothing to do with the vol / share / snap stuff as this is done via an import. I say you try it and see what happens. Worst case it fails horribly and you have to re-install. As long as you have a copy of that config on a client machine you can always try uploading it and restoring again. Although if you have used the system disk for any config or data shares you will loose those with a re-install.

Let us know how it goes and remember to take a look or even tail the logs while it’s running as it will show what’s happening and that should help locate any failures / requirements for manual config there after…

… getting the web UI going again.

This should work (but as per a fresh install) as soon as you have completed the initial admin user setup screen.

Hope that helps.

Oh and if you do re-install make sure to update to as latest version as your channel allows as you will need as many fixes in as are available. This will help with better import of both the vols and the config.

[doenietzomoeilijk (Max)]

I’ll give the restore a go first, then, see where that takes me. I’d like to not having to reinstall things, if at all possible. I’ll post back with the results (but it’ll be tomorrow)!

[doenietzomoeilijk (Max)]

Well, that was a bit anticlimatic… Created a new admin user, uploaded the backed up config and hit the restore button, got the message about things maybe taking a bit to propagate…
Lots of noise in rockstor.log, as expected, but some settings appear to have been changed to their old values now. Groups, users, pools, shares… nope. Seems like they all got skipped over due to already existing on the system.

Looks like I’ll have to add the database records by hand, after all. At least there’s the JSON to guide me! And it’s not that many users and shares, luckily…

[phillxnet (Philip Guyton)]

@doenietzomoeilijk This bit is important:

… pools, shares… nope.

These are not included in the config restore but are handled by the import mechanism via the Disks page:

Import BTRFS Pool quoting from this doc link we have:

"The BTRFS Pool import procedure imports the following:-

Pools
Shares
Snapshots"

Just in case you missed that in my previous posts here.

Also if you detail how the restore process has failed you here it might help to lead to improvements in that order, ie if a user already exists then we shouldn’t overwrite it and I was under the impression that is what has happened here. I.e your users already existed (linux not Rockstor level). There is definite improvements to be had in this area, although the main focus is not for db wipes of course but could help with developing a more robust mechanism anyway. Maybe copy in to the thread the log sections where you see the restore failing your needs.

Hope that helps and well done for persevering.

[doenietzomoeilijk (Max)]

These are not included in the config restore but are handled by the import mechanism via the Disks page:

Import BTRFS Pool quoting from this doc link we have:

"The BTRFS Pool import procedure imports the following:-

Pools
Shares
Snapshots"

Just in case you missed that in my previous posts here.

Erm, yes, I totally missed that, my bad.

Also if you detail how the restore process has failed you here it might help to lead to improvements in that order, ie if a user already exists then we shouldn’t overwrite it and I was under the impression that is what has happened here. I.e your users already existed (linux not Rockstor level). There is definite improvements to be had in this area, although the main focus is not for db wipes of course but could help with developing a more robust mechanism anyway. Maybe copy in to the thread the log sections where you see the restore failing your needs.

Frankly, I don’t see how one could improve the process in this case, really - I was silly enough to wipe my database, there’s no cure for stupid. This restore would’ve worked on a clean install, which is its purpose. As you said, the import thingy worked as advertised and skipped over the already existing system users.

If anything, the only thing that could be changed is feedback; I had uploaded an un-gziped version of the JSON at first, which resulted in nothing at all happening after I tried to restore that - that could’ve either notified me of the error I was making, or it could’ve gone and actually accept the un-gziped JSON.

Also, the feedback from the import itself was mostly me peering in the logs, so yeah, that could change. Then again, afaik there’s no message queue like system in RS at the moment, and since the handling of the import is async, messaging would have to be as well.

One final thing, apart from imports, might be the UI to import an existing system user into RS, to bring them under RS control. No idea how feasible that actually is, but I had one or two users that I added from the command line, and I’d love to see those in the proper list as well. But that’s a separate issue, really.

I’ll see what importing the Pool does for me and do the users and such by hand. Persevering is part of the fun, really.

[doenietzomoeilijk (Max)]

Allrighty - update from this end: importing the pool went A-OK! Getting the groups and users back into the database was pretty straightforward as well - it’s not that many records really.

That leaves me with setting up the Rockons and Samba shares - I’ve done the latter from the web-UI since it were just a handful. As for the Rockons - the reason I was considering going with a third party solution in the first place, was to be independent of containers being available as Rockons, basically. I’m comfortable enough with Docker to get it to do what I want, anyway, so in many cases it’s easier to just fire up a docker-compose file than to add it through the UI.

Then again, this whole persevering thing is kinda fun, too, so I’ll see what I’ll do. I wonder what’ll happen if I add a Rockon for a container that already is running, though…

Anyway, that’s a different subject. My problem is solved, as it turns out I’m not all that hosed, really. Maybe I’ll set up a recurring task for snapshotting the boot drive, just to make sure, though.

[phillxnet (Philip Guyton)]

@doenietzomoeilijk Thanks for you feedback here. As per:

@doenietzomoeilijk

If anything, the only thing that could be changed is feedback; I had uploaded an un-gziped version of the JSON at first, which resulted in nothing at all happening after I tried to restore that - that could’ve either notified me of the error I was making, or it could’ve gone and actually accept the un-gziped JSON.

Definitely. This is bad / buggy behaviour: Fancy opening a GitHub issue for that one.

@doenietzomoeilijk

Also, the feedback from the import itself was mostly me peering in the logs, so yeah, that could change. Then again, afaik there’s no message queue like system in RS at the moment, and since the handling of the import is async, messaging would have to be as well.

Yes, a definite weakness currently. There has been some discussion on this in the following GitHub issue where @maxhq and @henfri give examples of using logcheck to ‘filter’ logs and generate email reports:

https://github.com/rockstor/rockstor-core/issues/701

We could take the same approach with extract and forward the config-restor log entries. But of course there may be a chicken and egg situation going on there with regard to email. But it may play a part in the initial filtering element.

And there is also @Flyer’s issue which sits at a different level (more Web-UI feedback related):

https://github.com/rockstor/rockstor-core/issues/1541

@doenietzomoeilijk

One final thing, apart from imports, might be the UI to import an existing system user into RS, to bring them under RS control. No idea how feasible that actually is, but I had one or two users that I added from the command line, and I’d love to see those in the proper list as well. But that’s a separate issue, really.

Yes that’s a tricky one as if a user was added from command line it really isn’t a Rockstor user. But maybe it would be ‘sane’ to add an existing user as Rockstor managed during a config restore. To me that make sense. If you agree please feel free to make an additional GitHub issue with that as a request. I think the ability to change an existing user, via the Web-UI into a managed users is a little more complicated and definitely a different case. But restoring a user does rather pre-define that they were once added so is a much clearer case. Nice find.

@doenietzomoeilijk

As for the Rockons - the reason I was considering going with a third party solution in the first place, was to be independent of containers being available as Rockons, basically. I’m comfortable enough with Docker to get it to do what I want, anyway, so in many cases it’s easier to just fire up a docker-compose file than to add it through the UI.

Yes, that makes sense. And is of course one of the big wins of building on top of a general purpose Linux OS. More power to you, which of course there will be as any UI, especially one that caters to democratising such things will necessarily limit the available options: if for no other reason than there would be pages of them otherwise given modern software’s capabilities. But I stand by the simplifications that Rockstor instantiates as it’s good for all of us at one time or another.

@doenietzomoeilijk

My problem is solved, as it turns out I’m not all that hosed, really.

Cool: and thanks for sharing your experience on this one. And do please consider creating the one or two GitHub issues, as you see fit, as I think they are worthy candidates for future improvements.

And as per the Rock-on docker doubling up, remember there is always the following:

Force RockOn cleanup in postgres

@magicalyak Hello again, good ideas. Have you had a look at the currently undocumented: It was added to ‘force’ remove ‘problematic’ named rock-ons. One at a time, and might be a good place to start with extending such ‘emergency’ maintenance tools re rock-ons.

which might get you out of a dead end where that to happen, but that is of course once they are in the db. But good to know going forward.

Thanks for the update and feedback and glad you managed to claw you way out

[doenietzomoeilijk (Max)]

@phillxnet

Definitely. This is bad / buggy behaviour: Fancy opening a GitHub issue for that one.

doenietzomoeilijk:

Done.

@phillxnet

Yes, a definite weakness currently. There has been some discussion on this in the following GitHub issue where @maxhq and @henfri give examples of using logcheck to ‘filter’ logs and generate email reports:

Frankly, that sounds like after-the-fact reporting, which might have its use, but in this case I wouldn’t be against a more immediate, in-my-face message in the web UI, since I’m “doing things there” at that moment anyway. I might not be checking my mail at all, at that moment. I’m not against the more mundane fire-and-forget things like software updates being reported this way, but if I’m doing something “interesting” like restoring info, I’d like a bit more urgency, if that makes sense.

It might not even be that difficult to get going, since there’s already some polling / websocket stuff going on - that bit of infrastructure is already there. Ztask should be able to handle the queueing part,

After reading @Flyer’s issue: yeah, I think that’s pretty much what I agree to.

@phillxnet

But maybe it would be ‘sane’ to add an existing user as Rockstor managed during a config restore. To me that make sense. If you agree please feel free to make an additional GitHub issue with that as a request. I think the ability to change an existing user, via the Web-UI into a managed users is a little more complicated and definitely a different case. But restoring a user does rather pre-define that they were once added so is a much clearer case.

Hm, now that you mention it: it might be a choice to have when restoring a config backup: “if I encounter users already existing during restore, instead of completely skipping them, just add them to the database (if UIDs match etc etc)”.

As for changing a CLI-added user to a Web-UI managed one: that might just be a tiny bit of icing on the cake, although personally I wouldn’t consider it too far-fetched. When creating a user from the UI, I have a choice of making them “Rockstor-controlled”. If I choose not to at that point, I now never have an option to change my mind after the fact (short of horsing around in the database).

I’ll add an issue anyway, some more discussion could go there.

I’ll give the whole RockOn thing some more thought. It’s not that I’m against using RS for handling the containers, it’s just that the current UI is slightly lacking to do the job proper. Maybe this is something I just should get my hands dirty in - actually contributing code instead of just waffling on the forum. Will get back to it, or maybe in the other threads where there’s been discussion about it.

Thanks a ton for your your help, @phillxnet, and @Haioken too. I learned a bit more about how RS does things and how it all fits together.

[phillxnet (Philip Guyton)]

@doenietzomoeilijk Thanks for creating those GitHub issues; nice: I’ve commented, hopefully constructively, on one of them with a starting point code wise. And with regard to:

… it’s just that the current UI is slightly lacking to do the job proper. Maybe this is something I just should get my hands dirty in …

I’ll just leave these here:
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon.py
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon_helpers.py
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon_id.py
plus various others beginning with rockon.

But don’t feel tempted to look at those files , their just full of nice readable Python.

and of course the following doc entries are arbitrarily left here similarly:
Contributing to Rockstor - Overview
and subsection:
http://rockstor.com/docs/contribute.html#developers

Please feel free to drop me a forum pm, if that is preferred, were you to accidentally look at those files and find anything you can’t resist improving. Although those are almost exclusively @suman’s work but I and a few others have dabbled with a couple of fixes in that area You could also look, via blame / history, at what has been done there over time.

[doenietzomoeilijk (Max)]

Thanks for the code pointers - yes, I might just try and make some time to have a go at things. The user restore thing shouldn’t be too hard, even though my Python is slightly rusty (I’m a PHP/Symfony dev by trade).

I also have some thoughts about the whole Docker situation, but in the interest of keeping things manageable, I think I’ll either open or hijack an issue about that separately - this seems tangentially related the the restore thing, but I have other reasons for using Portainer over RS itself, and I’ll document those as needed.