[Solved] How hosed am I? (cleared out the database) [turns out: not *that* hosed]

Yesterday, I made a dumb mistake, I think - when troubleshooting a problem with the WebUI not showing up after a reboot (something I blame on me horsing around with nginx to run several sites on the same machine, but on a different IP - different story for a different time), I somehow convinced myself that removing /opt/rockstor/.initrock and running /opt/rockstor/bin/initrock would be a fine idea, to get things running again. Blame it on lack of caffeine. Once initrock mentioned doing things to the database, I somewhat realised what was happeningā€¦

So yeah, now my database is empty. No mention of users, shares, rockons, and while I managed to get the WebUI to function again (or it did that itself, at this point Iā€™m not even sure anymore), it greets me with the ā€œletā€™s get started with creating an admin userā€ thing.

Letā€™s not.

At this point in time, services are humming along just fine, all the containers are running. Iā€™m now wondering if thereā€™s a spot where a previous backup might be stored (because I know for certain that I did at some point create a backup), and whether I can restore at least the database. Failing that, Iā€™d have to set it up again, preferably with roughly the same setup I had before. Iā€™m sure thatā€™s a lot more workā€¦

Also, it might be an idea to have the root fs use snapshots as well, at least for dumb people like me. :wink:

For what itā€™s worth: this is a 3-disk, 1-pool, 24-share setup, with about 4 rock-ons running (and some containers outside of Rockstor).

So, where do I start? :slight_smile:

tl;dr after getting it solved:

  • Donā€™t clear out your database
  • Disks, Pools and Shares are very easy to get back
  • It helps if you have a Config Backup, thatā€™ll restore you some settings
  • Users, Groups and Samba shares etc take some manual labor
  • Really, donā€™t clear out your databaseā€¦

Hi @doenietzomoeilijk,

I donā€™t have access to my own RS machine right now (Iā€™m at work), however Iā€™d start by checking if you have any snapshots of your root BTRFS disk.

Your root BTRFS root ā€˜poolā€™ should (from memory) be called ā€˜rockstor_rockstorā€™. In this pool should be a share called ā€˜rootā€™.

You should be able to browse snapshots by going to Storage -> Shares -> Root -> Snapshots.
Assuming you have one or more snapshots prior to breaking things, these should be listed by snapshot name.

Locate the snapshot youā€™re looking for and click the ā€˜cloneā€™ button, give the share a name and your snapshot should be mounted in /mnt2/<sharename>.
You should now be able to stop your Rockstor service, copy back the data, and hopefully successfully restart the Rockstor service:

systemctl stop rockstor
cp -R /mnt2/<sharename>/opt/rockstor/.initrock /opt/rockstor/
systemctl start rockstor

You can give this a shot yourself as time permits (it shouldnā€™t leave you in any worse state than youā€™re in now!).

EDIT: Apparently root is not automatically scheduled for periodic snapshots, so unless youā€™ve manually done it - Youā€™re pretty much hosed. Import existing shares into new DB, hope and prey other things are detected, re-install if theyā€™re not. Unfortunately, some of Rockstor is still somewhat primitive for disaster recovery :confused:

Also, regarding altering the nginx configs, I suggest minimal alteration of the nginx config, and instead adding additional configs.

See my topic here: Using nginx/pi-hole to reverse proxy (the dodgy way)
This gives you a cronjob and script that can be run to inject a custom config location into the Rockstor nginx config.

You can then use this for creating additional ā€˜serverā€™ blocks, for which Iā€™m adding reverse proxies for all my Rockons - My missus isnā€™t so good at remembering IP address/port combinations, but ā€˜sonarr.homeā€™ or ā€˜tv.homeā€™ work well enough!

Specifically mentioned, the following will give you somewhere to have your non-standard configs:

Hope this helps!

1 Like

@doenietzomoeilijk

Yes that does sound like a fresh (empty) database.

This would help if you are referring to a Configuration back of the Configuration Backup and Restore type. You could check that route out. The fileā€™s contents, if it exists, is documented in that doc entry at the bottom. Note that there is now also a terminal option to do configuration backup added by forum member @dilli with assistance from @Flyer, in the following pull request:

https://github.com/rockstor/rockstor-core/pull/1576

And unfortunately I have dropped the ball in documenting this feature, Iā€™ll get to that as time allows; sorry about that people. And I donā€™t think it has a restore option as of yet. But note that the config backup files are simply gzipped json so, at a push, are human readable once unzipped.

But all in Iā€™m in agreement with @Haioken on this one, ie import your prior pools; which in turn brings shares and snapshots, and re-create the rest. Itā€™s a drag I know but if you do have that gzipped json you also have the last settings (mostly).

The Rocksons however will be tricky as they now have no db counterpart. Iā€™d go for a fresh rockon-root and once that is established remove the existing one, which by then should have been made redundant. You should still have, and can re-use during each rock-on reinstall, your existing individual rock-on config / data shares. So settings and data should be OK for those.

As per reasserting the admin user within the db Iā€™d:

userdel prior-admin-user

prior to re-entering it on that initial setup screen.

That way it doesnā€™t fail on a ā€˜user already existsā€™ type scenario.
Existing users directly after setup are treated differently to users that are created from within the UI so again it may just be easier to userdel them and re-create so that you have the same control over them from within the UI as is normal for Rockstor created users, though this isnā€™t mandatory.

If all else fails and this gets too messy you can always re-install and re-import but you are pretty much at this point anyway, bar the users and existing ghost rockon config.

Hope that helps and let us know how it goes.

@Haioken

also

Yes me too. And this is underway, and their is a hint / easter egg of the direction we are going with this in a recent image post of mine in this forum: just saying. But unfortunately everything just takes ages longer than all would normally like. Oh well in time we should be able to ā€˜inheritā€™ this feature (boot to snapshot) and maybe even integrate itā€™s capability within the Web-UI via a feature such as the long standing and very patient pr from @sfranzen:

https://github.com/rockstor/rockstor-core/pull/1491

which given our medium term plans will need a further rework but again, all in good time hopefully.

Agreed, and I think our best move here initially is to stand on the shoulders of a different giant (see above referenced hint). Which Iā€™m (evidently :slight_smile:) already working on.

@Haioken: Nope, no snapshots of the root partition, unfortunately. It might be something Iā€™ll just go and do manually from now on, at least before I start making big changesā€¦

As for the nginx part - yes, I kept the original as-is, and just added new configs in /etc/nginx/vhosts. Or at least, that was the plan. Seems to have been reverted at this moment, but thatā€™s something to sort later. Iā€™ll keep your dodgy topic in mind. :wink:

@phillxnet yes, thatā€™s the type I was in fact referring to - and Iā€™ve found the backup in the right spot. Itā€™s a couple of months old, but unless Iā€™m misremembering things, there shouldnā€™t have been too many changes, so this is a help. Too bad thereā€™s no restore script yet. :slight_smile:

All in all it looks like thereā€™s a decent amount of handywork involved in getting stuff back in shape - either I do the whole ā€œadd things, guided by the backed up JSON fileā€ song and dance, or I try and get my hands dirty with an actual restore script. Seems like itā€™s a matter of hydrating and storing the relevant models from the look of the JSON file - I could always use the backup script as a reference.

Iā€™ll keep you guys posted - thanks so far!

@doenietzomoeilijk

That would be nice, but given you currently have a functioning Web-UI you could use it to restore your settings via the Browse button option (top right) on the System - Config Backups page. That way once selected you can Upload that file as a known config. Note that you may face an issue with it already existing in the default directory. Ie you may have to move the saved config file off to a client machine first, just in case. The Web-UI has both the save and the restore. Itā€™s only the cli that I suspect as having only a save. But even if saved via a cli it is then available in the Web-UI as a file that can be restored, but only while the database remains.

So move that saved config to say /root, then copy it over to a client machine and upload it as above then it should show up as a know config (in the db) that can be re-played.

Hope that helps.

@phillxnet ah, misunderstood you there, then. But isnā€™t setting up through the web UI (and creating the root and admin user etc) going to interfere with whatever Iā€™m going to restore from the JSON file? Or is that going to overwrite those newer settings?

And the RockOns not having their database entries backed up is slightly unfortunate, but I suppose I can work around that - I should be able to gather some info from the running Docker environment, at least, and maybe Iā€™ll just go ahead and keep all the containers ā€œoutsideā€ RockStor, and use something like Portainer instead.

For now Iā€™m mostly interested in what sense a restore is going to overwrite anything new I create before restoring, in the process of getting the web UI going again.

@doenietzomoeilijk

Sort of but the root account always exist and isnā€™t restored as far as I can remember and where there is a conflict, ie if a user already exists, I think the restore process just moves onto the next item in the json. Not entirely sure as it was a while ago when I last looked at that code but pretty sure. Best to do a little as possible if you are to restore and then see whatā€™s left to do there after as itā€™s definitely not complete, as per the doc entry for it.

Yes, slightly for sure.

Always an option but a bit extreme. Might be easier just to stop and delete what images you did have as Rockons and just re-install them. Then at least you are back where you were and not creating more potential friction for yourself. Either way do let us know how you get on. Iā€™ve done a few bug fixes to the restore but itā€™s non trivial code so is bound to have some surprises left in there.

Again best to restore to a freshly installed system, which yours is not as itā€™s only db wiped and so has a bunch of stuff already setup so itā€™s difficult to account for that in a restore. But then if you wish to maintain that stuff it may make sense to take it from here and try the restore. It logs everything it does both success and failure (if I remember correctly) so you should be able to see what went wrong. But it has nothing to do with the vol / share / snap stuff as this is done via an import. I say you try it and see what happens. Worst case it fails horribly and you have to re-install. As long as you have a copy of that config on a client machine you can always try uploading it and restoring again. Although if you have used the system disk for any config or data shares you will loose those with a re-install.

Let us know how it goes and remember to take a look or even tail the logs while itā€™s running as it will show whatā€™s happening and that should help locate any failures / requirements for manual config there afterā€¦

This should work (but as per a fresh install) as soon as you have completed the initial admin user setup screen.

Hope that helps.

Oh and if you do re-install make sure to update to as latest version as your channel allows as you will need as many fixes in as are available. This will help with better import of both the vols and the config.

1 Like

Iā€™ll give the restore a go first, then, see where that takes me. Iā€™d like to not having to reinstall things, if at all possible. Iā€™ll post back with the results (but itā€™ll be tomorrow)!

Well, that was a bit anticlimaticā€¦ Created a new admin user, uploaded the backed up config and hit the restore button, got the message about things maybe taking a bit to propagateā€¦
Lots of noise in rockstor.log, as expected, but some settings appear to have been changed to their old values now. Groups, users, pools, sharesā€¦ nope. Seems like they all got skipped over due to already existing on the system.

Looks like Iā€™ll have to add the database records by hand, after all. At least thereā€™s the JSON to guide me! And itā€™s not that many users and shares, luckilyā€¦

@doenietzomoeilijk This bit is important:

These are not included in the config restore but are handled by the import mechanism via the Disks page:

Import BTRFS Pool quoting from this doc link we have:

"The BTRFS Pool import procedure imports the following:-

Pools
Shares
Snapshots"

Just in case you missed that in my previous posts here.

Also if you detail how the restore process has failed you here it might help to lead to improvements in that order, ie if a user already exists then we shouldnā€™t overwrite it and I was under the impression that is what has happened here. I.e your users already existed (linux not Rockstor level). There is definite improvements to be had in this area, although the main focus is not for db wipes of course but could help with developing a more robust mechanism anyway. Maybe copy in to the thread the log sections where you see the restore failing your needs.

Hope that helps and well done for persevering.

1 Like

Erm, yes, I totally missed that, my bad.

Frankly, I donā€™t see how one could improve the process in this case, really - I was silly enough to wipe my database, thereā€™s no cure for stupid. This restore wouldā€™ve worked on a clean install, which is its purpose. As you said, the import thingy worked as advertised and skipped over the already existing system users.

If anything, the only thing that could be changed is feedback; I had uploaded an un-gziped version of the JSON at first, which resulted in nothing at all happening after I tried to restore that - that couldā€™ve either notified me of the error I was making, or it couldā€™ve gone and actually accept the un-gziped JSON.

Also, the feedback from the import itself was mostly me peering in the logs, so yeah, that could change. Then again, afaik thereā€™s no message queue like system in RS at the moment, and since the handling of the import is async, messaging would have to be as well.

One final thing, apart from imports, might be the UI to import an existing system user into RS, to bring them under RS control. No idea how feasible that actually is, but I had one or two users that I added from the command line, and Iā€™d love to see those in the proper list as well. But thatā€™s a separate issue, really.

Iā€™ll see what importing the Pool does for me and do the users and such by hand. Persevering is part of the fun, really. :slight_smile:

1 Like

Allrighty - update from this end: importing the pool went A-OK! Getting the groups and users back into the database was pretty straightforward as well - itā€™s not that many records really.

That leaves me with setting up the Rockons and Samba shares - Iā€™ve done the latter from the web-UI since it were just a handful. As for the Rockons - the reason I was considering going with a third party solution in the first place, was to be independent of containers being available as Rockons, basically. Iā€™m comfortable enough with Docker to get it to do what I want, anyway, so in many cases itā€™s easier to just fire up a docker-compose file than to add it through the UI.

Then again, this whole persevering thing is kinda fun, too, so Iā€™ll see what Iā€™ll do. I wonder whatā€™ll happen if I add a Rockon for a container that already is running, thoughā€¦

Anyway, thatā€™s a different subject. My problem is solved, as it turns out Iā€™m not all that hosed, really. Maybe Iā€™ll set up a recurring task for snapshotting the boot drive, just to make sure, though. :slight_smile:

@doenietzomoeilijk Thanks for you feedback here. As per:

Definitely. This is bad / buggy behaviour: Fancy opening a GitHub issue for that one.

Yes, a definite weakness currently. There has been some discussion on this in the following GitHub issue where @maxhq and @henfri give examples of using logcheck to ā€˜filterā€™ logs and generate email reports:

https://github.com/rockstor/rockstor-core/issues/701

We could take the same approach with extract and forward the config-restor log entries. But of course there may be a chicken and egg situation going on there with regard to email. But it may play a part in the initial filtering element.

And there is also @Flyerā€™s issue which sits at a different level (more Web-UI feedback related):

https://github.com/rockstor/rockstor-core/issues/1541

Yes thatā€™s a tricky one as if a user was added from command line it really isnā€™t a Rockstor user. But maybe it would be ā€˜saneā€™ to add an existing user as Rockstor managed during a config restore. To me that make sense. If you agree please feel free to make an additional GitHub issue with that as a request. I think the ability to change an existing user, via the Web-UI into a managed users is a little more complicated and definitely a different case. But restoring a user does rather pre-define that they were once added so is a much clearer case. Nice find.

Yes, that makes sense. And is of course one of the big wins of building on top of a general purpose Linux OS. More power to you, which of course there will be as any UI, especially one that caters to democratising such things will necessarily limit the available options: if for no other reason than there would be pages of them otherwise given modern softwareā€™s capabilities. But I stand by the simplifications that Rockstor instantiates as itā€™s good for all of us at one time or another.

Cool: and thanks for sharing your experience on this one. And do please consider creating the one or two GitHub issues, as you see fit, as I think they are worthy candidates for future improvements.

And as per the Rock-on docker doubling up, remember there is always the following:

which might get you out of a dead end where that to happen, but that is of course once they are in the db. But good to know going forward.

Thanks for the update and feedback and glad you managed to claw you way out :slight_smile:

Done.

Frankly, that sounds like after-the-fact reporting, which might have its use, but in this case I wouldnā€™t be against a more immediate, in-my-face message in the web UI, since Iā€™m ā€œdoing things thereā€ at that moment anyway. I might not be checking my mail at all, at that moment. Iā€™m not against the more mundane fire-and-forget things like software updates being reported this way, but if Iā€™m doing something ā€œinterestingā€ like restoring info, Iā€™d like a bit more urgency, if that makes sense.

It might not even be that difficult to get going, since thereā€™s already some polling / websocket stuff going on - that bit of infrastructure is already there. Ztask should be able to handle the queueing part,

After reading @Flyerā€™s issue: yeah, I think thatā€™s pretty much what I agree to.

Hm, now that you mention it: it might be a choice to have when restoring a config backup: ā€œif I encounter users already existing during restore, instead of completely skipping them, just add them to the database (if UIDs match etc etc)ā€.

As for changing a CLI-added user to a Web-UI managed one: that might just be a tiny bit of icing on the cake, although personally I wouldnā€™t consider it too far-fetched. When creating a user from the UI, I have a choice of making them ā€œRockstor-controlledā€. If I choose not to at that point, I now never have an option to change my mind after the fact (short of horsing around in the database).

Iā€™ll add an issue anyway, some more discussion could go there.

Iā€™ll give the whole RockOn thing some more thought. Itā€™s not that Iā€™m against using RS for handling the containers, itā€™s just that the current UI is slightly lacking to do the job proper. Maybe this is something I just should get my hands dirty in - actually contributing code instead of just waffling on the forum. :wink: Will get back to it, or maybe in the other threads where thereā€™s been discussion about it.

Thanks a ton for your your help, @phillxnet, and @Haioken too. I learned a bit more about how RS does things and how it all fits together. :slight_smile:

@doenietzomoeilijk Thanks for creating those GitHub issues; nice: Iā€™ve commented, hopefully constructively, on one of them with a starting point code wise. And with regard to:

Iā€™ll just leave these here:
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon.py
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon_helpers.py
https://github.com/rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/rockon_id.py
plus various others beginning with rockon.

But donā€™t feel tempted to look at those files :slight_smile:, their just full of nice readable Python.

and of course the following doc entries are arbitrarily left here similarly:
Contributing to Rockstor - Overview
and subsection:
http://rockstor.com/docs/contribute.html#developers

Please feel free to drop me a forum pm, if that is preferred, were you to accidentally look at those files and find anything you canā€™t resist improving. Although those are almost exclusively @sumanā€™s work but I and a few others have dabbled with a couple of fixes in that area :smile: You could also look, via blame / history, at what has been done there over time.

Thanks for the code pointers - yes, I might just try and make some time to have a go at things. The user restore thing shouldnā€™t be too hard, even though my Python is slightly rusty (Iā€™m a PHP/Symfony dev by trade). :slight_smile:

I also have some thoughts about the whole Docker situation, but in the interest of keeping things manageable, I think Iā€™ll either open or hijack an issue about that separately - this seems tangentially related the the restore thing, but I have other reasons for using Portainer over RS itself, and Iā€™ll document those as needed.