Scheduled snapshots not running; most recent status says "error"

Thad · October 6, 2019, 5:55pm

Brief description of the problem

I went to check whether my scheduled snapshots are working. Apparently they haven’t been running for nearly two years(!) and show error status the last time they were run.

Detailed step by step instructions to reproduce the problem

System → Scheduled Tasks shows “error” in “Last run” column for all tasks.

Storage → Snapshots shows an extremely long gap in snapshots (November 4 2017 to October 5 2019).

Web-UI screenshot

Here’s the “Scheduled Tasks” page:

And here’s what I get when I click on the “error” link next to snapshot-ocdata-hourly (other snapshot tasks look similar):

Here are the settings for snapshot-ocdata-hourly, which look correct to me:

A look at Storage → Snapshots confirms that no scheduled snapshots have been taken since November 4, 2017. A series of snapshots was created yesterday, presumably when I set up the Nextcloud rock-on.

I tried creating some snapshots manually and that looks like it works:

Error Traceback provided on the Web-UI

I don’t see an error traceback in the Web UI. I’m hoping that there’s more detailed logging information than just “error” somewhere, but I’m not sure where to look for it; please advise. Thanks!

phillxnet · October 6, 2019, 6:08pm

@Thad Hello again. Re:

The main logs are in

/opt/rockstor/var/log/rockstor.log

But your issue looks very much like it was caused by a major change he had around about then. Especially given your more recent schedule tasks are working. The fix is to remove the failing scheduled tasks and re-create them. Should then set them up to be using the new API, such as your more recently created ones have done. We really tried to not have this happen but alas.

Let us know if deleting a scheduled task and re-creating it sorts if out for your.

Hope that helps.

Thad · October 6, 2019, 7:21pm

Thanks. I deleted and re-added some of my scheduled snapshots and the first one looks like it ran on schedule, so that appears to have done the job; I’ll let you know if I have any further issues.

phillxnet · October 6, 2019, 7:31pm

@Thad Thanks. Glad it looks to be on the mend.

It was a shame we ran into this but it should be good going forward. The changes that lead to this were very major but had to be done. We were essentially doing something ‘wrong’ and the fix was fairly deep, hence some fallout.

Apologies for the rough ride but hopefully you’re now getting sorted. Also note that if you setup Email Notifications you should hopefully be informed of failures such as this. At least that the hope .

Flox · October 6, 2019, 9:17pm

I’m glad you’re now sorted out, thanks to @phillxnet. I simply wanted to precise one thing that is quite important:

These “snapshots” that are of the rockons-root share aren’t actually snapshots, but rather correspond to the different layers that constitute a docker image. As these are stored as btrfs subvolumes and snapshots (see Docker documentation), Rockstor does detect them as snapshots and surfaces them in the webUI as such. As a result, deleting these snapshots will break your rock-ons. The only fix for that is to re-set the rock-on service and restart it, and then re-installing rock-ons.

There are plans to improve this part in Rockstor, but for the time being be careful not to delete them. It’s not the end of the world if it happens as no data will be lost, but it can be annoying to re-set all rock-ons.

Sorry for the confusion…I simply wanted to let you know in case you wondered why these “snapshots” were seemingly working but not your other “actual” snapshots.

Hope this helps.