Can't roll back to hourly snapshot; daily/weekly/monthly not showing up at all

BTRFS
1

I’m getting a blank white screen when I try to load Owncloud. The last time this happened, rolling back MariaDB fixed it.

So I got set to do that, only to find that there’s a problem with my snapshots.

I’ve got both mariadb-data and owncloud-data set to keep 12 hourly snapshots, 7 daily snapshots, 4 weekly snapshots, and 6 monthly snapshots.

The table says they’re running – Daily ran 18 hours ago, Weekly ran 7 days ago, etc.

But most of them aren’t there. When I pull up the mariadb-data share, all I have are the last 12 hourly snapshots and one snapshot I took manually back in December; no daily, weekly, or monthly. Worse, I can’t roll back to any of them; the “Rollback” button is grayed out.

The owncloud-data share is a little better, but not much. In addition to the 12 hourly snapshots, it’s also got 1 daily snapshot and 1 weekly snapshot. Plus two older ones (one from November and one from September) that I assume were also manual. In this case the “Rollback” button isn’t grayed out, but if I click it, it only allows me to select the snapshot from September; none of the other snapshots are even listed in the table.

Any idea what could be causing this? It’s a real mess.

2

@Thad First off a very belated welcome to the Rockstor community forum.

We do have a known issue where under some circumstances snapshots are taken but don’t then have a Rockstor UI entry. I’m not familiar with this issue myself but I know that the intrepid @Flyer has done some work on this. The following open issue has some details:

In which case you may find that via the command line your intended snapshots exist.

This is more in @Flyer territory I’m afraid.

3

Hi @Thad,
can you provide some logs from /opt/rockstor/var/log/rockstor.log ?

Snapshots rollback: only writable snapshots available for rollback

Mirko

4

@phillxnet Thanks, that’s helpful.

Sorry to ask you to spoonfeed me, but I don’t have a lot of experience with the btrfs tools, or with Rockstor’s directory structure; where should I look for the snapshots, and what’s the command to roll back?

@Flyer rockstor.log is 2800 lines long, but the last two days’ worth are just these two lines, repeated every time the snapshot task runs:

[15/Feb/2017 12:50:04] DEBUG [fs.btrfs:713] Quota inconsistency while assigning 0/1446. Rescan scheduled. [15/Feb/2017 12:50:05] DEBUG [scripts.scheduled_tasks.snapshot:107] created snapshot at shares/mariadb-data/snapshots/mariadata_201702151250

I note that the logs show that message not just for the hourly snapshots (that are showing up in the Rockstor web GUI) but also for the daily ones (that aren’t).

So can I toggle the “writable” flag on a snapshot and then roll back to it? How would I do that?

Or would they have to have been set writable at the time they were created? I do see there’s a “Make snapshots writable?” checkbox in the scheduled task settings.

5

Okay, I found the documentation: Internal implementation of Pools, Shares, Snapshots and Clones

Snapshot is a BTRFS snapshot of a Subvolume(aka Share). Snapshots of a Share are placed in /mnt2/<pool name>/.snapshots/<share name>/<snapshot name>.

So here’s what’s in /mnt2/rockons/.snapshots/mariadb-data:

drwxr-xr-x 1 mariadb mariadb 228 Nov 29 03:42 mariadata_201611290342 drwxr-xr-x 1 mariadb mariadb 228 Nov 29 03:42 mariadata_201612040300 drwxr-xr-x 1 mariadb mariadb 228 Dec 6 03:42 mariadata_201612060342 drwxr-xr-x 1 mariadb mariadb 550 Feb 18 01:30 mariadata_201702180150 drwxr-xr-x 1 mariadb mariadb 596 Feb 18 01:50 mariadata_201702180250 drwxr-xr-x 1 mariadb mariadb 642 Feb 18 02:50 mariadata_201702180350 drwxr-xr-x 1 mariadb mariadb 688 Feb 18 03:50 mariadata_201702180450 drwxr-xr-x 1 mariadb mariadb 734 Feb 18 04:50 mariadata_201702180550 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 05:50 mariadata_201702180650 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 06:50 mariadata_201702180750 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 07:50 mariadata_201702180850 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 08:50 mariadata_201702180950 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 09:50 mariadata_201702181050 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 10:50 mariadata_201702181150 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 11:50 mariadata_201702181250 drwxr-xr-x 1 mariadb mariadb 228 Oct 25 03:42 reverted-working-again

So it sure looks like none of my daily, weekly, or monthly snapshots are being taken. Which is pretty alarming.

(EDIT: Almost none. On a second look, the 1:30 AM snapshot is a daily snapshot. However, as of the following morning, it’s gone. There’s a new daily snapshot for Feb 19, but the Feb 18 one isn’t there anymore. So it appears that daily snapshots are being taken, but that they’re being treated as if the “Maximum Count” setting were 1 instead of 7.)

For an example of how I’m doing it, here’s how the Daily snapshot is set up:

Task name: snapshot-mariadata-daily
Task type: snapshot
Share: mariadb-data
Snapshot prefix: mariadata
Maximum count: 7
Make snapshots writable? Yes (was previously no; have changed to Yes at @Flyer’s suggestion)
Make snapshots visible? Yes
Enable task? Yes
Task frequency: Every day at 01:30

The Daily snapshot says that it finished, every day for the past 15 days.

The Weekly snapshot says that it finished on February 14th and 7th, errored on January 31st, 24th, 7th, and 10th, and finished on January 3rd and December 27th, 20th, and 13th.

The Monthly snapshot says that it had en error on February 4, but that it finished on January 4.

First of all, can anyone figure out what’s happening? If the daily, weekly, and monthly snapshots are finishing, why aren’t they showing up in /mnt2/rockons/.snapshots/mariadb-data? Is there anywhere else they could possibly be? (I notice that /mnt2/rockons/.snapshots/owncloud-data doesn’t even exist, and yet I can see OC snapshots in the Web interface. They must be somewhere. EDIT: I remembered owncloud-data is on a different pool.)

And second, what are my options at this point? Is there anything I can do besides wiping out my database and starting over from the beginning? Is there a way I can get Owncloud to attempt to repair the DB without having access to its web GUI?

6

This morning, after shutting down and restarting the MariaDB and Owncloud rock-ons, they are now working again. I have no idea why. (This was, of course, the very first thing I tried when the problem started; I don’t know why it worked today even though it didn’t work then.)

But the issue with missing snapshots persists. Here’s how my mariadb-data snapshots look now:

drwxr-xr-x 1 mariadb mariadb 228 Nov 29 03:42 mariadata_201611290342 drwxr-xr-x 1 mariadb mariadb 228 Nov 29 03:42 mariadata_201612040300 drwxr-xr-x 1 mariadb mariadb 228 Dec 6 03:42 mariadata_201612060342 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 22:50 mariadata_201702182350 drwxr-xr-x 1 mariadb mariadb 780 Feb 18 23:50 mariadata_201702190050 drwxr-xr-x 1 mariadb mariadb 550 Feb 19 01:30 mariadata_201702190130 drwxr-xr-x 1 mariadb mariadb 550 Feb 19 01:30 mariadata_201702190150 drwxr-xr-x 1 mariadb mariadb 596 Feb 19 01:50 mariadata_201702190250 drwxr-xr-x 1 mariadb mariadb 642 Feb 19 02:50 mariadata_201702190350 drwxr-xr-x 1 mariadb mariadb 688 Feb 19 03:50 mariadata_201702190450 drwxr-xr-x 1 mariadb mariadb 734 Feb 19 04:50 mariadata_201702190550 drwxr-xr-x 1 mariadb mariadb 780 Feb 19 05:50 mariadata_201702190650 drwxr-xr-x 1 mariadb mariadb 780 Feb 19 06:50 mariadata_201702190750 drwxr-xr-x 1 mariadb mariadb 780 Feb 19 07:50 mariadata_201702190850 drwxr-xr-x 1 mariadb mariadb 780 Feb 19 08:50 mariadata_201702190950 drwxr-xr-x 1 mariadb mariadb 228 Oct 25 03:42 reverted-working-again

But here’s what shows up in the Rockstor web GUI:

mariadata_201702190950 February 19th 2017, 9:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190850 February 19th 2017, 8:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190750 February 19th 2017, 7:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190650 February 19th 2017, 6:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190550 February 19th 2017, 5:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190450 February 19th 2017, 4:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190350 February 19th 2017, 3:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190250 February 19th 2017, 2:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190150 February 19th 2017, 1:50:04 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190130 February 19th 2017, 1:30:06 am Visible Yes 620.09 MB 320.00 KB mariadata_201702190050 February 19th 2017, 12:50:05 am Visible Yes 620.09 MB 320.00 KB mariadata_201702182350 February 18th 2017, 11:50:05 pm Visible Yes 620.09 MB 320.00 KB reverted-working-again December 10th 2016, 1:46:50 pm Hidden No 537.01 MB 243.98 MB

See the difference? The snapshots from 20161129, 20161204, and 20161206 aren’t listed.

I’ve followed the directions from the Internal Implementation thread to make the snapshots visible (mount -o subvol=.snapshots/mariadb-data/mariadata_201612060342 /dev/sdb /mnt2/mariadb-data/.mariadata_201612060342 etc.), rebooted Rockstor, and they’re still not there in the GUI.

So, while Owncloud and MariaDB are working for now, my snapshot issues persist:

  • Not all snapshots are being listed in the GUI, even after I mount them on the command line to make them visible;

  • Daily snapshots are not being preserved. They’re being taken, but they’re being removed after a day. For example, I have the mariadata_201702190130 snapshot that was taken at 01:30 this morning, but the one that was taken at 01:30 yesterday morning is gone.

  • Weekly and monthly snapshots are gone too. It’s not clear whether they’re not being taken at all, or whether they’re being taken and then removed in the same way that Daily ones are.

I’ve just thought of something: maybe it’s the filenames. I’ve used the same prefix (“mariadata”) for my hourly, daily, weekly, and monthly snapshots, and I currently have the most recent 12 snapshots listed. Not the most recent 12 hourly snapshots; there are 11 hourly snapshots and 1 daily snapshot.

This suggests it could be the filenames, and that the hourly scheduled task is removing all but the 12 most recent snapshots beginning with “mariadata_”.

I’m going to try changing them to use different prefixes (mariadata-hourly, mariadata-daily, etc.) and see if that makes a difference. It’ll take a couple of days to know for sure, but I’ll report back.

7

Confirmed; that was it. I changed the names and now daily and weekly snapshots are showing up correctly.

So now I know: don’t use the same naming convention for different scheduled snapshots. (Is that in the docs? I don’t remember seeing it.)

Now I just have to go through and rename the rest of my scheduled snapshots. I wish there were an easier way to do it than deleting and re-adding them.

1 Like