Can't access NFS shares that have snapshots

[mattyvau]

I have just discovered that some of my nfs shares are no longer accessible. The ones affected are those I have a scheduled snapshot task for. I have read the wiki detailing how you can’t rollback a snapshot while exported as a share, but I haven’t tried that and they previously worked.

Checking the web-UI scheduled tasks page, recent snapshots have failed with an unspecified “error”. Digging into journalctl, I only see:

-- The start-up result is done.
Feb 13 01:15:01 [myhostname] CROND[24965]: (root) CMD (/opt/rockstor/bin/st-snapshot 3 \*-*-*-*-*-*)
Feb 13 01:15:03 [myhostname] systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

If I try to delete the NFS share, the following error occurs:

If I try to delete either a snapshot from the schedule (read-only) or a manually created writeable snapshot, that doesn’t work either:

So it seems I’m stuck in a bit of a loop. The “failed” snapshots started around 2 weeks ago, so I wonder if part of the issue is related to "You are running Setuptools on Python 2, which is no longer supported" - #5 by Hooverdan

Any ideas how to move forward with this?

[mattyvau]

I’m still having trouble with this, but have had a chance to do a little more investigating.

Whilst I thought that the NFS mounts were empty, it turns out they are mounting my read-only snapshots, which all start with .xyz and thus appear hidden. Over SMB, there is no such issue. Showing hidden files in a file manager reveals the directory structure:

It takes a few levels to reach “Camera”, “Pictures” etc. And those directories remain unwriteable.

I have read through some related threads NFS is mounting empty - #2 by Hooverdan and A lower level error occurred while refreshing NFS exports: (Error running a command. cmd - #7 by phillxnet so have some other output to share.

less /opt/rockstor/var/log/rockstor.log

Gives:

ERROR [storageadmin.views.command:182] **Exception while bootstrapping NFS:** (["A lower level error occurred while refreshing NFS exports: (Error running a command. cmd = /usr/bin/mkdir -p /export/iron-photos/.photos-weekly_202001091515. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \\xe2\\x80\\x98/export/iron-photos/.photos-weekly_202001091515\\xe2\\x80\\x99: Operation not permitted', '']).", 'Traceback (most recent call last):\n  File "/opt/rockstor/src/rockstor/storageadmin/views/nfs_exports.py", line 139, in refresh_wrapper\n    refresh_nfs_exports(exports)\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 598, in refresh_nfs_exports\n    bind_mount(exports[e][0][\'mnt_pt\'], e)\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 568, in bind_mount\n    run_command([MKDIR, \'-p\', export_pt])\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 120, in run_command\n    raise CommandException(cmd, out, err, rc)\nCommandException: Error running a command. cmd = /usr/bin/mkdir -p /export/iron-photos/.photos-weekly_202001091515. rc = 1. stdout = [\'\']. stderr = [\'/usr/bin/mkdir: cannot create directory \\xe2\\x80\\x98/export/iron-photos/.photos-weekly_202001091515\\xe2\\x80\\x99: Operation not permitted\', \'\']\n']).

So you can see it’s not trying to mount the top level directory as defined in the NFS export on the Rockstor web-UI, but instead a read-only subvolume. This creates an error trying to create a directory.

[root@rockstor ~]# mount | grep ‘iron-photos’
… trimmed for readability
/dev/sde on /mnt2/iron-photos/.photos-weekly_202001310815 type btrfs (rw,relatime,space_cache,subvolid=470,subvol=/.snapshots/iron-photos/photos-weekly_202001310815)
/dev/sde on /export/iron-photos/.photos-weekly_202001091515 type btrfs (rw,relatime,space_cache,subvolid=446,subvol=/.snapshots/iron-photos/photos-weekly_202001091515)
/dev/sde on /export/iron-photos/.photos-monthly_202001042105 type btrfs (rw,relatime,space_cache,subvolid=439,subvol=/.snapshots/iron-photos/photos-monthly_202001042105)

A series of snapshots mounted as expected, then mounts to /exports are created from snapshots that are read only.

I also can’t find a way to:
a) remove the NFS mount definition from the web-UI due to permission errors
b) remove existing snapshots to at least go back to normal use until the NFS issue can be worked out
The issues seem to have created a bit of a loop.

Any help appreciated.

[phillxnet (Philip Guyton)]

@mattyvau Hello again:

Your findings are curios:

@mattyvau

a) remove the NFS mount definition from the web-UI due to permission errors

This brings to mind some immutable flags that especially older Rockstor versions tended to leave laying around. Take a look at the following forum post for @Jorma_Tuomainen’s notes:

Can't start rock-ons

Not docker, straced what it does and: # chmod 711 /mnt2/rockons chmod: changing permissions of ‘/mnt2/rockons/’: Operation not permitted

We also have, which may pertain to your:

@mattyvau

b) remove existing snapshots to at least go back to normal use until the NFS issue can be worked out

The following thread

[SOLVED] Cant delete empty snapshot folders created by Snapshot scheduled tasks

I’ve been playing with different automated ways of performing send/receive to allow for a more resilient backup setup and during this I noticed that in one of my shares I have what appears to be some orphaned directories that were created by the Scheduled task jobs I had created to perform snapshots on the shares. So for example, I have the following: [root@rockstor ]# tree -L 2 -a /mnt2/Docker-Persistent-Storage/ …

which in turn references similar immutable flag issues that were reported / worked around by @Haioken and @Rene_Castberg

Linked again here in case it’s relevant:

SOLVED: BTRFS Subvolume create/delete fail

Problem has been resolved, though I’m unsure why it had happened. Immutable flag was set on /mnt2/tempraid, meaning that the contents of /mnt2/tempraid was unmodifiable (though decendants could be altered). I know I didn’t set the flag, and according to BTRFS IRC, scrubs do not set the flag in the event of error. Problem was resolved with chattr -i /mnt2/tempraid Should have thought of it before - in fact, I could swear I checked it, but apparently I did not.

and

[solved] Unable to create new shares

Hi, hope you can help. Been playing with RS for about a week, so far so good. I’ve installed a plex server and all is running smoothly. now i’m trying to create a new share for transmission and it pops an error. recreate: storage > shares > create new here is the msg Error running a command. cmd = /sbin/btrfs subvolume create -i 2015/24 /mnt2/rockstor_rockstor/transm-data. rc = 1. stdout = [“Create subvolume ‘/mnt2/rockstor_rockstor/transm-data’”, ‘’]. stderr = [‘ERROR: cannot create subvolum…

Maybe there has been a similar rought immutable attribute that in hindering your circumstance.

Sorry I can’t put more time into this one at the moment. I’m hoping that now we have remove the AFP support we can focus more on improving what remains, ie samba and NFS mainly. And the NFS stuff hasn’t seen any attention for a while now.

Hope that helps.

[mattyvau]

Sorry I can’t put more time into this one at the moment.

No worries, thanks for having a brief look.

Through other searches I did come across the immutable mount issue and did a check on that. Had forgotten about it. The only directory I see that that flag under /mnt2 is root, so I think this looks correct.

[root@rockstor ~]# lsattr /mnt2/
---------------- /mnt2/rockstor_rockstor
---------------- /mnt2/home
----i----------- /mnt2/root
---------------- /mnt2/Red2x3
--------c------- /mnt2/Duplicati-Config
--------c------- /mnt2/Documents
---------------- /mnt2/Glenn
---------------- /mnt2/emby-media
---------------- /mnt2/sonarr-data
---------------- /mnt2/Photography
---------------- /mnt2/radarr-data
--------c------- /mnt2/sonarr-config
--------c------- /mnt2/radarr-config
---------------- /mnt2/transmission-data
--------c------- /mnt2/rock-ons-root
---------------- /mnt2/Video
---------------- /mnt2/nzbget-conig
---------------- /mnt2/Music
--------c------- /mnt2/emby-config
---------------- /mnt2/Iron4
---------------- /mnt2/iron4-backups
---------------- /mnt2/iron-photos

[mattyvau]

Small update. Good news, system now working properly!

Roughly what I did was disable NFS, manually unmount all the snapshots that were mounted at /exports/ then delete those snapshots. After a little testing, it seems NFS does not like “visible” snapshots but seems fine if that option is unchecked. I can safely enable NFS and scheduled snapshots on those volumes again, and NFS presents the volumes in the same way as samba.

I wasn’t clear on what the “visible” option was when setting up the schedules, perhaps a little extra documentation on that could be useful at some stage.

[phillxnet (Philip Guyton)]

@mattyvau Thanks for the update and glad you managed to get to the cause of all this. Well done.

@mattyvau

After a little testing, it seems NFS does not like “visible” snapshots but seems fine if that option is unchecked.

Nice find.

@mattyvau

… perhaps a little extra documentation on that could be useful at some stage.

Definitely. If you fancy doing a pull request on our docs repo I should be able to publish it fairly quickly.

GitHub - rockstor/rockstor-doc: Rockstor documentation

And the following looks to be the relevant file:

https://github.com/rockstor/rockstor-doc/blob/master/nfs_ops.rst

Thanks for sharing your findings on this one. We should probably also have an issue in rockstor-core as well with a tooltip or the like, at least for now, on visible snapshots of NFS shares being problematic.

[mattyvau]

Happy to look into a pull request (haven’t made one before). I will try to get to it on the weekend once I do a bit more reading on the topic of visible snapshots.

[phillxnet (Philip Guyton)]

@mattyvau Hello again.

While working on my current issue I’ve been making some changes to a file in Rockstor that relates to the visible snapshot mounting:

rockstor/rockstor-core/blob/master/src/rockstor/storageadmin/views/snapshot.py#L68-L87

I’m in the process of black formatting and moving to string.format in this file as well so do keep that in mind as if you end up doing a code pr involving this file you may have to rebase once I’ve got my current pull request finally sorted. Nearly there now so shouldn’t be too much longer. Plus I’m hoping the changes needed with regard to the NFS snapshot visible thing are fairly minor.

Just thought I’d mention here as this thread came to mind when I was there. Not certain if this is where the problem you encountered lies, code wise, but you can trace back/forward from there anyway so it’s at least a place to start.

Hope that helps.

[phillxnet (Philip Guyton)]

@mattyvau With regard to the following:

Happy to look into a pull request

That would be much appreciated; I’d forgot to link to our Community Contributions: http://rockstor.com/docs/contribute_section.html# doc section. And with regard to code contributions there is the subsection:

Developers: http://rockstor.com/docs/contribute.html#developers

Which should hopefully be of use here. That doc section goes through the git / GitHub side of getting started with a development setup so I had meant to post here previously.

Hope that helps.