Can't access NFS shares that have snapshots

I have just discovered that some of my nfs shares are no longer accessible. The ones affected are those I have a scheduled snapshot task for. I have read the wiki detailing how you can’t rollback a snapshot while exported as a share, but I haven’t tried that and they previously worked.

Checking the web-UI scheduled tasks page, recent snapshots have failed with an unspecified “error”. Digging into journalctl, I only see:

-- The start-up result is done.
Feb 13 01:15:01 [myhostname] CROND[24965]: (root) CMD (/opt/rockstor/bin/st-snapshot 3 \*-*-*-*-*-*)
Feb 13 01:15:03 [myhostname] systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

If I try to delete the NFS share, the following error occurs:
2020-02-15_10-54

If I try to delete either a snapshot from the schedule (read-only) or a manually created writeable snapshot, that doesn’t work either:
2020-02-15_10-54

So it seems I’m stuck in a bit of a loop. The “failed” snapshots started around 2 weeks ago, so I wonder if part of the issue is related to "You are running Setuptools on Python 2, which is no longer supported"

Any ideas how to move forward with this?

I’m still having trouble with this, but have had a chance to do a little more investigating.

Whilst I thought that the NFS mounts were empty, it turns out they are mounting my read-only snapshots, which all start with .xyz and thus appear hidden. Over SMB, there is no such issue. Showing hidden files in a file manager reveals the directory structure:
image
It takes a few levels to reach “Camera”, “Pictures” etc. And those directories remain unwriteable.

I have read through some related threads NFS is mounting empty and A lower level error occurred while refreshing NFS exports: (Error running a command. cmd so have some other output to share.

less /opt/rockstor/var/log/rockstor.log

Gives:

ERROR [storageadmin.views.command:182] **Exception while bootstrapping NFS:** (["A lower level error occurred while refreshing NFS exports: (Error running a command. cmd = /usr/bin/mkdir -p /export/iron-photos/.photos-weekly_202001091515. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \\xe2\\x80\\x98/export/iron-photos/.photos-weekly_202001091515\\xe2\\x80\\x99: Operation not permitted', '']).", 'Traceback (most recent call last):\n  File "/opt/rockstor/src/rockstor/storageadmin/views/nfs_exports.py", line 139, in refresh_wrapper\n    refresh_nfs_exports(exports)\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 598, in refresh_nfs_exports\n    bind_mount(exports[e][0][\'mnt_pt\'], e)\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 568, in bind_mount\n    run_command([MKDIR, \'-p\', export_pt])\n  File "/opt/rockstor/src/rockstor/system/osi.py", line 120, in run_command\n    raise CommandException(cmd, out, err, rc)\nCommandException: Error running a command. cmd = /usr/bin/mkdir -p /export/iron-photos/.photos-weekly_202001091515. rc = 1. stdout = [\'\']. stderr = [\'/usr/bin/mkdir: cannot create directory \\xe2\\x80\\x98/export/iron-photos/.photos-weekly_202001091515\\xe2\\x80\\x99: Operation not permitted\', \'\']\n']).

So you can see it’s not trying to mount the top level directory as defined in the NFS export on the Rockstor web-UI, but instead a read-only subvolume. This creates an error trying to create a directory.

[root@rockstor ~]# mount | grep ‘iron-photos’
… trimmed for readability
/dev/sde on /mnt2/iron-photos/.photos-weekly_202001310815 type btrfs (rw,relatime,space_cache,subvolid=470,subvol=/.snapshots/iron-photos/photos-weekly_202001310815)
/dev/sde on /export/iron-photos/.photos-weekly_202001091515 type btrfs (rw,relatime,space_cache,subvolid=446,subvol=/.snapshots/iron-photos/photos-weekly_202001091515)
/dev/sde on /export/iron-photos/.photos-monthly_202001042105 type btrfs (rw,relatime,space_cache,subvolid=439,subvol=/.snapshots/iron-photos/photos-monthly_202001042105)

A series of snapshots mounted as expected, then mounts to /exports are created from snapshots that are read only.

I also can’t find a way to:
a) remove the NFS mount definition from the web-UI due to permission errors
b) remove existing snapshots to at least go back to normal use until the NFS issue can be worked out
The issues seem to have created a bit of a loop.

Any help appreciated.

1 Like

@mattyvau Hello again:

Your findings are curios:

This brings to mind some immutable flags that especially older Rockstor versions tended to leave laying around. Take a look at the following forum post for @Jorma_Tuomainen’s notes:

We also have, which may pertain to your:

The following thread

which in turn references similar immutable flag issues that were reported / worked around by @Haioken and @Rene_Castberg

Linked again here in case it’s relevant:


and

Maybe there has been a similar rought immutable attribute that in hindering your circumstance.

Sorry I can’t put more time into this one at the moment. I’m hoping that now we have remove the AFP support we can focus more on improving what remains, ie samba and NFS mainly. And the NFS stuff hasn’t seen any attention for a while now.

Hope that helps.

1 Like

No worries, thanks for having a brief look.

Through other searches I did come across the immutable mount issue and did a check on that. Had forgotten about it. The only directory I see that that flag under /mnt2 is root, so I think this looks correct.

[root@rockstor ~]# lsattr /mnt2/
---------------- /mnt2/rockstor_rockstor
---------------- /mnt2/home
----i----------- /mnt2/root
---------------- /mnt2/Red2x3
--------c------- /mnt2/Duplicati-Config
--------c------- /mnt2/Documents
---------------- /mnt2/Glenn
---------------- /mnt2/emby-media
---------------- /mnt2/sonarr-data
---------------- /mnt2/Photography
---------------- /mnt2/radarr-data
--------c------- /mnt2/sonarr-config
--------c------- /mnt2/radarr-config
---------------- /mnt2/transmission-data
--------c------- /mnt2/rock-ons-root
---------------- /mnt2/Video
---------------- /mnt2/nzbget-conig
---------------- /mnt2/Music
--------c------- /mnt2/emby-config
---------------- /mnt2/Iron4
---------------- /mnt2/iron4-backups
---------------- /mnt2/iron-photos

Small update. Good news, system now working properly!

Roughly what I did was disable NFS, manually unmount all the snapshots that were mounted at /exports/ then delete those snapshots. After a little testing, it seems NFS does not like “visible” snapshots but seems fine if that option is unchecked. I can safely enable NFS and scheduled snapshots on those volumes again, and NFS presents the volumes in the same way as samba.

I wasn’t clear on what the “visible” option was when setting up the schedules, perhaps a little extra documentation on that could be useful at some stage.

2 Likes

@mattyvau Thanks for the update and glad you managed to get to the cause of all this. Well done.

Nice find.

Definitely. If you fancy doing a pull request on our docs repo I should be able to publish it fairly quickly.

And the following looks to be the relevant file:

Thanks for sharing your findings on this one. We should probably also have an issue in rockstor-core as well with a tooltip or the like, at least for now, on visible snapshots of NFS shares being problematic.

1 Like

Happy to look into a pull request (haven’t made one before). I will try to get to it on the weekend once I do a bit more reading on the topic of visible snapshots.

2 Likes

@mattyvau Hello again.

While working on my current issue I’ve been making some changes to a file in Rockstor that relates to the visible snapshot mounting:

I’m in the process of black formatting and moving to string.format in this file as well so do keep that in mind as if you end up doing a code pr involving this file you may have to rebase once I’ve got my current pull request finally sorted. Nearly there now so shouldn’t be too much longer. Plus I’m hoping the changes needed with regard to the NFS snapshot visible thing are fairly minor.

Just thought I’d mention here as this thread came to mind when I was there. Not certain if this is where the problem you encountered lies, code wise, but you can trace back/forward from there anyway so it’s at least a place to start.

Hope that helps.

1 Like

@mattyvau With regard to the following:

That would be much appreciated; I’d forgot to link to our Community Contributions: http://rockstor.com/docs/contribute_section.html# doc section. And with regard to code contributions there is the subsection:

Developers: http://rockstor.com/docs/contribute.html#developers

Which should hopefully be of use here. That doc section goes through the git / GitHub side of getting started with a development setup so I had meant to post here previously.

Hope that helps.

1 Like