NFS and snapshots not working well together

delboy · May 23, 2025, 4:07pm

Long story short. I find I am unable to change the owner of a share, delete any visible snapshot, or add/modify an NFS share. All because I had first created an NFS share on a share that had visible snapshots, and then deleted the NFS share.

Creating an NFS share puts a line entry in /etc/export as expected and also creates a mount point in /exports/‘share name’

Deleting an NFS share removes the entry in /etc/export but does NOT remove the mount point in /export.

Now if you try to do anything in the UI regarding this share such as changing ownership or deleting a snapshot, then the system will try to create a directory /export/'share name'/.'snapshot name'
This command will fail with ‘not permitted’ (only visible snapshots cause this reply)
Even trying to delete a snapshot on a different share which has never been shared by NFS will cause this response.

I have tried to delete the mount points by hand through the command line, but still get ‘not permitted’ even though they are shown as permissions 755

After a long struggle I finally found the solution.
The system creates the mount points in /export as ‘immutable’ and in order to delete them execute the command
chattr -i /export/'share name'
and then rmdir will work.

Then snapshots can be deleted although it may be necessary to create a folder mkdir /export/'share name'/.'snapshot name' first and then delete it again when the snapshot is deleted

The main lesson I have learned is to keep away from NFS!. That way there be monsters.

I see there have been several posts over the years with similar problems although none of them seem to have found the solution of using chattr to remove the immutable file attribute. (I had never heard of it before I researched this problem)

Hooverdan · May 23, 2025, 7:22pm

Interesting, and thanks for putting down your experiences here.

You are correct that it’s creating the mounts as immutable, per:

github.com

rockstor/rockstor-core/blob/852877ecc8a78e0b9fcc2406e2af314305e8dfc1/src/rockstor/system/osi.py#L567-L572


      
          def bind_mount(mnt_pt, export_pt):
              if not is_mounted(export_pt):
                  run_command([MKDIR, "-p", export_pt])
                  toggle_path_rw(export_pt, rw=False)
                  return run_command([MOUNT, "--bind", mnt_pt, export_pt])
              return True

calling

toggle_path_rw(export_pt, rw=False)

github.com

rockstor/rockstor-core/blob/852877ecc8a78e0b9fcc2406e2af314305e8dfc1/src/rockstor/system/osi.py#L542-L546


      
          def toggle_path_rw(path, rw=True):
              attr = "-i"
              if not rw:
                  attr = "+i"
              return run_command([CHATTR, attr, path])

during the teardown when looking at the coding, it seems that the immutable flag is removed (-i)

github.com

rockstor/rockstor-core/blob/852877ecc8a78e0b9fcc2406e2af314305e8dfc1/src/rockstor/system/osi.py#L549-L564


      
          def nfs4_mount_teardown(export_pt):
              """
              reverse of setup. cleanup when there are no more exports
              """
              if is_mounted(export_pt):
                  run_command([UMOUNT, "-l", export_pt])
                  for i in range(10):
                      if not is_mounted(export_pt):
                          toggle_path_rw(export_pt, rw=True)
                          return run_command([RMDIR, export_pt])
                      time.sleep(1)
                  run_command([UMOUNT, "-f", export_pt])
              if os.path.exists(export_pt):
                  toggle_path_rw(export_pt, rw=True)
                  run_command([RMDIR, export_pt])
              return True

it calls the

toggle_path_rw(export_pt, rw=True)

and then it is supposed to proceed to remove the directory.

I would have expected that if the removal failed like in your case, the program would generate some error message (may be in the log only /opt/rockstor/var/log/rockstor.log?).

delboy · May 25, 2025, 10:22am

Thanks for the reply. It shows me where the problem area is. Clearly the code is not working as intended. I would like to get to the bottom of this and get it fixed. I am a bit busy at the moment, but over the next week I will get a Rockstor instance set up on a VM to use as a test bed and will start poking about in the code.

delboy · May 31, 2025, 11:22am

Well I now have a test bed set up on a Proxmox VM and can replicate the problem. It is caused as I thought by Rockstor creating a new NFS export for each snapshot, and then not successfully deleting them when the export for the owning share is deleted.

For example here the export for share 1 has been deleted, but its snapshot still has an NFS export

more /etc/exports
/export/share2 192.168.1.33(rw,no_root_squash,sync,insecure) 
/export/share1/.snap1-1 192.168.1.33(rw,async,insecure,nohide) 
/export/share2/.snap2-1 192.168.1.33(rw,async,insecure,nohide)

I have modified the code in accordance with the guidance for developers in the documentation and have added extra logging lines so I can see how the data changes and am working on a fix.

Being a new user and unfamiliar with the Rockstor architecture, I find my self wondering WHY Rockstor creates a new NFS export for each snapshot at all?

Snapshots are already visible as hidden folders within NFS share exports. They do not need their own separate NFS export to be visible to the end user.

I have compared the operation with Samba shares where snapshots are NOT given their own separate samba share.

So would an appropriate fix be to simply ignore snapshots when creating/deleting NFS exports? (KISS)

I will raise an issue on github for this problem.

delboy · May 31, 2025, 12:27pm

After further research I find that in order to be visible in an NFS export a snapshot needs to be mounted on /export/share/snapshot, but does not need its own line in /etc/exports. So it is not just a matter of ignoring snapshots when dealing with NFS.