I installed Rockstor from the latest ISO about two weeks ago. I created two NFS exports, and enabled snapshots on one.
A few days ago, I subscribed to and applied the latest updates. After upgrading, I rebooted. Since the reboot, I’ve been having three problems:
(First) Most notably, the data in my NFS exports are gone. Nothing is getting mounted under /export. The exports are listed in /etc/exports, but the directories are empty. There are no entries for exports in /etc/fstab.
(Second,) I am getting emails from anacron indicating errors in the snapshots.
Traceback (most recent call last):
File “/opt/rockstor/bin/st-snapshot”, line 45, in
sys.exit(scripts.scheduled_tasks.snapshot.main())
File “/opt/rockstor/src/rockstor/scripts/scheduled_tasks/snapshot.py”, line 95, in main
validate_snap_meta(meta)
File “/opt/rockstor/src/rockstor/scripts/scheduled_tasks/snapshot.py”, line 39, in validate_snap_meta
if meta[‘share’].isdigit():
AttributeError: ‘int’ object has no attribute ‘isdigit’
(Finally,) I am unable to create new NFS shares (which is what led to this bug report, full details below).
##### Houston, we've had a problem.
A lower level error occurred while refreshing NFS exports: (Error running a command. cmd = /usr/bin/mkdir -p /export/data/.nfs_data_201911020300. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \xe2\x80\x98/export/data/.nfs_data_201911020300\xe2\x80\x99: Operation not permitted', '']).
NOTE THE HEX ESCAPE SEQUENCES before /export in the mkdir command above! These don’t appear normal, but I don’t know where they would have come from.
Detailed step by step instructions to reproduce the problem
created new share.
attempted to create a new NFS export using new share.
Web-UI screenshot
[Drag and drop the image here]
Error Traceback provided on the Web-UI
Traceback (most recent call last):
File "/opt/rockstor/src/rockstor/storageadmin/views/nfs_exports.py", line 139, in refresh_wrapper
refresh_nfs_exports(exports)
File "/opt/rockstor/src/rockstor/system/osi.py", line 599, in refresh_nfs_exports
bind_mount(exports[e][0]['mnt_pt'], e)
File "/opt/rockstor/src/rockstor/system/osi.py", line 569, in bind_mount
run_command([MKDIR, '-p', export_pt])
File "/opt/rockstor/src/rockstor/system/osi.py", line 121, in run_command
raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/mkdir -p /export/data/.nfs_data_201911020300. rc = 1. stdout = ['']. stderr = ['/usr/bin/mkdir: cannot create directory \xe2\x80\x98/export/data/.nfs_data_201911020300\xe2\x80\x99: Operation not permitted', '']
After fixing the mounts (which must have failed during boot), I am able to successfully create the new NFS export.
After a reboot, the data is gone again. The volumes are mounted under /mnt2, but not under /export.
There are some errors in /var/log/messages:
Nov 18 23:11:32 rockstor systemd: Starting NFS Mount Daemon...
Nov 18 23:11:32 rockstor systemd: Starting NFS status monitor for NFSv2/3 locking....
Nov 18 23:11:32 rockstor systemd: Started NFS status monitor for NFSv2/3 locking..
Nov 18 23:11:32 rockstor systemd: Started NFS Mount Daemon.
Nov 18 23:11:32 rockstor systemd: Starting NFS server and services...
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911100300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911090300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911080300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911110300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911130300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911120300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911050300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911040300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911070300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911060300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911030300: No such file or directory
Nov 18 23:11:32 rockstor exportfs: exportfs: Failed to stat /export/data/.nfs_data_201911020300: No such file or directory
Nov 18 23:11:32 rockstor kernel: NFSD: starting 90-second grace period (net ffffffff81dd5400)
Nov 18 23:11:32 rockstor systemd: Started NFS server and services.
Nov 18 23:11:32 rockstor systemd: Starting Notify NFS peers of a restart...
Nov 18 23:11:32 rockstor systemd: Started Notify NFS peers of a restart.
Unfortunately the internal format of scheduled tasks created pre stable update from our now ageing iso changed in the interim time, which broke prior tasks. The fix is to delete and re-create those scheduled tasks. This was a pain for all concerned but we had to make major API changes and although we tried to seamlessly migrate these changes from prior Stable installs in the case of the scheduled tasks this was not the case mostly.
So in short you will have to delete and re-create all the scheduled tasks you created prior to doing the stable updates subscription. They should then be good as the indicated API changes are now done and will then be in place with our next ISO. Apologies for this, as I say it was a pain all around but necessary to allow for greater flexibility going forward, such as pool / share renaming (pending).
There are other form members far more familiar with NFS than myself but I’ll try and take a look later at this issue and chip in if I can, but from a quick look you may be using a configuration that is not understood by Rockstor. I.e. the main support from the Web-UI is in exporting shares only. We have the following doc entry that may help: Network File System (NFS)
i.e. exporting a share “jenkins_home” via the Web-UI results in:
/export/jenkins_home *(rw,async,insecure)
And I don’t believe /etc/fstab is involved in this operation as Rockstor is doing the mounting.
mount | grep export
/dev/sda3 on /export/jenkins_home type btrfs (rw,relatime,space_cache,subvolid=326,subvol=/jenkins_home)
Again I’m pretty rusty with NFS so hopefully others will chip in before I get a chance to have a proper look.
But form a client machine the above export is then visible via:
sudo showmount -e rockprod.local
...
Export list for rockprod.local:
/export/jenkins_home *
Sorry not to be of more help on this but right now, and thanks for the details reports. If you could add exactly how you are creating these exports that might also help others chip in. I.e. are you using “Enter manual export entries below.” method?
Thanks for the response. I’ve deleted my scheduled snapshots, so I’ll see if they run tonight.
The exports were created through the UI – I created a share, then made it an NFS export.
What has me puzzled is the exports are entered into /etc/exports correctly, but the exports are pointed to /export rather than /mnt2. The shares are being mounted under /mnt2 correctly, but the corresponding entries under /export are empty.
My mounts include the lines:
/dev/sdc on /mnt2/isos type btrfs (rw,relatime,compress=lzo,space_cache,subvolid=344,subvol=/isos)
/dev/sdc on /mnt2/data type btrfs (rw,relatime,compress=lzo,space_cache,subvolid=258,subvol=/data)
/dev/sdc on /mnt2/ovirt type btrfs (rw,relatime,compress=lzo,space_cache,subvolid=363,subvol=/ovirt)
But after a reboot, there is nothing in the /export mountpoints. Normally, the mounts would be listed in /etc/fstab, but it appears to be ignored for both /mnt2 and /export in this case.
Where are the volume/mountpoint mappings maintained?
If you take a look at the following forum post where the various Rockstor services were discussed, in a related but different context, you will see the rough structure of the services:
But to accomplish what Rockstor does, or tries to do in some circumstances :), is actually more complicated than might first be thought. So it turns out that volume/mountpoint mappings is mostly what Rockstor does in the background, in the context of user programmed preferences via Web-UI input stored in a database (Django/Postgresql). So this question is hard to answer briefly; but in short it’s instantiated via our codes interpretation of the database of prior entered preferences or defaults.
Essentially first rockstor-pre.service runs /opt/rockstor/bin/initrock which in turn runs /opt/rockstor/src/rockstor/scripts/initrock.py
(basic sanity checks and db migration if required)
rockstor.service runs “/opt/rockstor/bin/supervisord -c /opt/rockstor/etc/supervisord.conf”
further environment (more sub services) setup.
Followed finally by rockstor-bootstrap which kicks off the mount stuff essentially by running: /opt/rockstor/bin/bootstrap which setups up the env to run /opt/rockstor/src/rockstor/scripts/bootstrap.py which in turn mainly does some basic btrfs prep, some rockstor network setup (which uses nmcli under the hood) and to your request finally does an API call on commands/bootstrap, hence the services name.
So we finally get to what kicks off the mounts etc in the API here:
which continues one for a bit to try and account for all supported capabilities.
This is to ‘top’ or bottom level of rockstor depending on how you look at it.
And to your specific issue, rockstor failing to setup NFS mounts after reboot with your vol/subvol/exports setup, we have a little further down the in that same file the following:
Having now hopefully answered that question, at least initially I have attempted to reproduce your issue with the following setup via Rockstor’s Web-UI (non advanced mode, ie just via form entry) and have the following directly after setting 3 NFS exports up:
with the consequent mounts setup ‘live’ via Rockstor as expected via the given command:
mount | grep 'data\|ovirt\|isos'
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/sdb on /mnt2/data type btrfs (rw,relatime,space_cache,subvolid=310,subvol=/data)
/dev/sdb on /mnt2/isos type btrfs (rw,relatime,space_cache,subvolid=309,subvol=/isos)
/dev/sdb on /mnt2/ovirt type btrfs (rw,relatime,space_cache,subvolid=311,subvol=/ovirt)
/dev/sdb on /export/data type btrfs (rw,relatime,space_cache,subvolid=310,subvol=/data)
/dev/sdb on /export/ovirt type btrfs (rw,relatime,space_cache,subvolid=311,subvol=/ovirt)
/dev/sdb on /export/isos type btrfs (rw,relatime,space_cache,subvolid=309,subvol=/isos)
These command outcomes were unaffected post a subsequent reboot.
However on trying ‘harder’ to reproduce you issue I think I have identified a bug re our mounts of NFS. I am as yet unsure if this could lead to your exact issue but theirs definitely something not quite right here. But I strongly suspect it is related. Try also using my above command re the grep of mount. I’m hoping this may give us more info on what’s going wrong here.
Could you also look to your systems journalctl and the contents of the main rockstor.log:
less /opt/rockstor/var/log/rockstor.log
Directly after a reboot where you had previously established NFS exports.
From the latter above code snippet a failure in the NFS part of bootstrap their should have left an “Exception while bootstrapping NFS: …” log entry which may be of some use in pinning down why your NFS bootstrap fails and mine here does not (yet at least).
Yes, as these subvols are rockstor managed/mounted and not fstab managed they don’t appear in /etc/fstab. Fstab is the origin of a request not the reflection of a mount existing. So stuff in fstab is enacted at boot time by systemd these days but as we manage the mounting the rockstor code does, or tries to do, these mounts such that it can apply the Web-UI stuff etc without live editing /etc/fstab which is left to manage the base system critical mounts only.
Once we manage to pin down the exact reason for your observed failure, I can create an issue with it’s definition. This should in time lead to a fix. But we first have to identify the exact reason and get a simple reproducer so we an define the issue appropriately.
Apologies for the slow response but NFS is not something I’ve looked at for a while, both in itself and Rockstor’s implementation of it.
I’ve been unable to reproduce the issue I thought may have been related thus far and have so far tested the following, setup via Web-UI, as the closest to yours from the information provided:
mount | grep 'data\|ovirt\|isos'
...
/dev/sdb on /mnt2/data type btrfs (rw,relatime,space_cache,subvolid=310,subvol=/data)
/dev/sdb on /mnt2/isos type btrfs (rw,relatime,space_cache,subvolid=309,subvol=/isos)
...
/dev/sdb on /mnt2/ovirt type btrfs (rw,relatime,space_cache,subvolid=311,subvol=/ovirt)
/dev/sdb on /export/data type btrfs (rw,relatime,space_cache,subvolid=310,subvol=/data)
/dev/sdb on /export/ovirt type btrfs (rw,relatime,space_cache,subvolid=311,subvol=/ovirt)
/dev/sdb on /export/isos type btrfs (rw,relatime,space_cache,subvolid=309,subvol=/isos)
And this arrangement then persisted post reboot.
So we are still looking for a simple reproducer of your observed issue of NFS mounts not persisting reboot as the closest I can replicate here from your last exports file info looks to be working OK. I think we need more log entries if possible to see if their is a reported reason for the NFS bootstrap stuff to be failing.
