[SOLVED] Cant delete empty snapshot folders created by Snapshot scheduled tasks

I’ve been playing with different automated ways of performing send/receive to allow for a more resilient backup setup and during this I noticed that in one of my shares I have what appears to be some orphaned directories that were created by the Scheduled task jobs I had created to perform snapshots on the shares.

So for example, I have the following:

[root@rockstor ]# tree -L 2 -a /mnt2/Docker-Persistent-Storage/                                                                                
/mnt2/Docker-Persistent-Storage/                                                                                                                      
├── .Docker-Persistent-Storage-Backup_201809260200                                                                                                    
├── .Docker-Persistent-Storage-Backup_20180927                                                                                                        
├── .Docker-Persistent-Storage-Backup_201809280300                                                                                                    
├── .Docker-Persistent-Storage-Backup_201809290300                                                                                                    
├── .Docker-Persistent-Storage-Backup_201809300300                                                                                                    
├── ombi                                                                                                                                              
│   └── config                                                                                                                                        
├── portainer                                                                                                                                         
│   ├── compose                                                                                                                                       
│   ├── config.json                                                                                                                                   
│   ├── data                                                                                                                                          
│   ├── portainer.db                                                                                                                                  
│   ├── portainer.key                                                                                                                                 
│   ├── portainer.pub                                                                                                                                 
│   └── tls

The ‘.Docker-Persistent-Storage-Backup*’ directories were created as part of a scheduled task config’ed via the GUI. During my testing I had decided to remove the scheduled task and also to delete the relevant snapshots listed in the Shares > Snapshots for the aforementioned Share. Also reviewed the overall ‘Snapshots’ page in the GUI to ensure no other snapshots existed.

They all were removed fine from the UI but when navigating around in the CLI they still appear. They are not sub volumes (btrfs sub del doesn’t work) and a simple rm -rf also doesn’t work (says “operation not permitted” even as root user). They dont appear to be impacting anything but my OCD certainly finds them troublesome. Also, I’d just like to understand why they cant be simply removed…

Thanks for any input.

@t3kka Hello again, I can chip in on this one:

Yes, we have tracked this one down to our use of setting and un-setting the immutable flag on directories to avoid the scenario where a failure to mount a subvol doesn’t end up with it’s mount point being populated via a potential share export. We have more recently fixed, in the stable channel updates, a couple of instances where the code failed to un-set these flags to the detriment of normal functions but there may be some remaining.

See the following thread where @Haioken and @Rene_Castberg reported an earlier more problematic instance of this:

which has now been sorted, and a later report by @satello of a similar instance:

which has also no be fixed in the stable channel via the pull request indicated in that thread.

Hope that helps to explain this ‘operation not permitted’ thing.

1 Like

Hey @phillxnet, thanks! I was doing a lot of sleuthing around after I posted and I stumbled on a thread somewhere that also mentioned something about checking the folder attributes and sure enough when I looked at the lsattr of the folders in question they had the ‘i’ flag set. After removing the flag I still had issue removing them because they for whatever reason were currently in use. I rebooted which released whatever was still holding on to them and I could successfully remove them.

Further digging around showed a similar issue had also occurred with some shares/NFS exports that I had deleted in the GUI but were still showing up in the filesystem (albeit empty). Those too were fixed by simply changing the immutable flag and doing a rm -r.

I’ll mark as solved and again, thanks for the quick response!!!

1 Like