SOLVED: BTRFS Subvolume create/delete fail

Support
1

Hi Rockstor Community,

I had previously installed a number of Rockons that I didn’t require.
These have now been removed, however the config and data shares seem to be unremovable.
I am also now unable to add new shares.

The following is what I receive from the UI:

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 301, in delete
    force=force)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 578, in remove_share
    run_command(delete_cmd, log=True)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 121, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /sbin/btrfs subvolume delete /mnt2/tempraid/cfgLazyLibrarian. rc = 1. stdout = ["Delete subvolume (no-commit): '/mnt2/tempraid/cfgLazyLibrarian'", '']. stderr = ["ERROR: cannot delete '/mnt2/tempraid/cfgLazyLibrarian': Operation not permitted", '']

I attempted a share create as well, with similar results.

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 171, in post
    add_share(pool, sname, pqid)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 347, in add_share
    return run_command(sub_vol_cmd)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 121, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /sbin/btrfs subvolume create -i 2015/48 /mnt2/tempraid/gronk. rc = 1. stdout = ["Create subvolume '/mnt2/tempraid/gronk'", '']. stderr = ['ERROR: cannot create subvolume: Operation not permitted', '']

At this point, I assumed that it might be a permissions issue with the Rockstor UI user I had configured, so I tried running the subvolume delete manually as root:

[ root@rockout (pass 0s) ~ ]# btrfs subvolume delete /mnt2/tempraid/cfgLazyLibrarian
Delete subvolume (no-commit): '/mnt2/tempraid/cfgLazyLibrarian'
ERROR: cannot delete '/mnt2/tempraid/cfgLazyLibrarian': Operation not permitted

As you can see, this was no more successful.

“btrfs subvol list /mnt2/tempraid/cfgLazyLibrarian” shows a number of subvolumes that are not within this subvolume (IE: other top level shares, cfgSonarr, cfgCouchPotato, etc.)

Can anybody shed some light on this?

2

An update on this, it appears I can create and remove subvolumes under the existing subvolumes without error:

# btrfs subvol create /mnt2/tempraid/cfgLazyLibrarian/gronk
Create subvolume '/mnt2/tempraid/cfgZoneMinder/gronk'

and

# btrfs subvol delete /mnt2/tempraid/cfgLazyLibrarian/gronk
Delete subvolume (no-commit): '/mnt2/tempraid/cfgLazyLibrarian/gronk'

Still no idea why I cannot create or delete subvols of /mnt2/tempraid though…

3

Problem has been resolved, though I’m unsure why it had happened.

Immutable flag was set on /mnt2/tempraid, meaning that the contents of /mnt2/tempraid was unmodifiable (though decendants could be altered).
I know I didn’t set the flag, and according to BTRFS IRC, scrubs do not set the flag in the event of error.

Problem was resolved with

chattr -i /mnt2/tempraid

Should have thought of it before - in fact, I could swear I checked it, but apparently I did not.

1 Like
4

Thanks for your update, i had exactly the same issue, and i encourage people to check if they are getting the operation not permitted error.

lsattr /mnt2/

and check for the “i” flag.

I should point out that i have not intentionally set this flag and have no idea why it was set.

2 Likes
5

@Haioken and @Rene_Castberg A somewhat belated thanks for your input on this one.

Funnily enough I just ran into this one (or it’s kin) myself while working on fixing replication, which internally deletes a receiver side share just prior to supplanting it with a more recent btrfs received snapshot. Anyway same thing pretty much (random immutable flag appeared, but on a share (subvol)) and ended up at this thread when diagnosing it. Consequently I have created the following issue:

https://github.com/rockstor/rockstor-core/issues/1882

and have prepared the following pull request:

https://github.com/rockstor/rockstor-core/pull/1883

which is awaiting review.

Hope that helps and apologies for the slow response on this one.

6

@Haioken and @Rene_Castberg As per stable release version 3.9.2-12:

https://github.com/rockstor/rockstor-core/issues/1882

has now been closed as the earlier sighted associated pull request has now been merged and release.

I appreciate that this thread also concerns the immutable flag’s appearance on the pool (btrfs volume), and it’s ramifications, but at least we have a fix for the share (btrfs subvol) in place now.

Thanks again for posting your findings; most helpful.

7

@phillxnet

Thanks for the update.

It occurs to me that perhaps there should be some UI element for confirmation before removing the immutable flag - in the event that someone has deliberately set it in order to prevent subvol removal.

Also, any idea when testing releases will be available again in some form or other?

8

@Haioken Hello again:

Yes, agreed. Though in your’s, @Rene_Castberg, and my cases the flag appeared without having been manually set and this lead to breakage in base functions including replication (as noted in the issue and pull request text) so I favoured removing it without question for now at least. Though I did limit this ‘heavy handed approach’ to just subvol delete (and then not recursively) as that was the most pressing ‘show stopper’. So going forward I definitely intend to more correctly observe the immutable flag but that’s a far larger issue and for the time being it was required that we ‘fix’ by brute force for the time being, especially given the apparent accidental and sporadic nature of it’s appearance across 3 different reports, or 4 if we include @oberlixgmbh recent report (again on the pool but presumably related).

Issue text on this:

“Under as yet undetermined circumstances a share (btrfs subvol) can have it’s immutable flag set. The consequence of this is a failure to be able to delete the affected share. Given that Rockstor currently has no immutable flag observance and that the cause of this flag having been set is unknown it seems prudent for the time being to be robust in this scenario and simply remove the flag prior to a share’s delete (which is itself accompanied by a GUI warning of relevant data loss). This issue was also observed to randomly break replication whilst working on #1853 as the receiver periodically removes and supplants a share with a received snapshot. As and when the true cause of this flag’s having been set is identified we can more correctly observe it’s use project wide (along with appropriate user messaging), but currently our base ability to manage shares and have more robust replication is judged as taking priority. This element is a key factor in breaking out this change from #1853 as it is important that it be revisited in due cause in it’s own right.”

And pull request text on the same:

“As an immutable flag can break share delete and given we already provide a UI warning, this preemption and circumvention of an existing immutable flag is deemed appropriate. Especially in the context of potentially breaking replication: during snap-to-share supplanting. It is however advised to re-address this approach once the sporadic nature of immutable flags appearing without manual intervention is resolved.”

Bolding in above quotes added for emphasis relevant to this discussion.

Essentially if it is appearing sporadically without intent then it is currently doing more harm than good as it’s blocking key functions (only subvol delete in the addressed case) which is itself enacted with intent (UI interaction along with appropriate data loss warning) or background snap-to-share rotation automation on the replication mechanism front. Hence we are favouring the intentional over the unintentional from a UI perspective: at least until we more properly observe this flag and have the means to set / remove it from within the UI. Our focus has to be on making what features we currently support (within the system and fs) take priority over those that we don’t currently: especially if they can appear without intent.

But yes, in the future we can surface this and other flags within the UI and support/observe them properly in such cases as the subvol delete and replication back-end mechanism. Bit of a tight rope really, all in all.

Thanks for you feedback and input on this one and I hope my explanation as to my actions thus far on this issue is helpful.

I’ll open an issue to at least surface these flags within the UI soon. That work could then be extended later, via additional issues, to altering those same flags.

9

Hey I just ran into the subvolume being marked as immutable issue though not in the context of subvolume deletion and landed on this thread.

My share is used over Samba and users began reporting a “Permission Denied” error when creating a new folder within the share. I proceeded to check the UI and for some reason, my share which I had created with myuser:users ownership had switched in the UI to root:root. I tried updating the permissions and received a similar error message.

Turns out that for some reason the immutable flag had been set on the share mount directory. Once this was fixed, I was able to update the share ownership in the UI as well.

Just thought I should report that this issue (immutable flag being set randomly) can impact normal usage of live shares since new file and directory creation at the top level of the share is no longer possible.

1 Like
10

@parth Welcome to the Rockstor community.

Thanks for the additional input on this one. Could you state if this was experienced with our now legacy CentOS based v3 version or with our current stable v4.1.0-0 version included in our new installer downloadable here?

https://rockstor.com/dls.html

As we did make some fixes in this area over time, although I think they were towards the end of the v3 stable release actually, which ended at v3.9.2-57.

Thanks again for highlighting this additional ‘surfacing’ of the immutable flag issue we saw for a while back in the v3 days at least.

Cheers.

1 Like
11

This happened in the new Rockstor v4.1.0 version. Although, I did recently migrate from Rockstor v3.9 to 4.1.0 and imported this filesystem and then extended it by adding 2 more disks. I did face a few issues during the migration but thanks to this forum was able to resolve them.

One of the problems was that on import the share had the same root:root ownership even though the user and group had been created in the new setup as well before importing the disks. I was able to change the ownership using the UI in that instance. Then came this immutable flag issue and having never dealt with extended attributes and ACLs before, it kind of threw me a bit.

Don’t know if this information helps.

2 Likes
12

@parth Re:

Yes that is a known issue actually and it has yet to be mentioned in our import docs. I’ve created the following issue to address this:

Thanks for the detailed feedback and prompt on the import limitation/bug/feature.

2 Likes