This 'ERROR: unable to assign quota group: File exists', is very likely the same, now fixed issue reported again recently by forum member @Stevek in the following forum thread:
Our latest 5.0.13-0 testing release (in Release Candidate phase), has the fix:
The caveat here, and why it has taken a while to identify at least one cause of this issue, is that it pertains to imported Pool. Not newly created pools, or pools that have had quotas disabled and then re-enabled - hence the nicely reported work around here by @MyOwn.
Details of the fix, which relate to our still developing support for btrfs quotas are available in the following GitHub pull request where there are examples of what happens on imported pools where quotas were, and remain, active; that can lead to this now fixed bug:
There is also an example of the same reproducer Pool being reset to no longer provoke this bug due to a quote disable/re-enable cycle.