Repeated qgroup error in Rockstor Log

GeoffA · December 8, 2020, 8:28am

Although I’m not having any issues with my Rockstor 4, I decided to play admin and have a quick nose in the logs this morning while waiting for my toaster to do its thing.
I came across this repeated ‘error’:

[08/Dec/2020 08:01:44] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/ROOT/rockon/btrfs/subvolumes/cc4846ad408f378b2311d854fca88fcfc6b05547c1b20cca2515cf7fe00ca230']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']

This error is logged repeatedly, every few seconds.

Do I need to be concerned about this? Should I spend time Troubleshooting, or not worry?

EDIT: So, looking through the log it is basically reporting this ‘error’ for everything under the ‘rockon’ share on the ‘ROOT’ pool, for which quotas are disabled.
It doesn’t report the same for the ‘home’ share on the same pool though.
Rockon service is disabled after I experimented with a few of them.

Geoff

Flox · December 8, 2020, 12:02pm

Hello again, @GeoffA ,
I unfortunately do not have time to go into details at the moment, but wanted to briefly answer your questions below:

No need to worry as this is expected behavior when quotas are disabled. You see this repeatedly in the logs as this share scan procedure is part of the process to get share size (I believe), which is run at regular intervals.

Sorry for the very brief and rushed answer, but know that there’s nothing to worry about here.

GeoffA · December 8, 2020, 12:09pm

Thanks @Flox I appreciate your response. I shall not worry further about it

That’s my slight programming OCD from the old days I guess: Errors in my logs? NEVER!

Cheers

Geoff

Flox · December 8, 2020, 12:14pm

Yeah I understand and agree… There might be room for improvement here to limit this kind of things. Maybe we shouldn’t throw an error like that when quotas can’t be scanned when quotas are disabled. As long as we still catch and report genuine quotas scan errors of/when they happen, reducing this kind of noise can only help.

phillxnet · December 15, 2020, 12:46pm

@GeoffA and @Flox Re:

Yes me too. This one is down to me I’m afraid. Way back Rockstor considered quotas disabled as a show stopper. In the interim we have become compliant/functional with this state but haven’t quite moved to not logging wildly and it’s probably time we moved this log entry to debug only one. It can still be useful but yet, spamming logs with errors is not a good show really. Sorry folks, all in good time and I’m still not entirely happy with our quota disabled capabilities. Mainly in how we re-create our quota groups when quotas are re-enabled. So such messages still have a place but yes, probably due to be moved to debug really.
@GeoffA Thanks for yet another prompt. If you fancy making a GitHub issue for this it may well catch someones eye and given it’s a cosmetic nice-to-have, bar excessive log writes, it could be a nice beginner patch, i.e. move such messages from info to debug.

I’d also really like a Web-UI way to initiate debug message capability but that would of course be another issue :).

Exactly. And missing them as they are hidden in debug mode makes for slower discover. Hence the desire to allow debug log enablement from Web-UI mention. So many things but all in good time hopefully.

GeoffA · December 17, 2020, 7:03am

Done:

github.com/rockstor/rockstor-core

Repeated Log Errors for shares with disabled quotas

opened 07:02AM - 17 Dec 20 UTC

closed 03:34PM - 01 Feb 21 UTC

GeoffAd

Probably just a bit of LOG housekeeping really, but currently shares with quotas… disabled are being repeatedly logged as Errors. Example extract from Rockstor Log: [16/Dec/2020 18:27:33] ERROR [system.osi:174] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/ROOT/home']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", ''] Logs are repeatedly filled with this 'ERROR', which isn't an error as ROOT pool share quotas are disabled by default, and other non-ROOT share quotas might also be disabled by user. This information could be of use at times when tracking down other errors, so could these items be managed separately as DEBUG to avoid unnecessary filling of logs?

GeoffA · December 17, 2020, 11:54am

Although not quite sure what made me think ROOT quotas are disabled by default

phillxnet · December 18, 2020, 9:52am

Yes, all I could think was the docker-ce thing. Also the default snapper system depends on quotas to do it’s thing and we run with an only slightly modified setting of this boot to snapshot setup. See:

github.com

rockstor/rockstor-installer/blob/master/config.sh#L97-L111


#=====================================
# Configure snapper
#-------------------------------------
if [ "$kiwi_btrfs_root_is_snapshot" = 'true' ]; then
        echo "creating initial snapper config ..."
        # we can't call snapper here as the .snapshots subvolume
        # already exists and snapper create-config doens't like
        # that.
        cp /etc/snapper/config-templates/default /etc/snapper/configs/root
        # Change configuration to match SLES12-SP1 values
        sed -i -e '/^TIMELINE_CREATE=/s/yes/no/' /etc/snapper/configs/root
        sed -i -e '/^NUMBER_LIMIT=/s/50/10/'     /etc/snapper/configs/root


        baseUpdateSysConfig /etc/sysconfig/snapper SNAPPER_CONFIGS root
fi

for how we tweak the snapper config to hopefully require less space than by default. OpenSUSE had quite some comeback on root pools running out of space, hence dropping the defaults just a tad. We should keep an eye on this however. And there may well be ramifications of no quotas enabled on the system pool as a result. Not that up on snapper behaviour re quotas disabled so hopefully folks will chip in as stuff is discovered/uncovered.

Cheers

phillxnet · January 30, 2021, 7:03pm

@GeoffA

I’ve now prepared a pull request to address the excessive quotas disabled logging discussed in this thread and the GitHub issue you opened. Post review we should be able to pop this one in as it simply removed the explicit logging of the original btrfs ‘error’ messages which it turns out we already mostly catch anyway. At least that is my impression having just looked at it.

github.com/rockstor/rockstor-core

repeated log errors on disabled quotas (#2236)

rockstor:master ← phillxnet:2236_repeated_log_errors_on_disabled_quotas

opened 06:49PM - 30 Jan 21 UTC

phillxnet

+8 -8

Remove explicit logging of key 'btrfs qgroup show' commands to avoid excessive l…og spamming. As we have now supported the pool quota disabled state mostly successfully for some time we can remove the initial default of always logging these commands which are reported as errors by the underlying btrfs commands when quotas are disabled on the parent pool. And hence are at odds with their supported state within the Rockstor Web-UI. In most common cases we catch and log as info in addition to the prior 'raw' command log. Prior explicit log parameters have been kept as explicit to ease reversing this change. Includes minor comment typo. Fixes #2236 Ready for review. ## Testing ### Pre pr With a ROOT pool, quota disabled, a Shares refresh page produces the following log entry: > [30/Jan/2021 18:33:29] ERROR [system.osi:196] non-zero code(1) returned by command: ['/usr/sbin/btrfs', 'qgroup', 'show', '/mnt2/ROOT/home']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", ''] ### Post pr There is no ERROR message produced. N.B. there are still error states that we wish to surface and they should still be logged or surfaced. But for the time being this patch reduces our mostly redundant a and potentially misleading 'ERROR' log spamming for known common btrfs quota disabled states.

Cheers.

GeoffA · January 30, 2021, 7:20pm

@phillxnet sounds good to me. I realise it was just a minor issue, but good to get it nailed nevertheless. Nice one
Geoff