100 cpu core utilization

Troubleshooting
1

hi.

I have faced with strange activity. 100% cpu load on one core. Plex rock on not responding. Docker stop has no effect.

wp_ss_20161005_0001.png2560×1440 411 KB

:frowning: what is it?

2

troubles.PNG964×174 8.47 KB

another process.

I’ve rebooted my server but it not rebooted within 5 minutes. I hope my data is ok :frowning:

3

after reboot

troubles2.PNG809×198 8.82 KB

4

I’ve seen BTRFS or btrfs-transaction sit there at 100% for a while before, it’s usually when doing something like deleting a lot of data, running a balance or deleting snapshots.

Essentially anything that requires BTRFS to perform cleanup or re-write a load of metadata I would guess.

5

how often and how long it will be running?
System services are inaccessible. :frowning:

I have snapshot schedule for one share (12Gb) every 5 minutes.

6

Does that snapshot discard the old ones frequently and how much data changes?

It could well be that but I’m just guessing here.

7

If btrfs-transacti process is at 100%, type:
btrfs quota rescan -s <PATH_TO_POOL>

If a quota rescan is running and it’s always on the same key, this is the culprit. You need to disable quotas.
btrfs quota disable <PATH_TO_POOL>
You may need to reboot before this command will work.

If this fixes, you’ll then have another problem: the service rockstor-bootstrap.service won’t start without quotas enabled, which means dependent services like smb won’t start:
btrfs quota enable <PATH_TO_POOL>
Start services:
systemctl start rockstor-bootstrap.service
systemctl start smb
systemctl start <ANY_OTHER_SERVICES_YOU'RE_MISSING>

Disable quotas again:
btrfs quota disable <PATH_TO_POOL>

Can automate this in a startup shell script.

1 Like
8 
[root@server ~]# btrfs quota rescan -s /mnt2/documents/
rescan operation running (current key 1031017570305)
[root@server ~]# btrfs quota rescan -s /mnt2/documents/
rescan operation running (current key 1032447197185)
[root@server ~]# btrfs quota rescan -s /mnt2/documents/
rescan operation running (current key 1033334620161)
[root@server ~]# btrfs quota rescan -s /mnt2/documents/
rescan operation running (current key 1033772036097)
[root@server ~]# btrfs quota rescan -s /mnt2/documents/
rescan operation running (current key 1034042638337)

Key is different. Every 5 minutes (according to snapshot schedule) CPU utilization is 100%, this issue affects other services.
SMB share is not responsive.

9

I’d just disable quotas anyway (unless you need them). Btrfs quotas are still performance hogs: “When the quotas are turned on, they affect all extent processing, taking a performance hit. It is not recommended to turn on qgroups unless the user intends to actually use them”

1 Like
10

I’ve disabled quotas. Problem has gone.

Thanks!

2 Likes