I have been running rockstor for over a year now. I have this problem frequently as well. I make heavy use of scheduled snapshots on a 19TB btrfs formatted raw block device attached to a Rockstor VM.
When this happens, the btrfs-transacti consumes all available CPU on the VM it runs in. I am not able to access shares via smb during this time. If I leave it alone, it eventually finishes and everything comes back.
When I look at IOPS on the disk set holding the block device, I see minimal IO usage.
Are there any parameters to be tweaked to speed up this process? What data is the cpu crunching on if it is not generating disk IO? Can that data be optimized?