It sounds like a nasty situation.
Unfortunately, with no local shell, ssh nor gui, there’s no debug information available.
Be warned: While I try to provide helpful information and insights, I am just a user like you - with a decent linux knowledgebase. My answers may not be the best advice for a given situation, and may break your stuff, burn down your house or kill your dog. If this is the case, they do not reflect on Rockstor or its developers.
I’m fairly certain that it’s not supposed to become non-responsive during RAID level changes (at least, not by btrfs design), so I personally would suggest a hard reboot, then (from shell):
btrfs balance status /mnt2
btrfs fi show
The first will tell us the balance state from the raid conversion. If it had not completed, it should try to continue.
If it reports incomplete and doesn’t continue:
btrfs balance resume /mnt2
The second will tell us how balanced the disks actually are. As your disks are all 10Tb, the usage should theoretically be very close to even.
I myself had some issues with system lockups during a balance when adding disks and re-balancing, so I feel your pain. Mine stalled on balance 3 times requiring a hard reboot each time.
If your disks continue balancing, ensure that you have a monitor and keyboard attached at all times during the balance.
You can monitor the balance status live with:
watch -n 10 btrfs balance status /mnt2
Look for the line like:
450 out of about 750 chunks balanced (451 considered), 40% left
If the values dont change for more than (at a guess, I don’t have 10tb disks!) 15 minutes, it’s probably stalled and requiring a hard reboot.
Also worth watching ps, top and syslog in other tty’s. I was seeing btrfs processes stuck in uninterriptable sleep, increasing io wait as a result when I experienced balacing issues.
Hope this helps.