Many thanks! The problem is solved!
What I did was relatively simple:
I disabled the btrfs quotas after manually mounting the pool via the console.
After restarting Rockstor, I was able to successfully mount the pool via the Rockstor GUI.
So there seems to be a problem with btrfs or Rockstor to mount a pool including subvolumes, if the pool has the quotas enabled before. (bad quotas function implementation)
PS: the virtual system running the Rockstor VM has 8 vcores and 16GB RAM allocated, so there is no lack of performance.
The install ISO was created with Kiwi-NG and an OpenSuse LEap 15 installation.