BTRFS will only compress if it can save a descent amount of space doing so. This is for performance.
There is another mount option that will force compression but it’s not available from rockstor webui.
I guess that will be the main differance from your FreeNAS setup.
You could try and play with this on the extra mount options. “compress-force=zlib”
Enable compression. Starting with kernel 2.6.38 you can choose the algorithm for compression:
compress=zlib - Better compression ratio. It’s the default and safe for olders kernels.
compress=lzo - Faster compression.
compress=no - Disables compression (starting with kernel 3.6).
compress-force= - Enable compression even for files that don’t compress well, like videos and dd images of disks. The options compress-force=zlib and compress-force=lzo works for kernels >2.6.38.
Note that old (before 2012) btrfs-progs versions will probably fail some operations (e.g. fsck) on filesystems with LZO compression.