When used as virtual machine storage, Rockstor will store a small number of large virtual disk files that are frequently accessed. In order to prevent a massive slowdown in VM performance, BTRFS’s copy-on-write feature should be disabled by enabling the nodatacow attribute on the folder that stores the virtual machines. One downside of enabling nodatacow is it disables checksumming. Has anyone found this has impacted the resilience of their VMs? Apparently snapshots will continue to function, however “Reflink copies do not work on FILES that have NODATACOW on”. What are the implications of reflink not working? Will replication work without this (when this is fixed)?
i personally have not used btrfs for storing virtual machines but use zfs which should behave the same in regards of small writes and the fragmentation issue. i run my machines (20-30) off a 6 disk raid10 with zfs over the network with iscsi. Generally anything that uses async transfers over the network contributes to the ram “cache” which gets written to disk every X seconds (btrfs default is 5 afaik). This lowers the fragmentation to the amount of blocks written in such a time frame instead of every block (sync writes). I have not noticed any substantial performance drop since a year ago when i set it up. if you want to use kvm directly on rockstor (no network) things might be different for the “caching” in ram, but i would expect it to also cache in ram. Additionally zfs lacks a nice feature btrfs has: defragmentation, so if you notice a performance drop after a year or so you could just defrag it. ofc this whole thing depends on your expected load, running a heavy database server might clash this very fast (no experience though).
when checksumming is disabled your backing storage behaves just like any hdd in regards of bit flips. It wont crash your OS, but if uncorrectable within the OS one might lose a small file (eg checkdisk cant repair a file), personally i have only seen such a behaviour on a data storage raid with many disks but not on any VM i ran prior to zfs
I’m using Rockstor as storage for a separate host (via NFS). I guess this is all a bit cutting edge and I’ll just have to suck it and see to an extent (and have good backups!).
One immediately obvious improvement that could be made though. Add an “Advanced options” field to the Share creation\edit screen. Currently you can only add the nodatacow option at pool level (“Extra mount options”). Pools can’t easily be resized like a share, so people will end up with an entire pool optimised to run virtual machine storage, when they might like to use it for general storage too. Allowing nodatacow at the share level gives greater flexibility because it allows pools to be split between general and virtual machine storage. I’ve enabled nodatacow for a share by command line ("chattr -R +C ") but better to make this possible in the UI.
2 Likes