I read the regular explanation about how filling the memory with cached content is all good and normal however it doesnt play nice with system configured to use NUMA Nodes
Here is my particular use case but this should be repeatable without any hypervisor and just rockstor bare metal install
Rockstor is running in a VM under Proxmox, with VM configured to be NUMA aware and the server is configured to be in NUMA mode in the BIOS. All CPUs (2 sockets, 4 core/8 threads each) are given to the VM along with 10GB of ram. Plex is installed as rock-on and speeds up the testing because while working, cached memory starts to raise. numastat can track numa_miss and numa_foreign which is basically the misses for the RAM when data for a process on current node (cpu) appears to be in the memory of the other node - this is bad because retreiving this data is significantly slower. So witch caching increased it seems to first fill the memory of the first node - when there is like less than 100MB of free memory left on the node, it continues to cache on the memory of the other node but also makes the first node to start register numa_foreign and numa_miss is registered on the second node. Cache keep increasing untill it fills all available memory and then numa_foreign starts building for the second node together with numa_miss for the first node.
As I said this is not good. I want to reduce or disable caching. I wonder if someone knows how?