I get very spotty write performance over NFS to a RAID5 (4x 6TB wd red on an M1115 with IT firmware, noatime set, NFS with async) share. Local write is perfectly fine:
[root@rockstor ~]# dd if=/dev/zero of=/mnt2/data/test.data bs=4K count=9024000 9024000+0 records in 9024000+0 records out 36962304000 bytes (37 GB) copied, 108.87 s, 340 MB/s
But over NFS (transferring with rsync) the files copy with ~99MB/s and randomly drop to kB/s for some seconds to pick up again to up to 99MB/s.
iostat -x 2 shows peaks to 100% util on all drives with a drop in transfer speed. The utilization drops to nearly 0% and the speed picks up again.
It looks like the file gets written to the cache and when the cache is full it gets written to disk and there is no continuous write process.
Here is a small video of the whole process: https://www.dropbox.com/s/3nh6mk5bn0f3mua/rokstor_nfs_speed_test.mp4?dl=0 (download for better quality)
Cheers and thanks for reading!