Spotty NFS write performance to RAID5

Troubleshooting
1

Hi,

I get very spotty write performance over NFS to a RAID5 (4x 6TB wd red on an M1115 with IT firmware, noatime set, NFS with async) share. Local write is perfectly fine:

[root@rockstor ~]# dd if=/dev/zero of=/mnt2/data/test.data bs=4K count=9024000 9024000+0 records in 9024000+0 records out 36962304000 bytes (37 GB) copied, 108.87 s, 340 MB/s

But over NFS (transferring with rsync) the files copy with ~99MB/s and randomly drop to kB/s for some seconds to pick up again to up to 99MB/s.

iostat -x 2 shows peaks to 100% util on all drives with a drop in transfer speed. The utilization drops to nearly 0% and the speed picks up again.

It looks like the file gets written to the cache and when the cache is full it gets written to disk and there is no continuous write process.

Here is a small video of the whole process: https://www.dropbox.com/s/3nh6mk5bn0f3mua/rokstor_nfs_speed_test.mp4?dl=0 (download for better quality)

Cheers and thanks for reading!

2

What’s your network speed? Any interface errors?

3

1Gbit/s and both the hypervisor and the server I tested from are on the same switch. No interface errors.

4

I bet your client is caching. Do you have nfs mounted with sync?

5

I haven’t mounted the nfs share with a special option but the server is running the share async.

6

I’m having a very similar issue to what you are describing. Have you found any solution??