Slow NFS Write - RAID5

Trying out Rockstor to replace my existing DriveBender solution- built a dedicated box with M1015 card flashed to IT for Rockstor.

Drives are 3x 2TB Seagate Low Power drives. Everything seems to be setup fine but my writes are maxing out at 16MB/s - Reads are great at 95MB/s.

What am I missing / doing wrong?

you might have set the nfs export to be synchronous, async should be much faster.

the option should be within the nfs export

NFS export it set to async. I also tried Samba protocol and got same speeds.

I have compression: off & Extra Mount Options: noatime set.

EDIT: Added the following:

The USB it’s installed on is not a 3.0 USB and the install took HOURS. Could it possibly be related to something with a BIOS setting or USB legacy issue? From what I read the USB should be 99% read only for the OS and shouldn’t impact share write/read performance, but I could be wrong?

Memory Usage (8GB total) is also - Used: 9%, Cached: 69%, Free: 22% – It was Free: 0% as Cached was taking of all that used wasn’t.

the os install should not impact the speeds at all, strange

can you try the following:

dd if=/dev/zero of=/mnt2/YOUR_SHARENAME/test.data bs=4K count=1024000

it will create a file of 4GB consisting of zeros on the share, the command blocks till its completed, so dont worry if nothing happens. after it completed the overall speed is displayed. note that this only works if compression is disabled, but this should not be a problem for you if your share is mounted with compression off, too.

this is to check if its something with the networking or the share itself as the command writes directly to the share/disks.

1 Like

hmmm, interesting results…

[root@rockstore ~]# dd if=/dev/zero of=/mnt2/lp-seagate/test.data bs=4K count=1024000
1024000+0 records in
1024000+0 records out
4194304000 bytes (4.2 GB) copied, 14.4826 s, 290 MB/s
[root@rockstore ~]#

Indeed, so while it happens in smb too, i assume its nothing related to those daemons. Can you try client with some different Server? (Or assume its not the client when it worked before with another Server)
Its unlikely but maybe something with the network cards drivers? Did you have another system up and running without the Problem with the same Hardware?

Thinking it may be related to my NIC’s. My mobo is a AM1B-M which has a Realtek RTL8111GR onBoard NIC and I have a PCI card as well - an Airlink AG32PCI which has either a RTL8110 or RTL8169, cant confirm exactly…

I did a fresh install last night on a new USB 3.0 stick on a USB 3 port, had to turn USB 3.0 support in BIOS from ‘Auto’ to "Enabled’ to successfully boot and it took about 30mins whereas the previous install took hours… Took me forever to get network connectivity to work after the install unlike prev. both NIC’s obtained IP’s. I was able to get one card with an IP but unable to connect via web-ui but I can successfully ping both machines. Looking at ordering an Intel NIC this AM.

Do you know of any problems with the Realtek RTL811* Series NIC’s & CentOS? I’m an Ubuntu user mainly and not used to this yum stuff lol

hm the mobo seems old enough for correct drivers, can you test the raw network perf with iperf?

a gui windows/linux client is available here.

to install iperf on rockstor enter in the cli:

yum install epel-release
yum install iperf

answer all questions with y

on the server enter:

iperf -s -P 0 -i 1 -p 5001 -f M

which starts an iperf server process which listenes on port 5001

switch to the client: if its linux start the jperf.sh, if its windows the jperf.bat

set the mode to client, enter the rockstor ip and switch the ouput format to Mbytes

it should look like this:

press “run IPerf!” and check the reported Bandwith, which should look like this if on Gbit:

2 Likes

So I reinstalled last night (I think my previous install didn’t install properly, might have had some issues with method used to write the ISO to the USB) network issues have all been resolved, still having same issue with write speed however.

Tried to install epel-release & iperf but I’m getting a URL cant be resolved, trying another mirror error, looks like I might get success when I cancel with CTRL+C but it ends up failing. Looked into adding another repository but I honestly have no idea what I’m doing in CentOS, it’s so much different than Debian.

BTW- I GREATLY appreciate all your assistance and guidance @felixbrucker , I’m dead set on getting this to work and documenting everything I try. I’ve decided that I’ll be moving to Rockstor for my primary homelab NAS solution, It’s everything I’m looking for with btrfs I just need to figure out what I’m doing wrong to fix this 15MB/s write speed issue lol.

Everything is connected via GB LAN, the origin content drive is a 3TB Seagate 72k drive and has no issues writing to a local SSD. I do have the Rockstor box connected via a Airlink 5-Port 10/100/1000Mbps Green Switch temporarily as I rebuild my homelab. My laptop Win10 doesn’t seem to have the option in windows features to turn on NFS and haven’t tried any other machine but I didn’t have any issues writing at around 90MB/s when I was using DriveBender.

@ScottyEdmonds Hello. The epel-release package is just a CentOS package that adds an additional rpm/yum repository (the epel one); required in this case as iperf is not included in the default CentOS repositories.
You can configure repos using the files in /etc/yum.repos.d . So in Ubuntu terms its like a really common ppa that can be added by a build in package.
I would say you may have network issues if the rather excellent instructions from @felixbrucker for installing iperf are returning “URL cant be resolved” ie it can’t find / resolve the network name it has on record for epel or whatever else it needs.
Hope that helps.

Also I have often seen asynchronous speeds on network cards, ie sometimes they work great in one direction and less well in the other.

as @phillxnet pointed out your network configuration seems a bit off, is your dhcp/static config working?

check if your dns and internet routing is working by pinging google.com which first resolves google.com to an ip and then pings it, with your current configuration this should fail at some point, try to get to ping google (or any other reliable internet host by hostname) and then retry the yum lines.

the most likely problem is the dns here (you can check by pinging some address instead, like 8.8.8.8), check if you specified one when using static ip addresses.

@phillxnet async speeds for network cards, strange, havent seen such a thing till now, but you know, nothing is impossible :smiley:

You guys were right, had to set static IP’s to connect to the internet but wasn’t having any issues with LAN.

Was able to run iperf -

Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.101 port 5001 connected with 192.168.1.201 port 60584
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  10.1 MBytes  10.1 MBytes/sec
[  4]  1.0- 2.0 sec  10.3 MBytes  10.3 MBytes/sec
[  4]  2.0- 3.0 sec  10.5 MBytes  10.5 MBytes/sec
[  4]  3.0- 4.0 sec  10.3 MBytes  10.3 MBytes/sec
[  4]  4.0- 5.0 sec  10.6 MBytes  10.6 MBytes/sec
[  4]  5.0- 6.0 sec  8.33 MBytes  8.33 MBytes/sec
[  4]  6.0- 7.0 sec  7.66 MBytes  7.66 MBytes/sec
[  4]  7.0- 8.0 sec  8.75 MBytes  8.75 MBytes/sec
[  4]  8.0- 9.0 sec  7.77 MBytes  7.77 MBytes/sec
[  4]  9.0-10.0 sec  6.11 MBytes  6.11 MBytes/sec
[  4]  0.0-10.1 sec  91.4 MBytes  9.05 MBytes/sec

the connection between the client and the server is gbit?

can you post the output of

ifconfig -a

and

ethtool YOUR-NETWORK-INTERFACE-NAME

eg. ethtool enp4s0

for simplicity sake, I’ve unpluged one of the network connections.

[root@rockstor ~]# ifconfig -a
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.183  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::d250:99ff:fe61:e758  prefixlen 64  scopeid 0x20<link>
        ether d0:50:99:61:e7:58  txqueuelen 1000  (Ethernet)
        RX packets 98633  bytes 145800375 (139.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47887  bytes 2615408 (2.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether c4:6e:1f:02:08:a2  txqueuelen 1000  (Ethernet)
        RX packets 1283  bytes 93826 (91.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183  bytes 24119 (23.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 4595  bytes 339844 (331.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4595  bytes 339844 (331.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@rockstor ~]#

[root@rockstor ~]# ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
        Link detected: yes
[root@rockstor ~]#

seems normal to me, you were definitely able to push gbit speed with the same client before with some different server os?

if so it seems like the nic driver in centos has a bug somewhere which prevents it from getting gbit speeds

you can try to install some ubuntu server on some usb stick and try to reproduce the iperf test from there, but regarding nic driver issues and how to fix them my knowledge is limited :smiley:

maybe @phillxnet knows more about such things

Going to pull the additional NIC, and try installing Rockstor via Proxmox and use PCI Passthrough to the VM.

So I decided to go back and try an earlier version as I usually have success with that. And of course, getting full expected write speeds. Now let’s just see if an upgrade ruins my success.

Oh so a previous Version worked fine?

Good to hear, if an upgrade ruins that it should be easy to narrow down

@felixbrucker yes the upgrade to the 4.02 kernel killed it. Installed 3.8-0 and success until it asked me to upgrade the kernel.

EDIT: After some testing, I upgraded to latest version from 3.8-0 and rebooted and all seems to be working as it should!!! CELEBRATION TIME

EDIT: Actually no, I’m getting this which will cause the issue - You are running an unsupported kernel(4.0.2-1.el7.elrepo.x86_64). Some features may not work properly. Please reboot and the system will automatically boot using the supported kernel(4.1.0-1.el7.elrepo.x86_64) - 4.1.0-1 will kill my NIC’s download.

Sorry for joining the party late, but is there a simple script/list of steps that demonstrates the performance difference between 4.0.2 and 4.1.0 kernels? I am planning to make 4.2 default soon and would like to run the test on the kernel.

@ScottyEdmonds, btw, you can ignore the unsupported kernel warning as there is not much difference between 4.0.2 and 4.1, but I do hope you can use the 4.2 kernel coming soon.