Disk performance test

Performance
1

Hello guys!

Is there a tool to benchmark our rockstor setup? Like disk reading/writing tests, general system speed according to hardware, etc.?

I feel like my RAID is not performing as good as before and I would like to be able to investigate.

More precisely it seems it has having hard times doing file I/O operations. I have an application which needs to rebuild an XML (2 megs), every time it refreshes a certain gui page and it is taking more than 30 seconds lately, while it used to take less than 5.

Thanks a lot!!

2

@sanviler Hello again,

Any benchmarking tool available for linux, ideally packages by openSUSE also, should work just fine. We are essentially a minor re-config of openSUSE Leap or Tumbleweed depending on which installer you used and what you have done zypper dup wise there-after.

Our base filesystem, btrfs, being Copy-On-Write, does have additional overheads to manage it’s ‘magic’. So there can be performance (time related) drawbacks. If performance is key, avoid the parity raids and stick with the raid1 or raid 10 variants. Or use the more exotic mixes now supported, i.e. raid5-1c3 or raid6-1c3. But the btrfs parity raid levels of 5 & 6 are known to have performance issues at times. Getting better all the time but still bested by the raid1 or raid10 generally. But the mixed raid re data-metadata can help as metadata is often the costly part.

Another thing to assist here is to try running with quotas disabled. You then get no feedback on share size - but it can help greatly with performance. There does not exist a simple quota option in btrfs which is cheaper, work wise, to maintain and so quicker. But that is still too new to use in our Leap based options. And we as yet have no Web-UI integration/option to use that type of quota setup yet.

Hope that helps.

4 Likes
3

Thank you for your reply!

My installed version is openSUSE Leap 15.5. I’ve succesfully managed to run 2 hdparm tests:

  • Cached reads: 5639.20 MB/sec

  • Buffered disk reads: 100.15 MB/sec

I’ve also managed to do a “dd” for a sequential write speed test, which resulted in a 188 MB/sec result.

But what I really wanted to check is my disk random write speed and I am struggling to find a command line tool which could help me out. I have to admit that I am a very rookie linux user, so I would like to ask you to suggest me one or two tools which I could use to have a clean picture of my actual RAID writing speeds.

Thank you very much for your help, you’ve been very kind!

1 Like
4

You could take a look at fio (flexible I/O) - which allows you to specify anything you want (which also makes it more complex).

Install it using zypper in fio.

Documentation is here:
https://fio.readthedocs.io/en/latest/fio_doc.html
There are tutorials on how to use this pretty flexible tool, like this:

For the random I/O scenarios, this option is probably relevant for you:
--rw=str
some more description (while for a slightly different context, but should still be applicable):
https://portal.nutanix.com/page/documents/kbs/details?targetId=kA07V000000LX7xSAG

bonnie++ is also a decent suite apparently. But I am not sure how “functionally” old the latest release is.
Install it using: zypper install bonnie++

Here’s another overview for most of these tools with some usage examples:
https://linuxreviews.org/HOWTO_Test_Disk_I/O_Performance

4 Likes
5

Thanks! I’ve managed to install FIO and run a random write test and results are kinda good with an average of 100 MB/sec speed.

Now I am wondering why that application is taking that much to rebuild a 2MB xml file (almost 40 seconds). Application is running on Windows 11. Machine hardware is pretty decent (not a Tier S system, but good indeed). Rockstor share is mounted as NFS volume on Windows. Tried 2 different machines (one much much older and less performant, with Windows 10 installed on it) and results are the same. It feels like it is rockstor’s side, but I can’t be sure and I can’t find a way to investigate deeper this situation.

You guys have any suggestion?

6

Since a 100MB/s write speed seems to indicate that the disk array is not the bottleneck, I would next look at the network stack/connection.

You’re using NFS, which should be pretty fast. Could there be a network slowdown in the connection, causing the file rebuild taking longer these days? While NFS is probably the fastest connection, did you attempt to mount this as an SMB volume in windows to see the difference?

I assume, that’s not possible, but you can’t test run the application directly on a Linux environment, right?
Also, the application has not been recently updated since it went from the 5 secs to 30 secs?

3 Likes
7

Okay, my two cents may be helpful.

You can download and install for all users a Winderz program called CrystalDiskMark to test mapped network drives with one caveat
 it must run with normal non-administrator privileges.

The way to do it with Winderz 10/11 is:

  1. Download and install CrystalDiskMark
  2. Create a user account lacking admin privileges
  3. Log out of current session (or Lock your setup)
  4. Log back in using non-admin account
  5. Create a mapped drive to the network share you want to test
  6. Run CrystalDiskMark and do not allow admin privileges when it starts


Have fun!

:sunglasses:

CrystalDiskMark1
CrystalDiskMark11239×771 72.3 KB

CrystalDiskMark2
CrystalDiskMark21273×953 121 KB

CrystalDiskMark3
CrystalDiskMark31279×961 133 KB

2 Likes
8

Ok this is scary weird, but I want to try explaing what is happening, because it may have some valid reasons, but I don’t have the skill level nor the experience to give an answer. So here we go:

My main system is a windows 11 machine, where I am running a certain application, which (at boot) fetches various xml files, which are going to build the content of the application sub-menus (there are 20ish directories, each one containing an xml file, which store just some basic metadata of the files contained inside each directory).
These directories, and subsequentially every xml file are stored in my shared Rockstor pool, which is running on a gen3 i3 cpu, with 16GB of ddr3 ram, a cheap 128GB ssd where Rockstor is installed, another cheap 128GB ssd configured as cache disk and a couple of 3TB Western Digital RED (NAS Ware) @ 5400 rpm, configured in RAID1.
Rockstor is “exposing” the shared data with both NFS and SMB, and my main windows 11 system, where that applcation is running, is mounting the Rockstor share via NFS. Within this environment, the application takes more or less 100 seconds to launch. It was kinda not acceptable to me, even because I remember that it used to load very very faster than that and I couldn’t understand what was wrong. I tried to dive back in the past weeks to see if I could remember if I did some changes, somewhere, but I couldn’t, hence I started to think that my Rockstor system was somehow degraded (maybe the disks were failing or something like that) and so I started benchmarking everything out, but I couldn’t find anything related to the Rockstor performance.
And here we get to the point, when, among all the experiments I’ve done, I tried to mount the same share, on the same system, ALSO in SMB, so that we now have “Z:\blablabla” and “Y:\blablabla”: two identical mounted “network drives” (sorry if this is not the correct term), that just differ for their filesystem access mode (Z: is mounted as NFS, while Y: is mounted as SMB). I tried relaunching the application, having it configured to access those xml on Z (NFS), and it started up in 20 seconds (the same amount of time it used to!!). I then tried to do the same setting Y: (SMB) as target directory to fetch the xml files, configuring my application settings and it started up, again in 20 seconds.
Now the big question is: Why in this world mounting the same share two times, one in NFS and one in SMB, would improve so much my Rockstor disk access timings?
Sorry if I’ve wrote so much, but I thought that it could be interesting to investigate for someone. And I am competely at your disposal, if you want me to try, test and experiment some procedures for the academic purpose xD.

Thanks for your patience and your interest!

9

Umm, for me, I am in the process of building a third NAS setup just for testing purposes. The only times I have seen a slowdown in NAS (Rockstor) performance with Samba is when:

  1. Win11 did some auto-update thing and screws up the network speed until I reboot it
  2. Another computer has somehow caused my router to glitch
  3. I ran over a cable with the vacuum cleaner and ate it up
  4. Rockstor was doing a scrub or balance

Never tried NFS myself, don’t know how
 but that is another reason to build the test unit.

Probably nothing I mention here is helpful, but like you, I’m trying to figure things out as well.

:sunglasses:

3 Likes
10

does your application pull the xml files based on the mapped network drives (e.g. “read file from Y:\blablablaba”)?

11

Sorry but I didn’t understand the question :confused:

12

What I meant is that when your application collects files from Rockstor, how does it know where to get them from? Is it specified in the application to collect these files from a specific mapped network drive?

13

Oh yes, you do set that “storage” directory directly from within the application. The directory structure is like Z:\myStuff\myCollection\directory_1\content.xml; Z:\myStuff\myCollection\directory_2\content.xml, etc. etc. Let’s say the path I am setting is “Z:\myStuff\myCollection”. The application starts fetching from there for the given-named xml files it finds inside every subdirectory (directory_1, directory_2, etc.).

You can also edit the application config file to set that “storage” directory path.

14

Ok, that makes sense.

I still don’t have an inkling why your changed set up suddenly performed much better, but wanted to get more detail so I can think about it more :slight_smile: From what I read, NFS can do some file caching on the client after having read it from the server for subsequent read requests, however in your case it also sounds like that for every boot it does a single read of a given xml file and then doesn’t refer to them again 

And 
 SMB which wouldn’t do that, it also works 


15

The reading/writing performance in the pure sense of file copying, didn’t change. The only thing (for now at least) that changed is the speed that application can read/write those xml files. The application reads all of those xml files at every launch and it also “rebuild” them everytime a file of a given subdirectory is saved (it takes the entry of the last file used, change a value in the tag “last time used” and then append at the bottom of the xml, deleting the old entry).
I am describing it as a very shady application, but it is just to try describing exactly what it does. It is a retrogaming emulation frontend which I am talking about and if it is permitted or you need more details about that, I can try describing it in a more clear way. Just let me know :stuck_out_tongue:

16

Ah, is this something like Launchbox or EmulationStation?

What confuses me is that it now seems to work better, but there’s not really a good explanation for why a NFS/SMB combo to connect would improve the applications read/write performance (especially, since it now doesn’t seem to matter whether you’re mapping it to NFS or SMB). I am wondering whether something on your windows box or the network itself has changed allowing the application to better streamline the reads and appends/updates.

17

Yes, more precisely Emulation Station Desktop Edition (ES-DE). I have your exact same doubts and what I am suspecting now is that the reason why it used to run “normal” sometime ago, while having only an NFS mount in my windows system, is because I had an SMB mount, of the same share, on my iPhone. This would mean that it doesn’t matter if you do have an NFS and SMB mount on the same device, but it is enough that at least one of each is mounted somewhere. And this is insanely crazy. I could try to do some test and remove the SMB mount from my windows system and mount an SMB share on another computer (or directly into my iPhone like it was before). This would definitely give us a “true” response. But in that case, it would mean that it is something related to Rockstor itself? I’ll come back with some feedback, probably tomorrow (I hope).

18

Just for grins, I played with NFS a bit. First I enabled it Async, then tried Sync. I also did SMB transfers.

Over the 10Gbps LAN from the Winderz-11 setup to Rockstor, I get the following:

SMB Xfer
TestSMB0

NFS-Async Xfer
TestNFS0

NFS-Sync Xfer
TestNFS0Sync

The Winderz-11 Setup

TestNFS-SMB-Win11
TestNFS-SMB-Win111105×773 97.9 KB

The only real advantage I see to the NFS setup is that Winderz-11 can see it directly in the network map.

Also, notice the horrible Sync NFS speed! Possibly it was setup to use Sync instead of Async at first?

Just my lousy 2 cents


:sunglasses:

2 Likes
19

Thanks for the tests @Tex1954
I looked in the code, and I think Rockstor is creating an export with async mode:

unless specified differently by the user.

It seems, from the man page that:

   async  This option allows the NFS server to violate the NFS
          protocol and reply to requests before any changes made by
          that request have been committed to stable storage (e.g.
          disc drive).

          Using this option usually improves performance, but at the
          cost that an unclean server restart (i.e. a crash) can
          cause data to be lost or corrupted.

   sync   Reply to requests only after the changes have been
          committed to stable storage (see async above).

          In releases of nfs-utils up to and including 1.0.0, the
          async option was the default.  In all releases after
          1.0.0, sync is the default, and async must be explicitly
          requested if needed.

and the current version is 2.x, I believe.

but all this also doesn’t explain how SMB suddenly worked well, after testing all kinds of combos ,

20

didn’t change any export setting at all. It is all exactly how it has been since the first install. I am also addressing that this weird behaviour is not affecting the “simple” disk I/O operation, as a file-copying, at all, since there has been noticed no difference in the transfer speeds (with or without an SMB share mounted somewhere). Actually, the only difference (and a big one), is how fast that application (ES-DE) reads data from within those xml files (which basically store roms metadata of my emulated systems - one per directory) at launch. And how fast, the same application is able to modify and save the same xml files, when you quit a game and it automatically returns to the main GUI (the file operation is basically refreshing the “last time played” tag and save the file).