Samba write speeds are ultra slow

[orangejulio (Julio)]

Flox,

Again, I want to say thanks for you time. I get that this one is pretty mysterious, but I appreciate your efforts.

I’ve done as requested and made the updates you suggested to the config. The final version is at the bottom, so let me know if you spot anything that looks off. I also set the shares exactly as per the guide. I disabled shadow copy on them just to keep things simple.

One note is that I switched my Rockstor from the temp test version I had made the other day to rule out any install errors back to my original one with my actual data on the internal SSD. So even though the USB connection wasn’t impacting the xfer, it’s now out of the mix. Just mentioning this, as you’ll likely notice that there are more shares and that the names differ.

To test I copied a mixed folder of 2.6 GB (414 files: raw files, JPGs and tiny XMPs). While I completely agree that smaller files are going to perform worse, my Ubuntu server shows that this particular set of files can still achieve good speeds on writes. Also, Samba is able to get solid speeds in the inverse direction from both servers, so I don’t think that Samba as a protocol is inherently causing the slow speeds. But there could be some as-yet-undiscovered issue on my system that is causing the Rockstor flavor of Samba to be suffering.

Write:
To my old server: 26s (102 MB/s)
To my Rockstor: 465s (6 MB/s)

Read:
From my old server: 34s (78 MB/s)
From my Rockstor: 31s (85 MB/s)

I had disabled quotas a couple of days ago when you first asked about it, and the RAID level was already set to ‘single.’ I also just now tested the vfs objects = btrfs flag on one of the shares, but it didn’t make any difference.

So, it looks like I’m still at the same place, unless you spot something wrong in what I’ve shared. I thought of one more test that I could do, if I can find some time. I swap a drive and install Rockstor on my old Ubuntu hardware. If it worked well, then at least it would show that the issue is isolated to my new machine (even though it worked perfectly with Ubuntu and the network tests good with iperf). If it also performed poorly, then it would point to some kind of a flaw with the Rockstor version I installed. I should note that I have not done any Rockstor updates yet, because I don’t want to install the free beta versions but don’t want to pay for stable releases if I can’t get it to work right. So I’ll think about doing that.

Thanks!

Current smb.conf

[global]
log level = 3
map to guest = Bad User
cups options = raw
log file = /var/log/samba/log.%m
printcap name = /dev/null
load printers = no

####BEGIN: Rockstor SAMBA GLOBAL CUSTOM####
dns proxy = no
obey pam restrictions = yes
pam password change = yes
unix password sync = yes
workgroup = workgroup
####END: Rockstor SAMBA GLOBAL CUSTOM####

####BEGIN: Rockstor SAMBA CONFIG####
[temp]
root preexec = “/opt/rockstor/.venv/bin/mnt-share temp”
root preexec close = yes
comment = Samba-Export
path = /mnt2/temp
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[samba_share]
root preexec = “/opt/rockstor/.venv/bin/mnt-share samba_share”
root preexec close = yes
comment = Samba-Export
path = /mnt2/samba_share
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[media]
root preexec = “/opt/rockstor/.venv/bin/mnt-share media”
root preexec close = yes
comment = Samba-Export
path = /mnt2/media
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[archives]
root preexec = “/opt/rockstor/.venv/bin/mnt-share archives”
root preexec close = yes
comment = Samba-Export
path = /mnt2/archives
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[media_archived]
root preexec = “/opt/rockstor/.venv/bin/mnt-share media_archived”
root preexec close = yes
comment = Samba-Export
path = /mnt2/media_archived
browseable = yes
read only = no
guest ok = no
valid users = mediashare
####END: Rockstor SAMBA CONFIG####

[Hooverdan (Dan W.)]

My only major difference (in case that can help) to your config is that I have defined an admin user that I am utilizing to access the smb share (and copy data to).
Not sure, whether it matters, but I am also using the wsdd Rockon so the server appears in the Windows realm - mostly thinking about the get_myfulldnsname error you’re seeing.

I read some use case where a local DNS was configured and the host file had the incorrect entry causing the above error (and slow speed, too).

[orangejulio (Julio)]

Thanks, @Hooverdan.

I did initially have the shares set up with some admin users, but I found that it was writing files as root, which I didn’t want. I disabled them a few days ago, but I did also experience the poor write speeds when they were active.

Your 2nd point about a local DNS is interesting. I don’t know much about that. When I check the Rockstor network settings, I see 2 entries, which are pointing to CenturyLink servers. I didn’t define them, so they must have been pulled automatically during the install.

If there is something else that I should be checking, please let me know.

[Hooverdan (Dan W.)]

It doesn’t sound like you’re using anything like a local DNS, so I don’t think anything should be amiss on the hosts side, I assume, the only entry you have in the /etc/hosts file is something like this:

...
127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

It’s interesting that the admin users wrote actually as root. When I defined a non-root user as admin, that worked as well …
But … it does also not seem to be influencing your performance bottleneck.

if you run smbstatus what does the version come back with?

[orangejulio (Julio)]

Thanks @Hooverdan.

Yes, my hosts file looks just like that.

This is what smbstatus reports:

Also, I did go ahead and install Rockstor on my original (Ubuntu) server by disconnecting the root drive and installing it on a temp drive. I imported the BTRFS filesystem, so Rockstor was using the exact same hardware as Ubuntu had been. I had the same exact problem, where writes are too slow to be usable. I found errors in the log, but they were slightly different than what I was seeing on the other server:

I am really stumped by this! The part of me that likes to solve puzzles wants to keep going, but the part that wants to move on with my life just wants to install Ubuntu and set everything up manually, as I’ve done in the past. But I’d sure miss the nice GUI and ease of backing up the server config.

[Hooverdan (Dan W.)]

That is very strange - I am getting the 112MiB/sec transfer rates (large files though).

Only last thing I can think of is that you have a network card, for which the kernel drivers are not integrated in the Rockstor installation, forcing it into some random network driver which limits the speed that way. And that could be fixed by installing the missing ones …

similar to something like this (there were a couple of other threads out there building on this one, too):

[Solved] Missing network card drivers - Intel I225-V

I installed Rockstor 4.5.5 with the new Leap 15.4 installer. I did not install any kernel-firmware packages yet, just to get a baseline. Let’s see how that goes. Uptime for my previous install had dropped to 54%, which is … quite horrible and unreliable at best.

[orangejulio (Julio)]

@Hooverdan,

When I had tested with iperf, I got 950 mbps in both directions, so would your comment about the network card drivers still apply? I had also copied from the other server using SCP and got solid speeds (~500 mbps with mixed size files). This makes me think that even if the driver isn’t optimal, it’s functional.

[Hooverdan (Dan W.)]

That could be a good point.

I installed and ran iperf3 on my box and it seems to be close between that and my windows to Rockstor via samba physical file copy speed (~112-115MiB/s).

My system has this motherboard (so already a bit up there in age)

       product: C236 WSI
       vendor: ASRockRack

CPU:

        product: Intel(R) Core(TM) i3-6320 CPU @ 3.90GHz
        vendor: Intel Corp.

with this Network on it:

        product: I210 Gigabit Network Connection
        vendor: Intel Corporation
        driver version: 6.4.1-lp154.7.g3561b10-default

Note: I am using the Kernel backport so that driver version is a bit higher than what comes with the “normal” 4.6.0-0 Rockstor installation.

[Flox]

Thanks a lot @orangejulio and @Hooverdan for all the efforts here… I always find troubleshooting Samba frustrating as so many factors are at play. It does seems like we may have some DNS quirks happening here, somehow (“it’s always DNS”, after all).
@orangejulio, I’m not sure how this would differ between your tests on Ubuntu vs Rockstor, but maybe your network sees your Ubuntu and Rockstor machine “in a different way”… To make sure, is there anything particular with regards to DNS in your network? Also, would you think of anything from your client and its DNS configuration, or any particular setting that you specified in this client (your Windows machine) for the Ubuntu server vs Rockstor?

@orangejulio

I did initially have the shares set up with some admin users, but I found that it was writing files as root, which I didn’t want. I disabled them a few days ago, but I did also experience the poor write speeds when they were active.

That is indeed the normal behavior for the user(s) listed as admin users. The manpage for smb.conf reads:

This is a list of users who will be granted administrative privileges on the share. This means that they will do all file operations as the super-user (root).

Sorry again I still don’t have a clear answer here. I did find time to test yesterday and transfer of small files went fast from my laptop (openSUSE Tumbleweed) to a Rockstor VM on the same machine. Transferring from Windows is a whole different beast and I still need to re-setup my Windows 10 VM to test that. I do believe your screenshot comes from Win10 and not Win11, is that correct? I’m not sure it matters but I’d rather test something as close to your case as possible.

[orangejulio (Julio)]

Thanks, guys.

No custom DNS settings here. I am not a network person, so everything is basically out-of-the-box. If I don’t have to touch something network-related to get it working, I don’t.

@Flox you are correct, I am using Windows 10 on this machine. My wife’s computer is also Windows 10. If you are able to test it, I have the feeling that it will work fine, as it seems to for everybody else.

So to recap my experience so far:
Rockstor/new server: slow writes
Rockstor/new server, 2nd install: slow writes
Rockstor/old server: slow writes
Ubuntu/old server: fast writes
Ubuntu/new server: fast writes

At one point I had even uninstalled and reinstalled Samba on one of the Rockstor builds, and that made no difference.

It just doesn’t make sense.

[Flox]

Still no clear answer here, unfortunately, but I have taken time to set up a Win10 VM, connected to a test share exported via Samba by Rocktor, either directly from Windows Explorer, or via Add network location, or via “Map a network drive”. To test the transfer of small files, I tried transferring some random folders from the Windows directory as I know these are full of small files (as well as bigger ones). The transfer went fine, with speed ranging from sub 1 MB/s at times, but increasing for other files.

To compare with a base openSUSE Leap 15.4 OS install, on which Rockstor is based, I created a new VM with a Desktop installation of Leap 15.4, setup Samba and shared a folder using YaST so that the config would be optimal (as YaST does it “the right way”). Transferring the same Windows files from Win10 to this Samba share exported by Leap 15.4 shows the same behavior:

Rockstor thus seems to behave the same as a base Leap 15.4 install.
I’ll continue my attempts at digging further into this.

[Flox]

Another note: the DNS angle might be a red-herring as well.

This was referring to the get_myfulldnsname errors we are seeing in the samba logs on the server. This is only due to the fact that the server does not have an IP record for the hostname of the server. Adding that information to /etc/hosts fixes that issue but the speeds are still the same:

XXX.XXX.XXX.XXX <your-rocktor-hostname-here>

I’m tempted to try setting up an Ubuntu server and see… Honestly, the more I look at it, the more I don’t understand why you do NOT see slow downs with small files with your Ubuntu server…

I’ve also tested adding the rockstor hostname ↔ IP mapping info to the Win10 client’s hosts file, and connect to the Samba export using the hostname, but that didn’t make a difference either.

[Flox]

Noting also that I seem to see a bug from Win10 that was reported:

Windows 10 sends unauthenticated SMB requests - Microsoft Q&A

Windows 10 clients flooding Samba servers with phantom guest login attempts

Win10 keeps sending extra unauthenticated requests, which explains why I keep seeing lines like the following in the logs:

[2023/07/08 16:15:30.672648,  3] ../../source3/auth/auth.c:202(auth_check_ntlm_password)
  check_ntlm_password:  Checking password for unmapped user []\[]@[WINCLIENT] with the new password interface

I’m testing this with Win10 Pro version 22H2. I unfortunately cannot update that VM as it is too constrained in disk space (12G is apparently not enough for Win10).

Again, most likely not the culprit in your case, but spamming the Samba server with such requests most likely does not help.

[orangejulio (Julio)]

Wow, @Flox, I appreciate all that you’ve done here. That was very extensive.

I’m still giving myself some more time, but I find myself leaning towards just sticking with Ubuntu, as it seems that I’m the odd man out here, and I’m starting to think that it’s not worth all the effort, as no one else seems to be having the same experience as me.

But I’m not giving up yet.

[Flox]

@orangejulio,

I can be stubborn at times with this kind of things, and it bothers me not to be able to figure that one out. I tested with yet another debian-based server acting as the Samba server, and I still see the same behavior than with Rockstor or with a base Leap 15.4 installation. The more I try, the more I think it’s unrelated to the Samba config, especially given all the testing you have done on your end.

I’ll keep poking at it as I can…

[orangejulio (Julio)]

Thanks, @Flox.

I am thinking the same thing (not a Samba config), but then I can’t think of anything else that would cause it. So, so strange.

[Jorma_Tuomainen]

Have you tried to add Rockstor / Rockstor-i3 hostnames to /etc/hosts, to get rid of those errors (have been known to cause slowness)?

[orangejulio (Julio)]

This!

@Jorma_Tuomainen

Have you tried to add Rockstor / Rockstor-i3 hostnames to /etc/hosts, to get rid of those errors (have been known to cause slowness)?

I couldn’t believe it, but simply adding

192.168.0.30 Rockstor

to my hosts file was all it took. The difference is night and day! My write speed went from this:

to this:

I then removed the entry, and it slowed way down again. Then after putting it back in, it sped right back up.

The interesting thing is that I checked my Ubuntu server, and it doesn’t have any entry in the hosts file (besides localhost). So no idea why that one works fine w/o it, but Samba on Rockstor did. But I’ll treat this like I do everything Linux. I may not understand it, but I’m just happy that it works.

Thanks @Jorma_Tuomainen for getting this fixed for me! And thanks again, @Flox and @Hooverdan for all the time that you put into helping me. This was such a strange problem, and it made sense to try all the things that you suggested to get more info and rule things out. I was thinking I’d overwrite with Ubuntu by the weekend, so now I can really give Rockstor a shot as my permanent NAS solution!

[Flox]

Great news! I’m glad you got it resolved!

Thank you so much for saving the day, @Jorma_Tuomainen!

Maybe we can think of doing this automatically for users when turning the Samba service ON as I suspect it would affect quite a few users. We would need to be careful about it, though, as I don’t want Rockstor to impose a configuration of /etc/hosts upon users who may not want it for their own reason(s). Maybe as an option when configuring the Samba service… It’s worth thinking about it at least.

@orangejulio, from all the different things I had you test in vain, I think there is one worth considering keeping: adding vfs objects = btrfs to your Samba export customization. While it has not been thoroughly tested by us yet, this is something we are seriously thinking of adding by default as it is designed to bring some nice performance improvements when copying files within the share, for instance (if I recall correctly). It also has been around for quite some time now.

Either way, let us know how things go, and thanks for opening this thread!

[Jorma_Tuomainen]

It’s always DNS.

Well in this case local name resolution but it’s the same thing