Samba write speeds are ultra slow

I am a new Rockstor user. I have been running various versions of an Ubuntu server DIY NAS/container host, but I decided to give Rockstor a try this week, as it looked like it’d be more friendly to manage. I got a server set up very quickly, but then I found that my Samba write speeds were much slower than on my other server.

Samba read is great. Samba write is OK for large files, although even with them it will occasionally dip down to zero or near zero speeds. Even deleting folders can be quite slow. I fully understand that small files are not going to transfer as quickly, but the write will crawl and often drop down to zero.

My previous server (similar HW and network) will be at least many times faster for writing the same folders. As an example, a 1 GB (216 file) folder just took about 5 seconds to copy to my Ubuntu NAS, and it is only 1/2 done after 2:30 on my Rockstor. As you can see in the graph, it runs for a bit and then kind of craps out. Large files will do the same, but the speed will recover. With small files, it just stays slow for the rest of the transfer.

image

When I check the Samba log files, I see errors that I don’t see on my other server. Lots of these:

 Allowed connection from 192.168.0.3 (192.168.0.3)
[2023/07/03 17:13:44.482209,  1] ../../source3/smbd/service.c:357(create_connection_session_info)
  create_connection_session_info: **guest user** (from session setup) not permitted to access this share (seagate_test)
[2023/07/03 17:13:44.482231,  1] ../../source3/smbd/service.c:546(make_connection_snum)
  create_connection_session_info failed: **NT_STATUS_ACCESS_DENIED**

However, I had commented out map to guest = Bad User, and I successfully logged into the share.

I tested my network between machines, and I get 950+ mbps in both directions. And copying files via SCP fly, so I believe that Samba is getting hung up on this guest user thing. I’ve tried many suggestions I’ve found in forums, and in fact I just installed Rockstor from scratch to make sure that I didn’t inadvertently mess something up earlier. Same experience.

I really want to make Rockstor work (and subscribe), as it’ll make my NAS life easier, but I can’t switch over if I can’t get this working. I keep hoping that I’m missing something dumb, but I’ve never had this problem with Samba before. The fact that others aren’t complaining about it gives me hope that it’s fixable. If anyone could help me fix this, I would be incredibly grateful.

Thanks!

Smb.conf:

[global]
    log level = 3
#    map to guest = Bad User
    cups options = raw
    log file = /var/log/samba/log.%m
    printcap name = /dev/null
    load printers = no

####BEGIN: Rockstor SAMBA GLOBAL CUSTOM####
    workgroup = workgroup
####END: Rockstor SAMBA GLOBAL CUSTOM####

####BEGIN: Rockstor SAMBA CONFIG####
[seagate_test]
    root preexec = "/opt/rockstor/.venv/bin/mnt-share seagate_test"
    root preexec close = yes
    comment = Samba-Export
    path = /mnt2/seagate_test
    browseable = yes
    read only = no
    guest ok = no
    wide links = yes
    follow symlinks = yes
####END: Rockstor SAMBA CONFIG####

Testparm:

Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

# Global parameters
[global]
        load printers = No
        log file = /var/log/samba/log.%m
        printcap name = /dev/null
        idmap config * : backend = tdb
        cups options = raw


[seagate_test]
        comment = Samba-Export
        path = /mnt2/seagate_test
        read only = No
        root preexec = "/opt/rockstor/.venv/bin/mnt-share seagate_test"
        root preexec close = Yes
        wide links = Yes

Hardware:
Beelink Mins 12 Pro (N100) with gigabit LAN.
iperf shows gigabit speeds in both directions.
SCP copy to same share using same files is at normal, acceptable speeds.

1 Like

Hi @orangejulio , and welcome to the community!

I’m sorry you hit a snag; I’ll see if I can help you or at least move things forward.

If the guest access is indeed part of the problem here, could you confirm whether you want guest access on your share in question? That’ll help us make sure we’re digging in the correct direction.

Could you also provide a bit more details on your hardware, specifically your storage? Focusing on the seagate_test share we see in your smb.conf: what is the parent pool and its configuration?

  • raid level? single, raid1, etc…
  • quotas: enabled/disabled?
  • disk(s): how many disks comprise that pool and how are they connected to the system? Are they simply connected to the motherboard, or are you using an HBA card, or a HDD enclosure?

Would you happen to have access to the smb.conf of that other server where you do NOT see slowdowns, by any chance? It’ll help pinpoint the difference(s) and maybe identify a culprit that way.

Thanks in advance for the information; I’ll try to see if I can replicate this kind of slowdown in the meantime.

1 Like

Another set of question I forgot in my previous message:
Could you provide some detail on how you are accessing this Samba export? It’s from a Windows client as we can see, but how did you access the Samba export? Did you directly access it using its IP address, or using its hostname?

Thanks for the replies, @Flox!

The seagate_test share is just a spare hard drive that I added to the rebuilt version to rule out any issues with the SSD I used on my first rockstor build attempt. It’s using LUKS encyption, and the pool is using just the single disk. This new test disk is connected through USB3 (internal drive in a dock), but all my original testing was done on two pools, and internal SSD and an external USB3 drive. The write speed going down to zero was consistent across all drives and both sever builds.

I did have quotas enabled, as that was the default. I just now disabled quotas but saw no improvement. I also tried disabling compression, but that didn’t help either.

Regarding the guest user question, I would prefer not to have guest access. Either a machine would connect via the mediashare user or not at all.

I accessed through a Windows client using the IP address, not by the hostname.I just now tried connecting via the hostname, and I got the same super slow writes and errors in the samba log. My CPU is averaging < 5% during the transfer, so it’s not a limiting factor.

Pool:

Below is the smb.conf from my other server. It was created (years ago) from the sample file, and I altered just enough to get it working for me. I stripped out all the commenting to make it easier to read. A couple of days ago I used it as a base for test version for my rockstor build, and it didn’t make any difference. I am not saying there couldn’t be a problem in there, as I’m not a samba expert, but I did want to mention that I had tried doing that.

I really appreciate your time and help on this. I like rockstor quite a bit, so I would really love to get this working. Thanks!

working smb.conf from ubuntu server:

[global]
log level = 3
workgroup = WORKGROUP
server string = %h server (Samba, Ubuntu)
dns proxy = no
log file = /var/log/samba/log.%m
max log size = 1000
syslog = 0
panic action = /usr/share/samba/panic-action %d
server role = standalone server
passdb backend = tdbsam
obey pam restrictions = yes
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
map to guest = bad user
usershare allow guests = yes

[BigData]
path = /BigData
valid users = mediashare
follow symlinks = yes
read only = no
1 Like

I went ahead and did one more test to conclusively rule out any HW issues. Using the exact same system, I installed Ubuntu 22.04 on a spare drive then created a very basic Samba setup. I then copied over the same folder on my PC that I was using earlier.

It worked great and was actually slightly faster than my old server. As you can see below, it held very steady around 110 MB/s for most of the transfer and dipped down to 80 just once. It never stuttered or seemed to be confused.

So I am very confused as both times I’ve installed Rockstor, I get poor Samba write performance. But every time I use Samba on Ubuntu, I get normal speeds, even on the same exact HW. I’m totally open to the idea that I’m doing something wrong with Rockstor, as it doesn’t seem to affect others. But I just can’t figure out what it is! :grin:

I am thinking that I will give Rockstor a little more time, as I really would prefer to use it, but if I can’t get Samba working well in the near future, I will switch this new server to Ubuntu.

image

1 Like

Thanks for all the troubleshooting here. It’s very helpful to know you get normal performance on that same hardware. Using USB to connect the drive is really not optimal but it seems you know that already so I won’t repeat it :stuck_out_tongue:

My initial thought would be to try the settings from the old server that differ from those default in Rockstor. It seems you’ve already done some of that, but I thought I would make that explicit here to rule out any misuderstanding on my end. From your old server config, I see the following that differ from the default smb.conf settings (taken from smb.conf(5) — samba-client):

Global settings

  • dns proxy = no
  • obey pam restrictions = yes
  • unix password sync = yes
  • pam password change = yes

Share settings

  • valid users = mediashare

Also to make sure I don’t misunderstand your setup, make sure you configured Samba as described in our docs (Services — Rockstor documentation). To add the global settings listed above, simply paste them in the “Custom global configuration” text box there.
With regards to the Samba export, also refer to the docs as well (Samba/CIFS — Rockstor documentation), using same settings as in the screenshot there, and add the Share setting listed above in the “Custom configuration” text box as well.

I would also try with the Btrfs quotas disabled and a raid level of “single” as you have to reduce any potential factors at play.

There is also another setting in the smb.conf used by Rockstor that we can adjust, but it shouldn’t be a problem in your case so let’s keep it for another attempt.

These slow speeds observed only when transferring a large amount of small files with Samba seem to be not-unusual based on some readings and make sense given the fact that Samba seems to be re-negotiating the transfer before every single file to be transferred, thereby increasing the overhead when compared to transferring large files.

It could also be interesting to test the following optimization that we’ve considered a little while ago but haven’t had time to test yet: [Samba] Enable the btrfs vfs module · Issue #2136 · rockstor/rockstor-core · GitHub

   [share]
   vfs objects = btrfs

To test it, you would just need to add vfs objects = btrfs to the “Custom configuration” text box when creating a Samba export.

Sorry I can’t provide a more definitive answer yet; this really is a particular one and I haven’t had found the time to test it myself yet, unfortunately. Let’s hope we can make progress with it, and thank you very much for your willingness in helping troubleshoot this!

2 Likes

Flox,

Again, I want to say thanks for you time. I get that this one is pretty mysterious, but I appreciate your efforts.

I’ve done as requested and made the updates you suggested to the config. The final version is at the bottom, so let me know if you spot anything that looks off. I also set the shares exactly as per the guide. I disabled shadow copy on them just to keep things simple.

One note is that I switched my Rockstor from the temp test version I had made the other day to rule out any install errors back to my original one with my actual data on the internal SSD. So even though the USB connection wasn’t impacting the xfer, it’s now out of the mix. Just mentioning this, as you’ll likely notice that there are more shares and that the names differ.

To test I copied a mixed folder of 2.6 GB (414 files: raw files, JPGs and tiny XMPs). While I completely agree that smaller files are going to perform worse, my Ubuntu server shows that this particular set of files can still achieve good speeds on writes. Also, Samba is able to get solid speeds in the inverse direction from both servers, so I don’t think that Samba as a protocol is inherently causing the slow speeds. But there could be some as-yet-undiscovered issue on my system that is causing the Rockstor flavor of Samba to be suffering.

Write:
To my old server: 26s (102 MB/s)
To my Rockstor: 465s (6 MB/s)

Read:
From my old server: 34s (78 MB/s)
From my Rockstor: 31s (85 MB/s)

I had disabled quotas a couple of days ago when you first asked about it, and the RAID level was already set to ‘single.’ I also just now tested the vfs objects = btrfs flag on one of the shares, but it didn’t make any difference.

So, it looks like I’m still at the same place, unless you spot something wrong in what I’ve shared. I thought of one more test that I could do, if I can find some time. I swap a drive and install Rockstor on my old Ubuntu hardware. If it worked well, then at least it would show that the issue is isolated to my new machine (even though it worked perfectly with Ubuntu and the network tests good with iperf). If it also performed poorly, then it would point to some kind of a flaw with the Rockstor version I installed. I should note that I have not done any Rockstor updates yet, because I don’t want to install the free beta versions but don’t want to pay for stable releases if I can’t get it to work right. So I’ll think about doing that.

Thanks!

Current smb.conf

[global]
log level = 3
map to guest = Bad User
cups options = raw
log file = /var/log/samba/log.%m
printcap name = /dev/null
load printers = no

####BEGIN: Rockstor SAMBA GLOBAL CUSTOM####
dns proxy = no
obey pam restrictions = yes
pam password change = yes
unix password sync = yes
workgroup = workgroup
####END: Rockstor SAMBA GLOBAL CUSTOM####

####BEGIN: Rockstor SAMBA CONFIG####
[temp]
root preexec = “/opt/rockstor/.venv/bin/mnt-share temp”
root preexec close = yes
comment = Samba-Export
path = /mnt2/temp
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[samba_share]
root preexec = “/opt/rockstor/.venv/bin/mnt-share samba_share”
root preexec close = yes
comment = Samba-Export
path = /mnt2/samba_share
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[media]
root preexec = “/opt/rockstor/.venv/bin/mnt-share media”
root preexec close = yes
comment = Samba-Export
path = /mnt2/media
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[archives]
root preexec = “/opt/rockstor/.venv/bin/mnt-share archives”
root preexec close = yes
comment = Samba-Export
path = /mnt2/archives
browseable = yes
read only = no
guest ok = no
valid users = mediashare
[media_archived]
root preexec = “/opt/rockstor/.venv/bin/mnt-share media_archived”
root preexec close = yes
comment = Samba-Export
path = /mnt2/media_archived
browseable = yes
read only = no
guest ok = no
valid users = mediashare
####END: Rockstor SAMBA CONFIG####

2 Likes

My only major difference (in case that can help) to your config is that I have defined an admin user that I am utilizing to access the smb share (and copy data to).
Not sure, whether it matters, but I am also using the wsdd Rockon so the server appears in the Windows realm - mostly thinking about the get_myfulldnsname error you’re seeing.

I read some use case where a local DNS was configured and the host file had the incorrect entry causing the above error (and slow speed, too).

2 Likes

Thanks, @Hooverdan.

I did initially have the shares set up with some admin users, but I found that it was writing files as root, which I didn’t want. I disabled them a few days ago, but I did also experience the poor write speeds when they were active.

Your 2nd point about a local DNS is interesting. I don’t know much about that. When I check the Rockstor network settings, I see 2 entries, which are pointing to CenturyLink servers. I didn’t define them, so they must have been pulled automatically during the install.

If there is something else that I should be checking, please let me know.

1 Like

It doesn’t sound like you’re using anything like a local DNS, so I don’t think anything should be amiss on the hosts side, I assume, the only entry you have in the /etc/hosts file is something like this:

...
127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

It’s interesting that the admin users wrote actually as root. When I defined a non-root user as admin, that worked as well …
But … it does also not seem to be influencing your performance bottleneck.

if you run smbstatus what does the version come back with?

Thanks @Hooverdan.

Yes, my hosts file looks just like that.

This is what smbstatus reports:

Also, I did go ahead and install Rockstor on my original (Ubuntu) server by disconnecting the root drive and installing it on a temp drive. I imported the BTRFS filesystem, so Rockstor was using the exact same hardware as Ubuntu had been. I had the same exact problem, where writes are too slow to be usable. I found errors in the log, but they were slightly different than what I was seeing on the other server:

I am really stumped by this! The part of me that likes to solve puzzles wants to keep going, but the part that wants to move on with my life just wants to install Ubuntu and set everything up manually, as I’ve done in the past. But I’d sure miss the nice GUI and ease of backing up the server config.

1 Like

That is very strange - I am getting the 112MiB/sec transfer rates (large files though).

Only last thing I can think of is that you have a network card, for which the kernel drivers are not integrated in the Rockstor installation, forcing it into some random network driver which limits the speed that way. And that could be fixed by installing the missing ones …

similar to something like this (there were a couple of other threads out there building on this one, too):

2 Likes

@Hooverdan,

When I had tested with iperf, I got 950 mbps in both directions, so would your comment about the network card drivers still apply? I had also copied from the other server using SCP and got solid speeds (~500 mbps with mixed size files). This makes me think that even if the driver isn’t optimal, it’s functional.

That could be a good point.

I installed and ran iperf3 on my box and it seems to be close between that and my windows to Rockstor via samba physical file copy speed (~112-115MiB/s).

My system has this motherboard (so already a bit up there in age)

       product: C236 WSI
       vendor: ASRockRack

CPU:

        product: Intel(R) Core(TM) i3-6320 CPU @ 3.90GHz
        vendor: Intel Corp.

with this Network on it:

        product: I210 Gigabit Network Connection
        vendor: Intel Corporation
        driver version: 6.4.1-lp154.7.g3561b10-default

Note: I am using the Kernel backport so that driver version is a bit higher than what comes with the “normal” 4.6.0-0 Rockstor installation.

Thanks a lot @orangejulio and @Hooverdan for all the efforts here… I always find troubleshooting Samba frustrating as so many factors are at play. It does seems like we may have some DNS quirks happening here, somehow (“it’s always DNS”, after all).
@orangejulio, I’m not sure how this would differ between your tests on Ubuntu vs Rockstor, but maybe your network sees your Ubuntu and Rockstor machine “in a different way”… To make sure, is there anything particular with regards to DNS in your network? Also, would you think of anything from your client and its DNS configuration, or any particular setting that you specified in this client (your Windows machine) for the Ubuntu server vs Rockstor?

That is indeed the normal behavior for the user(s) listed as admin users. The manpage for smb.conf reads:

This is a list of users who will be granted administrative privileges on the share. This means that they will do all file operations as the super-user (root).

Sorry again I still don’t have a clear answer here. I did find time to test yesterday and transfer of small files went fast from my laptop (openSUSE Tumbleweed) to a Rockstor VM on the same machine. Transferring from Windows is a whole different beast and I still need to re-setup my Windows 10 VM to test that. I do believe your screenshot comes from Win10 and not Win11, is that correct? I’m not sure it matters but I’d rather test something as close to your case as possible.

2 Likes

Thanks, guys.

No custom DNS settings here. I am not a network person, so everything is basically out-of-the-box. If I don’t have to touch something network-related to get it working, I don’t.

@Flox you are correct, I am using Windows 10 on this machine. My wife’s computer is also Windows 10. If you are able to test it, I have the feeling that it will work fine, as it seems to for everybody else. :slight_smile:

So to recap my experience so far:
Rockstor/new server: slow writes
Rockstor/new server, 2nd install: slow writes
Rockstor/old server: slow writes
Ubuntu/old server: fast writes
Ubuntu/new server: fast writes

At one point I had even uninstalled and reinstalled Samba on one of the Rockstor builds, and that made no difference.

It just doesn’t make sense.

1 Like

Still no clear answer here, unfortunately, but I have taken time to set up a Win10 VM, connected to a test share exported via Samba by Rocktor, either directly from Windows Explorer, or via Add network location, or via “Map a network drive”. To test the transfer of small files, I tried transferring some random folders from the Windows directory as I know these are full of small files (as well as bigger ones). The transfer went fine, with speed ranging from sub 1 MB/s at times, but increasing for other files.

To compare with a base openSUSE Leap 15.4 OS install, on which Rockstor is based, I created a new VM with a Desktop installation of Leap 15.4, setup Samba and shared a folder using YaST so that the config would be optimal (as YaST does it “the right way”). Transferring the same Windows files from Win10 to this Samba share exported by Leap 15.4 shows the same behavior:
image

Rockstor thus seems to behave the same as a base Leap 15.4 install.
I’ll continue my attempts at digging further into this.

2 Likes

Another note: the DNS angle might be a red-herring as well.

This was referring to the get_myfulldnsname errors we are seeing in the samba logs on the server. This is only due to the fact that the server does not have an IP record for the hostname of the server. Adding that information to /etc/hosts fixes that issue but the speeds are still the same:

XXX.XXX.XXX.XXX <your-rocktor-hostname-here>

I’m tempted to try setting up an Ubuntu server and see… Honestly, the more I look at it, the more I don’t understand why you do NOT see slow downs with small files with your Ubuntu server…

I’ve also tested adding the rockstor hostname ↔ IP mapping info to the Win10 client’s hosts file, and connect to the Samba export using the hostname, but that didn’t make a difference either.

1 Like

Noting also that I seem to see a bug from Win10 that was reported:

Win10 keeps sending extra unauthenticated requests, which explains why I keep seeing lines like the following in the logs:

[2023/07/08 16:15:30.672648,  3] ../../source3/auth/auth.c:202(auth_check_ntlm_password)
  check_ntlm_password:  Checking password for unmapped user []\[]@[WINCLIENT] with the new password interface

I’m testing this with Win10 Pro version 22H2. I unfortunately cannot update that VM as it is too constrained in disk space (12G is apparently not enough for Win10).

Again, most likely not the culprit in your case, but spamming the Samba server with such requests most likely does not help.

1 Like

Wow, @Flox, I appreciate all that you’ve done here. That was very extensive.

I’m still giving myself some more time, but I find myself leaning towards just sticking with Ubuntu, as it seems that I’m the odd man out here, and I’m starting to think that it’s not worth all the effort, as no one else seems to be having the same experience as me.

But I’m not giving up yet. :slight_smile:

3 Likes