Another note: the DNS angle might be a red-herring as well.
This was referring to the get_myfulldnsname errors we are seeing in the samba logs on the server. This is only due to the fact that the server does not have an IP record for the hostname of the server. Adding that information to /etc/hosts fixes that issue but the speeds are still the same:
XXX.XXX.XXX.XXX <your-rocktor-hostname-here>
I’m tempted to try setting up an Ubuntu server and see… Honestly, the more I look at it, the more I don’t understand why you do NOT see slow downs with small files with your Ubuntu server…
I’ve also tested adding the rockstor hostname ↔ IP mapping info to the Win10 client’s hosts file, and connect to the Samba export using the hostname, but that didn’t make a difference either.
Noting also that I seem to see a bug from Win10 that was reported:
Win10 keeps sending extra unauthenticated requests, which explains why I keep seeing lines like the following in the logs:
[2023/07/08 16:15:30.672648, 3] ../../source3/auth/auth.c:202(auth_check_ntlm_password)
check_ntlm_password: Checking password for unmapped user []\[]@[WINCLIENT] with the new password interface
I’m testing this with Win10 Pro version 22H2. I unfortunately cannot update that VM as it is too constrained in disk space (12G is apparently not enough for Win10).
Again, most likely not the culprit in your case, but spamming the Samba server with such requests most likely does not help.
Wow, @Flox, I appreciate all that you’ve done here. That was very extensive.
I’m still giving myself some more time, but I find myself leaning towards just sticking with Ubuntu, as it seems that I’m the odd man out here, and I’m starting to think that it’s not worth all the effort, as no one else seems to be having the same experience as me.
I can be stubborn at times with this kind of things, and it bothers me not to be able to figure that one out. I tested with yet another debian-based server acting as the Samba server, and I still see the same behavior than with Rockstor or with a base Leap 15.4 installation. The more I try, the more I think it’s unrelated to the Samba config, especially given all the testing you have done on your end.
to my hosts file was all it took. The difference is night and day! My write speed went from this:
to this:
I then removed the entry, and it slowed way down again. Then after putting it back in, it sped right back up.
The interesting thing is that I checked my Ubuntu server, and it doesn’t have any entry in the hosts file (besides localhost). So no idea why that one works fine w/o it, but Samba on Rockstor did. But I’ll treat this like I do everything Linux. I may not understand it, but I’m just happy that it works.
Thanks @Jorma_Tuomainen for getting this fixed for me! And thanks again, @Flox and @Hooverdan for all the time that you put into helping me. This was such a strange problem, and it made sense to try all the things that you suggested to get more info and rule things out. I was thinking I’d overwrite with Ubuntu by the weekend, so now I can really give Rockstor a shot as my permanent NAS solution!
Maybe we can think of doing this automatically for users when turning the Samba service ON as I suspect it would affect quite a few users. We would need to be careful about it, though, as I don’t want Rockstor to impose a configuration of /etc/hosts upon users who may not want it for their own reason(s). Maybe as an option when configuring the Samba service… It’s worth thinking about it at least.
@orangejulio, from all the different things I had you test in vain, I think there is one worth considering keeping: adding vfs objects = btrfs to your Samba export customization. While it has not been thoroughly tested by us yet, this is something we are seriously thinking of adding by default as it is designed to bring some nice performance improvements when copying files within the share, for instance (if I recall correctly). It also has been around for quite some time now.
Either way, let us know how things go, and thanks for opening this thread!
Sure, I have no problem adding that. I think it would be preferable to help both performance and disk space.
I just tested it, but I don’t think it’s taking effect, as a copy test (from one folder to another on the same share) is taking the same amount of time and has similar network activity. I added that line to the Samba entry for one of my shares (temp), stopped and then re-started Samba. Went back and forth a few times, but all seemed to be about the same. Below are the network activity graphs (first few seconds of each are missing due to me switching back to the dashboard).
Did I implement it correctly, or is there something else I should be doing to enable it?
Not sure whether that makes the difference, but you have the vfs objects parameter listed twice (for shadow copy and for btrfs). My understanding is that you place all vfs objects in one line (separated by space), so it would be (assuming they’re loaded and used in order):
I believe @Hooverdan may have the explanation here. That being said, I have not found time to check how to properly test this module.
Nevertheless, you may have just found a little limitation in the ability to input VFS objects as customization. We indeed need the shadow _ copy one because you checked the windows snapshot checkbox; but the customization you entered was not aware of that. We will need to make that more “smart” to account for this kind of scenario.
In the meantime, you can test it manually but it will be overridden every time you edit a samba export or toggle the samba service ON/OFF using the UI. If you want to test it manually, you can export the share without specifying the btrfs object in the customization box, then manually edit smb.conf as @Hooverdan detailed, and then restart the SBA service manually (systemctl restart smb nmb).
As the first entry (shadow copy) was auto-generated, I manually updated the file to include both entries on one line as you suggested.
After more testing, I am now thinking that it is doing btrfs de-duped copies whether or not that flag is set. To make the test more consistent, I started copying a larger folder with larger files (3 files, 10 GB total). What I am seeing is that the xfer speeds are way beyond what my 1 gbps network can support, even without adding the btrfs flag.
So I’m not sure that this setting is needed. I tested it both with and without the btrfs flag, and the speed is similarly great. If I’m missing something here, let me know please.
@Flox I didn’t see your reply until after I replied to @Hooverdan.
In my limited experience, it appears that Rockstor will let you start/stop Samba without over-writing a manually-modified smb.conf. It seems to only alter the config if I make a change to a share (and presumably to the global config as well). But just to be sure, I redid the manual editing and did the manual restart each time.
Whether or not I have the btrfs flag in there (on the same line as the shadow_copy, the speeds are really high, and it’s clearly not actually transferring all the data back and forth over the network.
My thinking is that samba already knows that it’s a btrfs filesystem and is taking advantage of that already. Seems like that’s not expected behavior, but that’s what I’m seeing here. It would probably be good to have someone else test this as well.
No worries… I think I saw your message right as I was tapping the “submit” button.
Thanks a lot for testing that; it’s actually really informative and valuable. I’ll try to provide some elements of answers here and there.
Correct; it seems I was mis-remembering. There are pretty much 4 scenarios:
you manually restart Samba at the command line using systemctl restart smb. Rockstor doesn’t even know when you do that so there is no change in smb.conf. Rocsktor still checks the status of smb.service when you visit the “Services” or “Samba exports” pages, though.
you toggle the Samba service ON/OFF: all Rockstor does then, is the same as above, so no change in smb.conf.
you edit/create/delete a Samba export: Rockstor will overwrite the shares sections in smb.conf.
Impressive speeds indeed! Samba supports server-side copy since v4.1, it seems, which means that transfer over the network should NOT happen when copying a file within the same Samba share from the client. What the btrfs vfs object brings on top of that, is that you shouldn’t see disk activity either when doing that. See the following section of the Samba wiki for a much better explanation: Server-Side Copy - SambaWiki. Now, it is possible that recent Samba/kernel versions have a much better support for COW, which would make that aspect of the btrfs vfs object not that useful. That may be why you see what you see).
Now, the btrfs object does bring a bit more than that, and allow support for transparent file compression from Windows, for instance. See the nice section of openSUSE Leap’s docs on the matter: Samba | Reference | openSUSE Leap 15.5.
Thanks again for reporting this; that is extremely valuable.
I have created a corresponding issue (and referenced other vfs open items) to address the approach on how to deal with vfs modules in general via the Web-UI:
I think I am understanding it more. My speeds were high enough that it implies that it wasn’t doing any disk copying of the data, but I think I need to do more tests to actually prove it.
When I have a little more time I will set up a fresh share and do something conclusive, such as copying a very large file and taking snapshots and checking for unique data in them.
Regarding the ability to set compression from the Windows properties dialog, I did not have that option on mine. It didn’t have a button for ‘advanced.’ This laptop is on Win 11, so it may be different. Or since I have in enabled on the entire pool, perhaps it doesn’t share the option. I’ll try to look more into that, too.
Just following up after doing some more tests. I am not totally confident that regardless of whether vfs object = btrfs is set or not.
I created 2 new shares, one on my main data SSD (SATA) and another on the root volume (NVME) and exposed both via samba (default settings, so no vfs object param).
I copied a single 7.9 GB file to each of those shares. As expected per @Flox’s comments/links, there was no network transfer to or from my desktop. The copy to the SATA drive averaged 95 MB/s, which is reasonable given that it’s not a high end drive and is reading and writing from the same device. The copy to the NVME share was much faster at 280 MB/s. So that was working as designed.
I then made a copy of each of those files into the same directory. It was effectively instant in both cases, as the Windows copy window never even popped up. This tells me that it is already supporting COW out-of-the-box.
I then set vfs object = btrfs on both shares and restarted Samba. Same result, instant copies. However, I could now see the advanced button that let me set compression. This was on a Win 10 desktop. I went back to the Win 11 laptop, and I was able to see it on there as well. So I must have gotten confused earlier and checked it when it was not enabled.
I then enabled shadow copy and manually added the btrfs flag: vfs object = btrfs shadow_copy2.
I could access the compression successfully.
Then I restored it to vfs object = shadow_copy2 and added a 2nd entry for vfs object = btrfs. The compression option available this way, too, so it appears it doesn’t matter whether it’s combined or on a separate line. That means users should be able to add it themselves to a share on top of shadow copy, even without it being built-in to Rockstor.
Note that in both Win 10 and 11, there was no visual cue as to whether files were compressed. Per the link it should have colored the file names blue.
Well that’s interesting… rather confusing results but still very helpful.
That’s a great demonstration that not only does the server-side copy work, but so COW support.
I tested something related to the following:
I couldn’t find a clear answer in the manuals or elsewhere on how Samba deals with multiple definitions of vfs objects = for instance, so I tested it using the Samba testparm utility (testparm) as it returns the “parsed” version of the options defined in smb.conf.
