Limited upload-speed for Linux based systems through pfsense
-
As the title says, we are experiencing a very strange behaviour when uploading files from Linux-based systems behind a pfsense.
On to the story:
We use a pfsense box for a midsize non-profit company mainly as router, firewall and OpenVPN-Server.
It's a Windows-Domain with about 70 Windows-Clients and a Synology RS3621RPxs used as fileserver.
Internet connection is a symmetrical 1 GBit-Line.
Hardware for the pfsense is a Supermicro SYS-5018D-FN8T with a XEON D-1518 and 16 GB of RAM.About a month ago, users in homeoffice started complaining about slow file downloads from the office via OpenVPN.
After a few tests we could verify that the maximum speed of the downloads from the office was exactly 3,8 MByte/s.
We tried downloading files from several Windows-Servers in the office and there we got always the maximum possible speed, no matter if it was a physical or virtual machine.
So far we only tested transmissions via SMB-protocol through the OpenVPN-tunnel, as this is the typical use-case in the homeoffice.
Next we tested direct access via portforwardings, downloaded from the Synology Web-Interface (http) and downloaded via SFTP.
Download-rate was always exactly 3,8 MByte/s. So we knew that is wasn't a VPN-problem.One thing I didn't mention so far: up- and downloads in the office between the fileserver and the windows-clients are always at maximum possible speed, about 100 MByte/s.
Our first guess was, that there must be an issue with the Synology, so we contacted the Synology-Support.
They checked the configuration and the logs and told us that everything is fine.Next we tested with two other Synology-devices, a DS2415+ and an old DS214+. Download-speed alway stays at 3,8 MByte/s.
Now we tried a TrueNAS-device. Download-speed was even worse, about 2MByte/s.As VM-server in the office we use unraid and as I said before, the downloads for testing purposes from the Windows-VM's running in unraid are at maximum speed.
But when I download directly from a share in unraid through the Internet, the download-rate is: Yes, 3,8 MByte/s! Sounds crazy to me...To get our local infrastructure out of the equation, we attached a Synology device directly to an interface of the pfsense-box.
Download-rate: 3,8 MByte/s.Now we are running out of ideas.
Before anyone asks: There are no active limiters on the pfsense.I am not absolutely sure, but as far as I remember, a few months ago I got full 10 MByte/s download-speed via OpenVPN from this office.
"Full", because I have only a 100 MBit-Line at my side.So, could this be a bug with the latest pfsense-version?
Any other ideas on that mystery? Why are obviously only Linux-based devices limited?
Any help is highly appreciated! -
How are you testing downloads from Windows? Still smb only? Is it possible they have some sort of TCP accelerator?
I would try testing using iperf against both Windows and Linux and/or BSD.
You might also test an SFTP download from pfSense itself to rule out something LAN side.
A packet capture of the fast and slow downloads should at least show you any dramatic difference in TCP window size for example.
Steve
-
Hi Stephen, thank you for your thoughts about this mystery.
Yes, we tested from the windows machines only with SMB so far, as this is the common usecase in the homeoffice. I don't think, that there is some kind of TCP accelerator on the windows machines, as we tested with several Windows 10 workstations and servers, even with 2 domain controllers, one physical, one virtual. From my experience I would even expect worse transfer rates from the domain controllers.
And in general, I think, it should not be the question, why the windows machines perform as they should, according to the given bandwith. It should be the question, why the Linux-based machines don't.
Nevertheless, as any new input might be helpfull to find out what the reason for this weird behaviour is, I will test with other protocols from the windows machines and do packet captures.
I will also test the direct SFTP download from the pfsense, although we already tested without the LAN, as we directly attached a Synology device to one of the pfsense's LAN-ports.
When I have the results of the tests, I will post them here.
Again, thanks a lot for your input!
-
Are they using the same smb version? smb v3 is significantly better across any sort of latency.
-
Yes, on all SMB-Transmissions SMB v3 is used. I checked again this weekend, just to be sure.
I will do some final tests today or tomorrow and post the results here afterwards.
-
-
@gertjan said in Limited upload-speed for Linux based systems through pfsense:
SMB over VPN =>horror.
especially when used for speed testing
-
Ok, it took a long time, but there was a lot of testing and investigation needed.
First of all - It seems not to be a pfsense-related problem, it was a Synology issue.
Stephen suggested to do a SFTP-test from a Windows-Machine over WAN and to my surprise the download rate was also 3,8 MByte/s.
Afterwards I did the tests with iperf and the results were similar on the Windows- and the Synology-machines.
Download-Bandwith over WAN: 34 MBit/s on a single stream, but with multiple streams I was able to saturate the bandwith.In his first post Stephen made the assumption, that windows has some kind of TCP-accelerator.
Well, it is not a TCP-accelerator, turns out it's a SMB-accelerator. The "accelerator" is the SMB-multichannel-feature.On the Windows clients I could see, that the connections to the Windows machines and to the Synologys are both established via SMB 3.1.1, but only the connections to the Windows machines used multichannel.
So I went on and tried to enable multichannel on the Synologys. I found a lot of instructions how to do that, but they all didn't work for me.
Finally, figured out a combination of parameters to set in the smb.conf that enables SMB multichannel on LAGGS according to RFC 802.3ad.
With SMB multichannel activated, also the Synologys make use of the full bandwith via SMB over OpenVPN.
All Synologys? Unfortunately not.
Turns out, that Synology removed the SMB multichannel support in DSM 7. So I was only able to activate SMB multichannel on the Synologys running DSM 6.
In December 2022 Synology announces SMB multichannel as "new" feature in upcoming DSM 7.2 and started the Beta testing programm: https://community.synology.com/enu/forum/1/post/157265So, my thanks go out to Stephen, who came up with new suggestions, when I was stuck. The tests suggested by him finally led my to the solution.
The only questions, that still remains to me are:
Why is the SMB transfer rate in the LAN even without SMB multichannel from the Linux-based Servers the same as the from the SMB multichannel connected Windows-Servers?
And over WAN, even with a symmetrical 1 GBit-Link, so much worse?
Can anyone explain, why only the SMB multichannel feature gives us over WAN the possibilty to use the full bandwith, which is the same as within the LAN? -
Nice work!
It's probably because of the latency difference. With local transfers the latency is so low you can see the full bandwidth with a single TCP connection. SMB is notoriously latency sensitive. It looks like multichannel works around that to some extent.
Steve