OpenVPN - Working, but need help diagnosing why upload speed is 6Mb vs 35Mb
-
I've been able to successfully set up OpenVPN on 2.2.4 (latest release) from a standard install. I've created a CA, server certificate, and a couple of users. Users can log in with username/password/Cert without issue. Remote users can download to the protected LAN at full speed (35Mb), but if they try to upload from the protected LAN, it's limited to 6Mb. Setting up OpenVPN was just done using the wizard.
My assumption is the problem is not hardware related. I have a dual core 2GHz Pentium w/ 8GB RAM running pfSense - just an older mini-ITX board I had lying around. It doesn't have AES-NI extensions, and for my WAN port I'm using the onboard Realtek adapter, while LAN ports are using an Intel Pro/1000 PT. Watching CPU I don't even hit 10%, and less so when doing the upload.
Thoughts on some basic items to check? I'd think that copying a file from the protected LAN to the remote user would be just as fast as the download.
-
Upload HOW? (Not to mention, is your line actually symmetric?)
-
Yes, my connection is symmetric (35Mb up/down).
I'm just doing a simple file transfer test after RDPing into a box on the VPN side. In the RDP session I can pull down @ 35Mb, but if I try and upload it's limited to 6Mb.
-
I'm afraid that "simple file transfer test" is a completely useless description… SMB/CIFS, NFS, FTP, HTTP, SCP, ... ??? ??? ???
-
Fair enough, it was an SMB test. I know SMB isn't the fastest protocol, but since I can pull one direction at full speed, I'm assuming I should be able to do the same in reverse?
I'm wondering if there's any reason, other than protocol, that would cause such a slowdown in transfer speed. Wondering if my OpenVPN default config needs to be tweaked.
-
I tried another protocol, this time FTP, and it bounced around quite a bit performance wise, but was still slower when sending data from the protected LAN out to the remote client.
My only thought hardware-wise was the Realtek adapter might be slowing things down? Not sure how big of an issue non-Intel hardware is. My CPU is an older Pentium (socket 1156). Not my first choice in hardware, but I already had it lying around so I figured I'd give it a shot. Maybe the CPU is more of a bottleneck than the CPU Usage dashboard monitor would lead me to believe?
-
SMB over some UDP VPN is a horrible throughput "benchmark". As for CPU, you can easily check with top or something.
-
I'm seeing similar activity over FTP. My CPU hasn't gone over 20% utilization, so I don't believe that's an issue.
So, is there some configuration items that could be affecting throughput? I'm just using the wizard defaults (AES-256 CBC, SHA1, 2048bit DH, TLS Auth, UDP). Based on what I'm seeing, I don't think hardware is my limitation. Is there a better test I could be running to identify where the bottleneck is?
-
So can we get some clarification of how your testing.
So without a vpn session and your machine you say do a speedtest you see 35/35.
What is the speed connection of this remote user? are they also 35/35 or higher? So they vpn into your network via udp or tcp? And then they hit a file share off a server on your network.. And then they download from this server to their machine they see what speeds? How are you checking these speeds, are you using say robocopy to do the smb copy? What flavor of smb? 1, 2, 3, 3.1.1? What are the OSes involved
And then when they upload a file to your server what are the speeds?
Have you done a simple ipferf test between their machine and your server?
-
Certainly - my users are normally seeing 35/35 via speedtest with a direct WAN connection. My goal is to get in that ballpark for remote users connecting to the LAN via OpenVPN. This isn't critical, and I'm completely comfortable if it's a hardware issue and I need to upgrade. Right now I'm just trying to teach myself best practices on configuration and testing.
When remote users do a simple file copy via FTP or and RDP session from their remote server to a server on the LAN, they can write to the OpenVPN protected LAN side at near 35Mbps. SMB should be v2 I believe, since this is using a standard RDP session on Windows 7 & Windows 2008 R2. FTP is done using an FTP server hosted on the LAN side, and a remote user connecting via FTP client (FTPzilla). Writing either via FTP or RDP is coming in ~35Mbps.
The slowness that has me concerned is writing from an OpenVPN protected LAN client out to a remote user. Or, a remote user downloading from the OpenVPN protected LAN. Either way, SMB or FTP, is transferring at <10Mbps. I don't see any issues monitoring CPU, RAM, MBUF on the pfSense box monitoring via Dashboard or top.
So, any data sourcing from the protected LAN out to a remote user is slower than I'd expect. I've not done an iperf test, will do so when I have a moment.
-
Watching CPU I don't even hit 10%, and less so when doing the upload.
Ok this is on your side and what is going on the client then?
Your CPU is reaching something around 10% perhaps, but on the other side the CPU
is perhaps striking sparks out? The Hardware on both ends of a VPN connection should
be able to realize the expected throughput and not only on one side.Either way, SMB or FTP, is transferring at <10Mbps.
And this could not be a hint to the different NICs or their vendors?
Let us imagine you get a so called server grade Intel 4 Port NIC and try it out
once more again, then you could easily saying it is not related to the NICs for sure.
But not now with a Realtek and an Intel based one.I don't see any issues monitoring CPU, RAM, MBUF on the pfSense box monitoring via Dashboard or top.
And on the other VPN ends?
So, any data sourcing from the protected LAN out to a remote user is slower than I'd expect.
And wehat kind of hardware this remote user is using in this "game".
I've not done an iperf test, will do so when I have a moment.
Would be the best for you and us, but please from both sides!!! It makes no sense
to know that your pfSense machine is capable of 1000 OpenVPN tunnels and the
other end is creeping down by doing the iPerf test.May be it sounds strength a little bit to you, but only knowing one of two or more sides
and finding nothing out then will be also not reaching the goal. -
I've run iperf using this command: "iperf3 -c 10.10.10.15 -V" on the remote user side, and I have "iperf3 -s" running on the protected LAN side. In this case, both users are running Windows 7. I'm not seeing any issues CPU wise on either machine. Both have 8GB RAM and two CPUs.
On the client side, which is the remote user, I'm seeing a summary of 33.4 Mbits/sec on both sender and receiver. If I reverse the command above (iperf3 -c 10.10.10.15 -V -R), then I actually see nearly the same results.
One question - should I be running this with the "-u" flag since I'm using OpenVPN via UDP? I realize the test above is for TCP. Based on what I'm seeing from the results, it appears my speed is actually 35Mbps both ways? If that's true, then that certainly means the lack of speed I was seeing before was 100% not the problem of OpenVPN or pfSense.
Am I missing anything?
-
The outer part of the VPN's transport protocol (TCP or UDP) doesn't matter for what you're testing with inside the VPN. It's best in that circumstance to use TCP with iperf.
That shows the VPN's performing exactly as it should. SMB performs horribly as latency increases, so it's probably to blame.
-
Now smb 3.1.1 has made some adjustments that should make work better over a wan.. But yeah smb in general is very chatty and high latency doesn't help
What is your ping times when your remote users ping the server?
Prob better to use http to xfer files over wan with any latency at all. Doesn't matter if vpn or point to point link in a company, etc.. SMB signing could account for differences in read vs write as well..
-
Yeah if you can use only SMBv3, it shouldn't be as bad as earlier versions.
-
Appreciate all the help and feedback!
Tested ping times, and I'm looking at a range of 70-110ms when just doing a simple ping from the command line. That doesn't seem exceptionally high, but that could be enough to impact SMB? I expect that some users will use SMB transfers because it's simple when working over an RDP session. I do have an externally available FTP that can be used as well, which is where I'll direct them to go for larger transfers.
-
70-110 with SMB is horrific.. SMB is fine when your on a lan and your taking 1ms – you put it over a wan and the chattiness can be terrible on performance..
Why don't you take a little look see on a file copy with SMB.. Now
Again when your working over RDP?? Doing a copy inside your RDP session back to your local disk via that tsclient mapping is not the same as smb copy.. I am almost positive that is just a rdpclip method of moving the file.. Yeah that is not going to be fast either.
See the very small length of each request and then now multiple those by 70ms vs 1ms for large files, etc.. SMB over WAN is going to be slow ;)
-
Makes sense - and you're correct, I was using rdpclip to test, but I assumed it was SMB. Very interesting, I hadn't dove into the details of latency and the impacts on file transfers. Typically most of my work is done on a LAN.
All this is great info, thank you for sharing! I've set up VPNs before using pfSense, but primarily for my own personal use. This is new territory for me on setting one up using pfSense in a more professional manner. Functionally, I'm finding the setup much better to use versus what I had before on my router (running Tomato). Very impressive how much you can do, even on old hardware.
Questions:
-
Right now I'm using an old Pentium CPU, Realtek NIC for WAN (Intel PT for LAN), and 8GB RAM. If I were to switch to an Intel NIC for the WAN would that improve performance or latency?
-
Would a chip with AES-NI extensions improve my VPN performance or latency? I'm using AES-256 CBC as my encryption algorithm under OpenVPN, but don't currently see any CPU activity > 20% as shown on the dashboard during transfers.
-
-
switch to an Intel NIC for the WAN would that improve performance or latency?
It can be, but it is not a must be!
) Would a chip with AES-NI extensions improve my VPN performance or latency?
It weould perhaps make sense to insert a Soekris vpn 1411 or vpn 1401 miniPCI/PCI adapter but
on so old hardware, sorry, but on so old hardware I want more have a looking eye on newer
hardware that is able to delvers other numbers. Or if you will go on with this hardware, to
upgrade them only.stronger cpu or latest available for your socket
more ram
SATA or IDE ssd
Intel NIC for WAN
vpn adapter -
As to changing a nic for latency? Over a wan with 70-110 currently?? No sorry that is not going to make any sort of difference what so ever..
As you saw with your iperf test your seeing your wire speed… Now if you were sucking up large amounts of cpu when you did that, ok AES-NI could help.. None of that stuff is going to fix LATENCY.. None of that stuff has anything to do with the fact that older version of smb are chatty as shit and suck over wan.. Than your inside a vpn is not your issue. 70+ ms is your problem in moving files with smb.. Or any other protocol that not well suited for wan.. You want to move files over high latency wan, you need multiple streams you need large receive window, etc. etc.
Do the math.. lets see with 1 stream and 110ms using the default window size of what 64KBytes best you could do is about 4.8 Mbps..If you wanted to MAX out your 35 mbps you would need a window size of 470KBytes.. Or you need more streams!
