Slow VPN performance Wireguard & OpenVPN on Netgate 6100
-
Hi,
I have a remote Netgate 6100 running the latest pfsense 24.03. I have been running OpenVPN (split-tunnel) on this Netgate for a while, and seem to be getting speeds around 10-15MB/s using iperf3 through the VPN to my home.The internet speeds at both locations are (checked against an online speedtest):
- Remote location (running the Netgate and VPN servers): 3Gbit/s download, 2.6Gbit/s upload
- Home location: 400 Mbit/s download, 50 Mbit/s upload
I was hoping to get (significantly) higher speeds using wireguard, so I set up a wireguard server (split-tunnel) on the Netgate (following the instructions from this video).
I've done some iperf3 speedtests (from my home Win10 to a Ubuntu VM behind the remote Netgate) for OpenVPN and Wireguard. The performance remains rather low unfortunately and interestingly is pretty much unchanged when trying different crypto settings (see list below):
Wireguard speeds typically appx.:
- 1 iperf connection: 7.8 MB/s
- 5 iperf connections: 8.2 MB/s total
- 10 iperf connections: 9.2 MB/s total
OpenVPN speeds typically appx.:
- 1 iperf connection: 10.9 MB/s
- 5 iperf connections: 11.1 MB/s total
- 10 iperf connections: 13.0 MB/s total
Tested some different settings, which have not significantly impacted the above speed results:
- Tried turning off OpenVPN server (only running wireguard)
- Turned off AES-NI CPU crypto and turned off QAT Crypto. Only IPsec-MB still on.
- Turned ON QAT crypto again (IPsec-MB also still on, AES-NI crypto turned off)
- Turned ON AES-NI crypto (IPsec-MB also still on, QAT crypto turned off)
- Turned OFF IPsec-MB (AES-NI still ON, QAT crypto is OFF)
- Disable hardware checksum offload
- Enable hardware checksum offload (and AES-NI and IPsec-MB and QAT turned on)
I expected significantly higher speeds for VPN (especially using Wireguard). Does anyone have any ideas what the problem could be? Thanks for any feedback you may have!
Side notes: Before updating to the latest pfsense, I was running 23.09.1-RELEASE and that showed in the OpenVPN log every 30 sec "Note: OpenSSL hardware crypto engine functionality is not available". After updating to the latest pfsense version this does no longer show up though. Also I noticed that in the previous version it said "AES-NI CPU Crypto: Yes (inactive)". I've now changed this to be active though. OpenVPN is using TCP with port-sharing with a webserver on port 443. Sometimes during iperf3 testing I can see occasional short "spikes" of up to 25MB/s for OpenVPN.
-
I installed a standalone wireguard server on a VM behind the Netgate, to see if that would show any difference in performance. Interesting result: I am getting the exact same slow speed. So I guess that means this is not a netgate (crypto) CPU limit but something else on my network is bottlenecking the VPN traffic..
Any ideas?
-
I did not see it in your list for WG. Adjust the MTU on the tunnel. 7100 <-> 6100 with an MTU of 1420 I am touching 800Mbps on a 10ms latency link.
1420 may not be the best, but it is what we settled on when building the tunnel. -
@AndyRH thanks for the hint, definitely worth trying.
Stupid question, but following the above linked video instructions from Tom Lauwrence there is no Wireguard interface in the list in pfsense (even though the wireguard server runs). I can manually add this interface though, see screenshot below.
Do these fields look good? What do I need to put under IPv4 address/ in this case? Thanks!
-
We left MSS blank.
I am not at the system, but I think the IP address is the gateway on the other side of the tunnel. I could be remembering wrong, it could be the local gateway to the other side. -
@AndyRH I think I am a bit stuck here then at this point because I am not doing a site to site wireguard tunnel. I just have the wireguard "server" set up on this netgate and connect with clients to it. Following the official netgate docs for setting up wireguard/the video I shared above, makes the wireguard server work but it does not create the wireguard interface.
So if anyone could point me at what exact settings to tweak where on my pfsense to set the MTU to a lower value that would be really useful (also for OpenVPN if possible). Also wondering if i.e. I need to change the MTU on all other interfaces for WAN/LAN/BRIDGE too?
-
Okay, closing in on the issue.. Maxing out the 400/50 Mbit wire speed with the Wireguard tunnels using a different machine at home. Getting 220 Mbit/50 Mbit on OpenVPN at home on that other machine. Really only remaining points to troubleshoot why I get only ~80Mbit for wireguard/OpenVPN (but 400/50Mbit internet..) on my Win10 machine at home would be something in that OS itself/router.. Didn't quite expect this issue to be on that end!
Out of interest I've been playing with changing some of the crypto settings in the first post to see if I could get the OpenVPN speed even above the 220 MBit/s I get now, but it didn't seem to make much of a difference. I guess 220 MBit/t is the max it can do..?
-
@nvs are both computers using wired connections? Wireless throws in a bunch of variables - AP, chipsets, drivers. Such as one computer might support 802.11ac and the other 802.11ax .
How far is distance between clients and server? Latency can affect performance. Can try https://speed.cloudflare.com/
RE: OpenVPN - OpenVPN is single threaded process. Look to see if you have 1 CPU at 100%
-
Good news. I've managed to figure out my major bottleneck. My main Win10 machine at home has a stu*** Intel I225-V 2.5 Gbit on board ethernet NIC that I was using. It has caused weird issues in the past with random disconnects and is globally known for its issues.. I have plugged in a ASUS USB 2.5 Gbit adapter and sure enough I am able to max out my wire speed now using the wireguard tunnel.
OpenVPN also works equally well now as on the other machine, and I figured out that enabling DCO in the OpenVPN client side can bring me close to peak speeds of 400Mbit wire speed. I haven't enabled DCO on the OpenVPN server side yet. I am still using OpenVPN via TCP, and I read for DCO thats not supported/recommended - even though interestingly enough it already gives me a performance boost already now just enabling it on the client side. Switching OpenVPN server to using UDP and enabling DCO on the server side as well might further improve things I imagine - something I might try in the next days.
Hope this thread will help somebody down the road troubleshooting the same.
@ahking19 Thanks for your reply. Both machines are on wired ethernet on the same router at home. I think I am not near maxing out the Netgate CPU, but I've attached a screenshot below to be sure (not sure how to interpret that plot exactly):
Latency wise: I have about 9-15 ms ping through the OpenVPN tunnel from client to my netgate.
Using wireguard is about 9-11ms.
