Help: Strange performance issue with Wireguard
-
I have setup wireguard with BulletVPN but I'm getting very strange latency/performance issue with my pfsense installation.
My internet:
Fiber 1 gigabit up/downMy PC:
AMD 5900x with Intel I211 Gigabit ethernet adapter running Windows 11.When connected to BulletVPN's wireguard endpoint on Windows 11 directly, I got:
50~60ms ping time and
average throughput around 500 mbpsWhen connected to the SAME exact wireguard endpoint on pfsense, I got:
300~310ms ping time and
average throughput around 80 mbps max?!?!?!?!?My pfsense machine spec:
Intel i5 6260u (dual core at 1.8 GHz).I disabled Suricata and I monitored my CPU usage using top while running speed tests and I only see a spike of 6~7% CPU usage, so it's not due to Suricata or a CPU bottleneck (at least the evidence is not there).
I tried everything (including adjusting different MTU/MSS sizes) and I can't for the life of me figure out why my pfsense wireguard is SO MUCH slower... I originally thought was the VPN provider's fault but after I did a direct connection from Windows 11 to the same EXACT endpoint I got MUCH better performance so it's NOT their fault... there's something NOT right with pfsense's wireguard... My plan was to use wireguard to achieve as close to my native fiber speed (since OpenVPN max out around 250 mbps), but now with this HUGE latency and bandwidth issue, I'm super disappointed with wireguard on pfsense...
on Windows 11:
Tracing route to www.google.com [142.250.64.132]
over a maximum of 30 hops:1 58 ms 58 ms 58 ms 10.0.0.1
2 59 ms 61 ms 59 ms 45.58.126.1
3 66 ms 59 ms 60 ms ae0-1983.cr8-mia1.ip4.gtt.net [76.74.95.137]
4 71 ms 63 ms 76 ms ae34.cr6-mia1.ip4.gtt.net [213.200.113.142]
5 59 ms 59 ms 61 ms 72.14.210.140
6 60 ms 61 ms 61 ms 108.170.253.1
7 60 ms 60 ms 61 ms 209.85.241.235
8 61 ms 61 ms 61 ms mia09s21-in-f4.1e100.net [142.250.64.132]On PFsense:
Tracing route to mia09s21-in-f4.1e100.net [142.250.64.132]
over a maximum of 30 hops:1 301 ms 301 ms 299 ms 10.100.0.1
2 300 ms 302 ms 301 ms 45.58.126.1
3 310 ms 301 ms 306 ms ae0-1983.cr8-mia1.ip4.gtt.net [76.74.95.137]
4 299 ms 301 ms 303 ms ae34.cr6-mia1.ip4.gtt.net [213.200.113.142]
5 321 ms 300 ms 302 ms 72.14.210.140
6 302 ms 303 ms 302 ms 108.170.253.1
7 304 ms 304 ms 302 ms 209.85.241.235
8 302 ms 301 ms 302 ms mia09s21-in-f4.1e100.net [142.250.64.132] -
@faux123
Did you enable DCO for OpenVPN. That should give you better throughput than wireguard.That being said, i have found wireguard on pfsense to not be very performant in general. Although to be honest i have seen at least a 10% reduction in speeds but what you are describing seems more of a local issue to pfsense. The fact that both OpenVPN and WG, entirely different protocols, suffer poor performance points to a device issue.
My first step would be to re-install the WG package. -
@michmoor Thanks for the suggestion. I just uninstalled and reinstalled WG, rebuilt all the tunnels again.. still getting the 300 ms ping time and poor overall bandwidth around 70~80 mbps
-
@faux123 No problems here. Tested with my own VPS.
So I guess it has something to do how you made the tunnel work with pfSense? -
@bob-dig There's definitely something not quite right with pfsense... so after uninstall/reinstall process, I did a cold reboot and checked again it didn't change anything still at 300 ms ping...
But then I decided to reboot it a couple more times WITHOUT changing anything.. Now my ping has come down to around 120 ms which is 2x the Windows 11 ping to the same exact endpoint... This is still quite baffling to me why without changing anything just by rebooting would affect Wireguard behavior....
I will continue to investigate on my own a bit more as to why behaviors are so different to the same exact endpoint on wireguard.
-
@faux123 its probably more helpful to say there is an issue with your possible hardware or software set up. Pfsense operates on variety of machines and there are typically no issues involved. I have ran it on Protectli hardware and now running on Netgate hardware. Absolutely no issues aside from a misconfiguration.
As i mentioned before, the fact that you have poor performance on two different VPN protocols speaks to something other than pfsense here.
What NIC drivers are you using?
-
@michmoor My pfsense is an Intel Nuc with i5-6260u processor. It has a built-in Intel Gigabit LAN and I added an M.2->Gigabit ethernet adapter so I can have dual LAN controller. The added LAN is based on RealTek chipset. I'm aware of realtek issues historically, but I have the latest 1.96.04 driver installed and I haven't had any issues with it (no drop outs, running Suricata DPS with no loss in performance). I routinely get close to my maximum speed (1 gbps) testing with speed tests from different sites...
-
@faux123 What is performance like when just using the internet without tunneling all your traffic to a 3rd party provider?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/low-throughput.html#insufficient-hardware
-
@michmoor with direct connection, I can get close to the theoretical 1 gbps on my fiber link up and down.
BTW, I have SOLVED my problems!
After posting here, I decided to SWAP between my WAN (intel) and LAN (realtek) adapters... Now I have WAN with Realtek and LAN with Intel and this apparently FIXED the high latency and bandwidth issue at the SAME time!
It seems that WG is a kernel implementation, it must have some sort of direct interaction with the underlying kernel ethernet drivers... There exist some compatibility issues with WG Kernel driver and Realtek driver where latency and bandwidth were severely affected. Now with Intel driver and WG talking to each other (since WG is on the LAN side), everything is working as expected. I'm now getting the same 60ms ish pings with Pfsense and around 500 mbps bandwidth same as direct Windows 11 WG tunnel!
TL;DR:
If you run WIREGUARD, make sure it is paired with the INTEL ETHERNET Driver to NOT have funny latency and bandwidth issues!
-
@faux123 Why do you think it is paired with the LAN Adapter? WireGuard is most probably running on every interface. I have no doubt, that your problem is somehow related to your NICs, that seems to be a given.
-
@bob-dig agreed. there is some configuration here thats not good.
-
@bob-dig the reason I suspect the Realtek LAN driver is tied to WG is due to the fact I didn't change ANY settings other than swapping the interfaces between LAN and WAN. Wireguard tunnels, routing and firewall rules all remained exactly the same as before (when I changed things, I always tried to change 1 thing at a time so I know what each change's effects were).
Due to this experience, I just bought another M.2->LAN adapter with Intel chipset this time for my Intel Nuc. I will swap my Realtek adapter (currently running as WAN) and try a few more experiments to see if it had something to do with the m.2->LAN issue or not.
