Poor Client Performance Across Wireguard Site-to-Site Tunnel
-
I've got a 7100-1U router at two sites, both with 1000/1000 internet, and a wireguard tunnel between the two sites. When running iperf server on one router and iperf client on the other, it usually peaks around 500Mbps, give or take. However, when attempting to run iperf from a client on the LAN across the tunnel, performance is limited to roughly 115Mbps.
I've used iperf between the two routers and from servers on both sides to the routers. I've also confirmed that this affects "real-world" traffic between sites as well, which is why I'm looking at this to begin with. I'm thinking it may have something to do with WG using MTU of 1420 while the LAN has MTU of 1500.
Does anyone have any input, ideas, suggestions?
-
@wblanton I almost get my line-speed, to a weak vps.
Windows Pro > pfSense >>WG>> Windows Server
C:\>iperf3.exe -c vps.home.arpa -P 4 -R Connecting to host vps.home.arpa, port 5201 Reverse mode, remote host vps.home.arpa is sending - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 70.2 MBytes 58.9 Mbits/sec sender [ 4] 0.00-10.00 sec 70.2 MBytes 58.9 Mbits/sec receiver [ 6] 0.00-10.00 sec 77.5 MBytes 65.0 Mbits/sec sender [ 6] 0.00-10.00 sec 77.4 MBytes 64.9 Mbits/sec receiver [ 8] 0.00-10.00 sec 64.0 MBytes 53.7 Mbits/sec sender [ 8] 0.00-10.00 sec 63.9 MBytes 53.6 Mbits/sec receiver [ 10] 0.00-10.00 sec 64.4 MBytes 54.0 Mbits/sec sender [ 10] 0.00-10.00 sec 64.4 MBytes 54.0 Mbits/sec receiver [SUM] 0.00-10.00 sec 276 MBytes 232 Mbits/sec sender [SUM] 0.00-10.00 sec 276 MBytes 231 Mbits/sec receiver iperf Done.pfSense >>WG>> Windows Server
Shell Output - iperf3 -c vps.home.arpa -P 4 -R Connecting to host vps.home.arpa, port 5201 Reverse mode, remote host vps.home.arpa is sending - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 104 MBytes 87.1 Mbits/sec sender [ 5] 0.00-10.00 sec 104 MBytes 86.9 Mbits/sec receiver [ 7] 0.00-10.00 sec 41.2 MBytes 34.6 Mbits/sec sender [ 7] 0.00-10.00 sec 41.0 MBytes 34.4 Mbits/sec receiver [ 11] 0.00-10.00 sec 77.6 MBytes 65.1 Mbits/sec sender [ 11] 0.00-10.00 sec 77.5 MBytes 65.0 Mbits/sec receiver [ 20] 0.00-10.00 sec 47.5 MBytes 39.8 Mbits/sec sender [ 20] 0.00-10.00 sec 47.3 MBytes 39.7 Mbits/sec receiver [SUM] 0.00-10.00 sec 270 MBytes 227 Mbits/sec sender [SUM] 0.00-10.00 sec 269 MBytes 226 Mbits/sec receiver iperf Done. -
@bob-dig So I did notice later that running parallel tests on the clients seems to help, but I didn't need to run parallel tests on the pfSense client. But I did try to run 4 threads like you have, and I get about ~236 Mbps over to the pfsense on the other side. But when I run it in reverse (-R), I'm it's peaking at over 400Mbps.
Unfortunately, the program that that I need to sync over the link appears to run on only a single thread. When the same process was running over a L2 500Mbps point-to-point, the entire synchronization process completed in about 2 minutes. Over this VPN tunnel, it's taking around 8minutes. I really would like to be able to optimize this to get more bandwidth on a single thread.
-
Also, I've noticed that this doesn't seem to be limited to WireGuard, as I'm seeing similar performance with IPSec.
-
I have now enabled QAT on both side. When using a linux client, the throughput is incredible, hitting over 600Mbps at times. However, the windows clients are still back at ~110Mbps.
-
Update - Running iperf3 on Windows and setting the "-w" flag to "1m" gets me closer to ~450Mbs. Now I've got to figure out how to get windows to do that by default...
