slow pfsense IPSec performance
-
@cool_corona Should I set these values on OPT1 and WAN interfaces disabling MSS clamping for VPN traffic in Sys > Adv > Firewall & NAT ?
-
@mauro-tridici I’m guessing the reason is virtualisation.
Virtualisation has a hefty penalty cost in bandwidth when it comes to such things as interrupt based packet handling in a guest VM. This is due to the hypervisor context switching taking place several times across the hypervisor and guest VM for every packet interrupt that needs to be handled.
If the traffic does not scale properly across CPU cores and is “serialised” on a single core, then throughput will be very low, and yet there will be no real CPU utilisation because all the time is spent waiting on context switches for interrupt handling rather than actually processing traffic payload.Love the no fuss of using the official appliances :-)
-
Setting MSS clamping there will apply it to all traffic that matches a defined P2 subnet. On any interface. That should be sufficient.
1ms between the end points seems unexpectedly low! Is this a test setup with local VMs?
Steve
-
@mauro-tridici Yes. Report back.
-
@keyser Just not true. The performance penalty is minimal compared to bare metal.
-
@cool_corona said in slow pfsense IPSec performance:
@keyser Just not true. The performance penalty is minimal compared to bare metal.
Ehh no, depends very much on what hardware you have (and it’s level of virtualization assist). But if it’s reasonably new x86-64 and above Atom level Level CPU, then yes, that will see the cost of virtualisation dvindle. But if we are taking Atom level Jxxxx series CPUs, then those speeds are very much at the very limit of the hardware when doing virtualisation.
-
@keyser Nobody uses Atoms for Virtualization.....
-
@stephenw10 thank you for the clarification.
I will try to describe the scenario-
we have two different VMware ESXi hypervisors (A and B) in two different sites
-
on the ESXI_A has been deployed pfsense endpoint PF_A
-
on the ESX_B has been deployed pfsense endpoint PF_B
-
PF_A and PF_B are geographically connected using a 1Gb network link and WAN-to-WAN iperf test is good (about 900Mbps)
-
HOST_1 is a physical server on a LAN behind the PF_A
-
HOST_2 is a test virtual machine deployed on ESX_B on a LAN behind the PF_B (so, HOST_2 and PF_B are on the same hypervisor and they are "virtually" connected, there is no a LAN cable between HOST_2 and PF_B since they are on the same hypervisor)
When I try to run iperf2 between HOST1 and HOST2 I obtain only 240/290 Mbps
I hope it helps.
-
-
@cool_corona unfortunately nothing changed, thank you.
-
@mauro-tridici Your ISP is not throttling VPN's??
-
Mmm, 1ms between then is like in the same data center. Or at least geographically very close. What is the route between them?
When you ran the test outside the tunnel, how was that done? Still between the two ESXi hosts?
-
@cool_corona said in slow pfsense IPSec performance:
Nobody uses Atoms for Virtualization.....
Ha. Assume nothing!
-
@cool_corona my ISP is not throttling the VPNs, we already use several VPNs (host to LAN) without any problem (iperf test bitrate is optimal). We are experiencing this low bitrate only with IPSEC LAN2LAN VPN
-
@stephenw10 we have 2 data centres in the same city, they are interconnected with a dedicated 1Gb link on the GARR network.
the test outside the tunnel is between the WAN interfaces of the two pfsense instances: that is between PF_A and PF_B -
@stephenw10 our hypervisors have "2 x Intel(R) Xeon(R) Gold 5218 CPU - 32 cores @ 2.30GHz"
-
Hmm, that should be plenty fast enough. What happens if you test across the tunnel between the two pfSense instances directly? So set the source IP on the client to be in the P2.
-
@mauro-tridici said in slow pfsense IPSec performance:
@stephenw10 our hypervisors have "2 x Intel(R) Xeon(R) Gold 5218 CPU - 32 cores @ 2.30GHz"
Then it’s definitely not hardware that is limiting the transferspeed. Those CPU’s/platforms have loads of power for this usecase.
-
@stephenw10 Sorry, I didn't understand the test I should do?
Should I do an iperf or a ping test between PF_A[opt1] and PF_B[opt2]?PF_A has
WAN IP: xxxxxxxx
LAN IP (for management only): 192.168.240.11
OPT1 IP: 192.168.202.1PF_B has
WAN IP: yyyyyyyy
LAN IP (for management only): 192.168.220.123
OPT1 IP: 192.168.201.1 -
Normally you run iperf3 server on one pfSense box then run iperf3 client on the other one and give it the WAN address of the first one to connect to.
But to test over the VPN the traffic has to match the defined P2 policy so at the client end you need to set the Bind address to, say, the LAN IP and then point it at the LAN IP of the server end.
Then you are testing directly across the tunnel without going through any internal interfaces that might be throttling.
So run
iperf3 -son the PF_A as normal.
Then on PF_B runiperf3 -c -B 192.168.220.123 192.168.240.11 -
@stephenw10 ok, thanks. this is the output of the iperf test:
iperf3 -B 192.168.201.1 -c 192.168.202.1
Connecting to host 192.168.202.1, port 5201
[ 5] local 192.168.201.1 port 2715 connected to 192.168.202.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 25.5 MBytes 205 Mbits/sec 0 639 KBytes
[ 5] 1.04-2.01 sec 27.0 MBytes 234 Mbits/sec 0 720 KBytes
[ 5] 2.01-3.03 sec 29.7 MBytes 246 Mbits/sec 0 736 KBytes
[ 5] 3.03-4.01 sec 28.2 MBytes 240 Mbits/sec 1 439 KBytes
[ 5] 4.01-5.05 sec 28.2 MBytes 229 Mbits/sec 0 521 KBytes
[ 5] 5.05-6.04 sec 25.2 MBytes 213 Mbits/sec 0 585 KBytes
[ 5] 6.04-7.01 sec 25.8 MBytes 222 Mbits/sec 0 642 KBytes
[ 5] 7.01-8.01 sec 28.4 MBytes 240 Mbits/sec 0 701 KBytes
[ 5] 8.01-9.00 sec 28.0 MBytes 235 Mbits/sec 0 735 KBytes
[ 5] 9.00-10.04 sec 29.1 MBytes 237 Mbits/sec 0 735 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 275 MBytes 230 Mbits/sec 1 sender
[ 5] 0.00-10.13 sec 275 MBytes 228 Mbits/sec receiverPlease note that the interfaces OPT1 are the ones involved in the P2. LAN interfaces are used only to reach and manage the pfsense instances.
