slow pfsense IPSec performance
-
@stephenw10 ok, thanks. this is the output of the iperf test:
iperf3 -B 192.168.201.1 -c 192.168.202.1
Connecting to host 192.168.202.1, port 5201
[ 5] local 192.168.201.1 port 2715 connected to 192.168.202.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 25.5 MBytes 205 Mbits/sec 0 639 KBytes
[ 5] 1.04-2.01 sec 27.0 MBytes 234 Mbits/sec 0 720 KBytes
[ 5] 2.01-3.03 sec 29.7 MBytes 246 Mbits/sec 0 736 KBytes
[ 5] 3.03-4.01 sec 28.2 MBytes 240 Mbits/sec 1 439 KBytes
[ 5] 4.01-5.05 sec 28.2 MBytes 229 Mbits/sec 0 521 KBytes
[ 5] 5.05-6.04 sec 25.2 MBytes 213 Mbits/sec 0 585 KBytes
[ 5] 6.04-7.01 sec 25.8 MBytes 222 Mbits/sec 0 642 KBytes
[ 5] 7.01-8.01 sec 28.4 MBytes 240 Mbits/sec 0 701 KBytes
[ 5] 8.01-9.00 sec 28.0 MBytes 235 Mbits/sec 0 735 KBytes
[ 5] 9.00-10.04 sec 29.1 MBytes 237 Mbits/sec 0 735 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 275 MBytes 230 Mbits/sec 1 sender
[ 5] 0.00-10.13 sec 275 MBytes 228 Mbits/sec receiverPlease note that the interfaces OPT1 are the ones involved in the P2. LAN interfaces are used only to reach and manage the pfsense instances.
-
Ok, so no difference. Do you see any imrovement with more parallel streams?
-P 4Edit: Or actually slightly slower but testing from the firewall itself usually is.
-
@stephenw10 mmmh, no, no improvement, I'm sorry.
It is a big mystery :(
I don't know what I should check, where is my error...or the issue... -
Like identical total throughput?
Really starts to look like some limiting somewhere if so.
You could try an OpenVPN or Wireguard tunnel instead.
-
What is the NIC Type of theses VMs (vmxnet3 or e1000)?
You may want to check your IPsec Advanced Settings:
Ensure AES-NI is enabled and running:
The throughput is definitely to low on your VMs. Even unencrypted traffic, isn't hitting line rate. Assuming the Gigabit link is idling.
I measured on a 4 CPU VM, running on the Xeon E3 Box, around 600Mbit/s IPSec throughput.@stephenw10 said in slow pfsense IPSec performance:
You could try an OpenVPN or Wireguard tunnel instead.
I don't think OpenVPN can outperform IPsec, at least without DCO. Haven't tried that feature yet :)
-
It is surprising with DCO but I would still use IPSec in a situation like this. I only suggested that as a test in case there is something throttling IPSec specifically.
Definitely worth checking async-cypto. Usually that kills throughput almost completely on hardware that isn't compatible though.
Steve
-
@averlon sorry for my late answer, but I have been busy during the last days.
AES-NI is available only on one of the hypervisor, so I can't enable it on both ends.I would like to know the level of impact of NIC types of the VMs.
One VM uses vmxnet3 nic, the other one is using e1000. What do you suggest to do with these nics?Thank you,
Mauro -
Unless you're passing through hardware vmxnet will be faster.
But you must add a tunable to enable mutli-queue on them:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfacesSteve
-
@mauro-tridici said in slow pfsense IPSec performance:
AES-NI is available only on one of the hypervisor, so I can't enable it on both ends.
This is most likely the bottleneck here. You have to ensure, that AES-NI is available on both ends. Otherwise you won't see any higher throughput with 4 vCPUs for you IPsec traffic.
@stephenw10 said in slow pfsense IPSec performance:
Unless you're passing through hardware vmxnet will be faster.
vmxnet will have less overhead, but won't deliver necessary more throughput. I had horrible performance on pfSense 2.4.x / FreeBSD 11.x with vmxnet3. Something between average 600 and 700 Mbit with a high variance for bulk downloads.
I'm still using e1000 on good old pfSense 2.5.1. For Gigabit Link it delivers almost full rate. I just re-tested it from a VM NAT'ed by pfSense on the same ESXi. This is the same old Xeon E3-1245v6 Box
@mauro-tridici: You should test both NIC Types with the current Version and implement the tunable stephenw mentioned. It may improve non encrypted throughput, but won't solve you IPsec issue.
-
@stephenw10 nothing to do, VMXNET + tunable didn't help me. thank you again for you support.
I think I should change from IPSEC to a different lan to lan vpn solution.
Could you please say me the solution you suggest?Thank you in advance,
Mauro -
@averlon thank you for your support. unfortunately, vmxnet and tunable didn't help and I think I have to give up.
non encrypted throughput between the WAN interface of the two pfsense instances is very good. encrypted traffic on IPSEC tunnel is very poor...Is there any other solution to create a lan to lan vpn easily ?
Thank you,
Mauro -
-
@stephenw10 Thank you for your reply.
I just configured OpenVPN, it works, but it is slower than IPSEC...
So, I have a last attempt to do using wireguard, but I read that "WireGuard is available as an experimental add-on package" in pfsense 2.6.0.What do you think about that? Should I give up and buy two new hypervisors with AES-NI enabled in order to use IPSEC and reach the expected performance?
Thanks for your patience,
Mauro -
Hmm, well the first thing I would do it test running an IPSec tunnel between the sites using any other method. So preferably pfSense bare metal at each end. Make sure there isn't something in the route throttling VPN traffic.
This feels more like a virtualization issue though.How slow is the OpenVPN tunnel? How are you testing it? How is it configured?
Steve
-
[ 5] 0.00-10.09 sec 1.01 GBytes 864 Mbits/sec receiver
Plus the TCP/IP overheat on top you get nearly 1 GBit/s.
Getting from the pure "in real life" situation something like
30 % throughput out of this with IPSec is really nice in my eyes. Together with AES-NI you may get some numbers plus
and if QAT is on both ends in game you may get out once more again better numbers for your VPN.What do you think about that? Should I give up and buy two new hypervisors with AES-NI enabled in order to use IPSEC and reach the expected performance?
A small 2nd hand hardware, with Xeon E3-12xxv2/3
will do the job with ease for you. I would not buy fully
new VM host hardware. If money is rarely from eBay it
might be the best point to get hands on.refurbished server for ~165 €
All in one
refurbished server for ~180 €
Plus adding a case, psu and Intel i350 NIC -
@stephenw10 the OpenVPN tunnel bitrate is 120Mbps and I'm using the default configuration mentioned in the link you provided. I also tried to reduced the impact of encryption and authentication mode, but nothing changed.
During the next days I will try to use bare metal pfsense instances. No limitation or something similar is throttling the VPN traffic in the route.
Thank you very much.
-
@dobby_ thank you for sharing your suggestions. During the next days I will try to use two bare metal pfsense instances with AES-NI enabled. Thanks again for your help
-
@mauro-tridici said in slow pfsense IPSec performance:
@dobby_ thank you for sharing your suggestions. During the next days I will try to use two bare metal pfsense instances with AES-NI enabled. Thanks again for your help
If so, and AES-NI is in the game, I would try out IPSec
together with AES-NI using AES-GCM instead of OpenVPN. -
@dobby_ Sure, I will try to apply your suggestions. Should I activate some other option like "Cryptographic Hardware" in addition to your suggested settings?
-
120Mbps should be easily achievable without any crypto hardware. There's definitely something else going on here.
