2.2.4 IPSec VPN Very Slow…
-
Howdy! I'm having a heck of a time with an IPSec VPN I've setup between a couple of our offices. I'm running pfSense 2.2.4 on both sides and the connection is up and stable. Everything seems to be working, its just very slow. Internet at the main site is 75/50 and the offsite is 50/10, both through the same ISP. In testing I get barely 600K of throughput pulling from the main to offsite. I've tried various MTU and MSS Clamping settings to no avail.
Any suggestions as to how to speed this thing up a bit? Thanks!!
-
Would you please tell us more about your hardware?
A free PCI or miniPCI slot perhaps on both sites?
AES acceleration hardware or AES-NI supported CPU?
Non Intel but cheap consumer NICs or 100 MBit/s cards?
A modem Duplex miss match between the modem and the pfSense so that
the modem is connected only with 10 MBit/s?There are some options that could match also but there for we need more information about your hardware.
-
Sure thing, both are built using older PCs. Here are some details:
RAM and CPU don't appear to be under much load when VPN is used.
The CPUs are older so I don't expect they support AES acceleration AES-NI.
The NICs aren't anything fancy but the connection speeds and throughput outside the VPN are consistent with the Internet packages at each site according to SpeedTest.net.
Units are compact computers so they don't have any free slots.Main Office
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz, 2 CPUs: 1 package(s) x 1 core(s) x 2 HTT threads
RAM: 512 Megs
WAN Connection is 100baseTX <full-duplex>LAN Connection is 1000baseT <full-duplex>Satellite Office
CPU: Intel(R) Pentium(R) 4 CPU 2.26GHz
RAM 512 Megs
WAN Connection is 1000baseT <full-duplex,master>LAN Connection is 1000baseT <full-duplex>VPN is an IPSec with these settings:
Key Exchange version: V1
Authentication Method: Mutual PSK
Negotiation Mode: Main
Encryption algorithm: AES 256
Hash algorithm: SHA1
DH key group: 2
NAT Traversal: AutoPhase 2:
Protocol: ESP
Encryption algorithms: AES Auto
Hash algorithms: SHA1
PFS key group: OffI do have two Phase 2 entries if that matters, both networks are accessible.</full-duplex></full-duplex,master></full-duplex></full-duplex>
-
Units are compact computers so they don't have any free slots.
Really sad to hear this, there are some miniPCI and PCI vpn crypto accelerators
on the market that would be up gaining the entire throughput a really good bit.Perhaps you would be stich in more then 2 GB of RAM that the ACPI mode will be
used. And on the other hand enables PowerD on both machines, would be really
a point that could match to your case. -
That hardware's fast enough that a crypto accelerator isn't necessary for that kind of connection speed.
What's the latency like between the sites (what are the ping times)? How are you testing throughput?
-
Latency is very low, ping times are around 18 ms both inside and outside. To test, I've been trying to copy a 1.5 GB file from a server at the office to the other site. I fired up an older copy of QCheck from Ixia and it reports around 20 Mbps of throughput. But if that's true, I feel like I should be getting way more from a file copy and application speed then I am. Right?
-
To test, I've been trying to copy a 1.5 GB file from a server at the office to the other site. I fired up an older copy of QCheck from Ixia and it reports around 20 Mbps of throughput. But if that's true, I feel like I should be getting way more from a file copy and application speed then I am. Right?
Are you using Windows copy/paste, or another method like FTP? At $DAYJOB we have many locations tied together using VPN over WAN connections of 50/50, 100/100, or higher. Windows CIFS is a chatty protocol and performance suffers when latency goes above the 3-5 ms range. In our private VPN WAN, latency is in the 3-24 ms range, depending on location.
Offices with under ~8 ms latency, we get Windows copy/paste in the 15-20 Mbps range. As we get toward the higher end at 20+ milliseconds, we often get single-digit Mbps.
But if I do an FTP transfer, I get much closer to actual link speed when using an FTP client that can do multi-threaded multi-segment transfers (multiple simultaneous connections, each carrying a segment of the file).
-
Roll back to 2.2.2. CMB walked me through testing a few things, the only thing that worked was rolling back.
-
2.2.4 should be much better than 2.2.2.
-
I can demonstrate going from 2.2.2 to 2.2.4 using pfSense store purchased sg-4860 tunneled to a Palo Alto Networks 3000 series that ipsec throughput is roughly 8 times slower.
The same transition on home built pfSense firewalls does not have the same problem. Either there is something wrong with the sg-4860, or 2.2.4 needs something specific for the sg platform config.
I'd be happy to demonstrate off the forums to anyone willing to do so.
-
Either there is something wrong with the sg-4860, or 2.2.4 needs something specific for the sg platform config.
or there is something wrong with your setup, or the AES-NI module is still broken for non AES-GCM modes, or…
There is a lot of work on IPSec and AES-NI that is about to roll into pfSense (we've been fixing things in FreeBSD-land).
-
Either there is something wrong with the sg-4860, or 2.2.4 needs something specific for the sg platform config.
Did you re-install pfSense on the SG-4860 unit by using the ADI image or did you try out
another one from the public download folders? ::) -
Both, in that order.
-
Both, in that order.
This would be not really good as I see it. The ADI Image comes with all kind of tuning
and specifics that this must be not done by yours. So I would prefer the ADI image over the normal
other community versions. -
Roll back to 2.2.2.
No, don't do that. There does seem to be something to your other thread that's replicable and I'm looking into it, but in general that's not necessary.
Are you using Windows copy/paste, or another method like FTP? At $DAYJOB we have many locations tied together using VPN over WAN connections of 50/50, 100/100, or higher. Windows CIFS is a chatty protocol and performance suffers when latency goes above the 3-5 ms range. In our private VPN WAN, latency is in the 3-24 ms range, depending on location.
Offices with under ~8 ms latency, we get Windows copy/paste in the 15-20 Mbps range. As we get toward the higher end at 20+ milliseconds, we often get single-digit Mbps.
But if I do an FTP transfer, I get much closer to actual link speed when using an FTP client that can do multi-threaded multi-segment transfers (multiple simultaneous connections, each carrying a segment of the file).
Yes, this is exactly my guess. The connection is definitely capable of more as other tests have shown. Verify other types of transfers beyond the old Qcheck which may not be all that reliable on current systems, anything other than Windows file sharing pretty much. HTTP or FTP would be good choices.
-
After reading this post I did some testing myself to check the performance of IPsec.
Dedicated 1Gbit switch on WAN interface.
Two fedora laptops with 1gbit ethernet is connected to the switch. Both laptops is connected as client-to-site into an virtual ip pool.
Using pfsense sg-2440 hardware with AES-GCM128 with AES-NI enabled.
Both version 2.2.3 and 2.2.4 tested with pretty much the same result.
Test results outside IPsec tunnel.
[root@localhost bth]# iperf -c 192.168.1.52 -P 5
–----------------------------------------------------------
Client connecting to 192.168.1.52, TCP port 5001
TCP window size: 85.0 KByte (default)[ 7] local 192.168.1.51 port 48584 connected with 192.168.1.52 port 5001
[ 3] local 192.168.1.51 port 48580 connected with 192.168.1.52 port 5001
[ 4] local 192.168.1.51 port 48581 connected with 192.168.1.52 port 5001
[ 5] local 192.168.1.51 port 48582 connected with 192.168.1.52 port 5001
[ 6] local 192.168.1.51 port 48583 connected with 192.168.1.52 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 278 MBytes 233 Mbits/sec
[ 7] 0.0-10.0 sec 201 MBytes 168 Mbits/sec
[ 6] 0.0-10.0 sec 187 MBytes 156 Mbits/sec
[ 3] 0.0-10.0 sec 223 MBytes 186 Mbits/sec
[ 4] 0.0-10.0 sec 195 MBytes 163 Mbits/sec
[SUM] 0.0-10.0 sec 1.06 GBytes 904 Mbits/secTest result inside VPN tunnel
[root@localhost bth]# iperf -c 10.75.0.4 -P 5
–----------------------------------------------------------
Client connecting to 10.75.0.4, TCP port 5001
TCP window size: 85.0 KByte (default)[ 7] local 10.75.0.3 port 36632 connected with 10.75.0.4 port 5001
[ 3] local 10.75.0.3 port 36629 connected with 10.75.0.4 port 5001
[ 6] local 10.75.0.3 port 36630 connected with 10.75.0.4 port 5001
[ 5] local 10.75.0.3 port 36631 connected with 10.75.0.4 port 5001
[ 4] local 10.75.0.3 port 36628 connected with 10.75.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-10.0 sec 19.8 MBytes 16.5 Mbits/sec
[ 7] 0.0-10.1 sec 14.4 MBytes 12.0 Mbits/sec
[ 3] 0.0-10.1 sec 5.88 MBytes 4.88 Mbits/sec
[ 5] 0.0-10.1 sec 36.8 MBytes 30.4 Mbits/sec
[ 4] 0.0-10.2 sec 16.2 MBytes 13.4 Mbits/sec
[SUM] 0.0-10.2 sec 93.0 MBytes 76.4 Mbits/secAlso tested without AES-NI with gave a factor 5 decrease of throughput.
The above results is from 2.2.3 and initial results of 2.2.4 showed even worse performance. Using AES-NI I expected an performance of >600Mbit/s so I can only conclude that something must be wrong.
CPU load was under 5% load at all times and Interrupt idle time over 95%.
-
Seems like 2.2.4 got even worse performance. The results is fluctuating and I'am not sure if AES-NI is even being used. Anyone got a working IPSEC setup using AES-NI?
[root@test3 strongswan]# iperf -n 32M -c 10.75.0.1 -P 5
–----------------------------------------------------------
Client connecting to 10.75.0.1, TCP port 5001
TCP window size: 204 KByte (default)[ 7] local 10.75.0.3 port 42604 connected with 10.75.0.1 port 5001
[ 3] local 10.75.0.3 port 42600 connected with 10.75.0.1 port 5001
[ 4] local 10.75.0.3 port 42601 connected with 10.75.0.1 port 5001
[ 5] local 10.75.0.3 port 42602 connected with 10.75.0.1 port 5001
[ 6] local 10.75.0.3 port 42603 connected with 10.75.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0- 8.0 sec 32.0 MBytes 33.5 Mbits/sec
[ 7] 0.0-17.2 sec 32.0 MBytes 15.6 Mbits/sec
[ 5] 0.0-20.0 sec 32.0 MBytes 13.4 Mbits/sec
[ 3] 0.0-25.6 sec 32.0 MBytes 10.5 Mbits/sec
[ 4] 0.0-26.5 sec 32.0 MBytes 10.1 Mbits/sec
[SUM] 0.0-26.5 sec 160 MBytes 50.7 Mbits/secNote: Have now discovered that "top" shows some load.. Idle interupt goes to zero and "nice" goes up:
