Mystified by iperf results across IPsec tunnel
I am using two installations of pfSense to establish an IPsec tunnel from Verizon Gigabit Fios on the East Coast to Centurylink Gigabit fiber in the inter-mountain West. Here are the basic spec's:
On Verizon Fios Gigabit port, I have:
Dual-six core Intel 5650, 8GB RAM (HP DL360 - overkill, I realize)
4 X Broadcom NetExtreme 1000Base-SX (WAN, LAN and 2 spare)
On the Centurylink Gigabit fiber port, I have:
2 vCPU Intel 5650, 2GB RAM - Running as a VMware VM
2 X Intel E1000 vNIC's in the VM (WAN and LAN)
I recognize the difference in provisioning, but I have never seen the VM become even 20% utilized with RAM or CPU.
The IPsec VPN is: Blowfish (256 bits)/SHA1 - Blowfish (256 bits)/SHA1 (very stable)
I have recently been trying to optimize throughput, and been somewhat mystified by the results. When I run:
iperf -c 192.168.100.29 I might get: [SUM] 0.0-13.6 sec 89.2 MBytes 55.2 Mbits/sec
iperf -c 192.168.100.29 -P 40 -t 60 I might get: SUM] 0.0-63.4 sec 739 MBytes 97.8 Mbits/sec
I had previously thought those were pretty good results for any VPN, but then I thought to set-up three clients and three servers and was surprised to see the VPN gracefully handle all of the traffic! Running three of each of the previous tests, I saw almost identical results (simultaneously). In other words: the VPN was pushing 300 Mbps for 60 seconds at one point. BTW, in all cases iperf is running from CentOS 6 X64m as I never trust Windows TCP/IP stack.
Now here is the problem: The VPN is for disaster Recovery and the source and destination Proxies are Windows Server 2008 X64 and I never get throughput greater than 80-100 Mbps.
The question: If the VPN can push 300 Mbps (or more) across the WAN, why does a single connection from one IP (at the source) to one IP (at the destination) seem to be limited to 60-100 Mbps?