Question about throughput

commgdog

Hello,

I set up a site-to-site VPN between two pfSense boxes.

Site A is a residential connection (1 Gbps down/15 Mbps up), and site B is in a data center geographically about 20 miles away (1 Gbps up/down). Over WAN I am able to download from the data center at the full 1 Gbps throughput.

I set up the IPSec connection between the two, and everything connects fine.

Phase 1: IKEv1 PSK, AES 128/SHA256/DH5
Phase 2: ESP AES128-GCM 128/SHA256/PFS 5

When I download from the data center over the VPN, my bandwidth seems to be capped at around 15 Mbps (the upload speed on site A).

I'm wondering, is IPSec bandwidth limited by the lowest speed in any direction in the connection?

I was hoping to get higher throughput downloading from site B.

Bandwidth tested using iperf and scp'ing iso files from the data center.

Using AES-NI, e3-1220 v3 on both sides. Hardware is super under utilized.

Derelict

What's the latency over the link?

You can disable SHA256 on the P2. GCM is an authenticatiod cipher so you don't need another hashing layer.

commgdog

Average 32ms between the sites.

I disabled the hashing on the P2, seems to have helped it a bit, average is about 30 Mbits/sec between the sites.

I did some more testing and set up a tunnel with the same settings between site B and another site in the same rack and was able to push 160 Mbits between the two machines locally (CPU bottleneck at that point), so I'm starting to think its an issue with my bandwidth/ISP on site A. Would the 32ms latency between the sites really be enough to degrade the connection to 30 Mbits/sec? As mentioned before, not over IPSec I am able to download at full gigabit speeds from site B to site A.

chrismacmahon

@commgdog said in Question about throughput:

When I download from the data center over the VPN, my bandwidth seems to be capped at around 15 Mbps (the upload speed on site A).

What/how are you downloading, is it a SMB file share, FTP, scp?

commgdog

Tested with iperf3 and scp.

On site A: iperf3 -t 10 -R -c x.x.x.x
On site B: iperf3 -s

SCP was some ISO file transfers.

Derelict

32ms across IPsec?

If so it sounds like you're getting right about what you should for a single-stream TCP session with 32ms latency and a 128KB buffer.

That is probably a little high since you have the 30Mbit upstream at one end and certainly not a 1460 MSS across IPsec.

Bandwidth-delay Product and buffer size
BDP (1000 Mbit/sec, 32.0 ms) = 4.00 MByte
required tcp buffer to reach 1000 Mbps with RTT of 32.0 ms >= 3906.2 KByte
maximum throughput with a TCP window of 128 KByte and RTT of 32.0 ms <= **32.77 Mbit/sec.**

You could try giving a -P4 or -P8 to the iperf client to see if running multiple streams helps.

Or switch to UDP and see how high you can take the -b parameter before you start experiencing loss.