IPSec VPN iperf3 Speeds For Single Stream Slow
-
Hoping to get some insight here to be sure I'm not missing something major when testing some IPSec VPN performance.
I have a IPSec connection setup between 2 Netgate 1541's with a roughly 300 meg pipe for them to work with, this is for a pretty high data throughput application so I've been doing some iperf3 tests just to see how well it's performing.
Thing is, when doing an iperf3 test from a client on the LAN of Box1 to the iperf3 server of Box2, AND only using a single stream, I'm seeing roughly 5 megabits per second of bandwidth. However, I know for sure some devices on this link are communicating around 30 megabits per second and if I set the -P for streams to say 20 I get in the realm of 150 megabits per second, which is lower than the 300 allotted but I assume there are some other bottlenecks in my testing causing that.
Really though, what I am trying to determine, is, does this make any sense? Should a single iperf3 stream be that slow? I feel like I've seen plenty of iperf tests in the realm of 1 Gbps or higher, so this feels off? Unfortunately I can't really test iperf over the link to another client on the Box2 LAN which is why I'm using the Box2 iperf package to test this.
For some additional info, because I'm sure it'll come up here:
-
I do have hardware accel on for these firewalls and it is working
-
I do NOT think I need to adjust MTU here in the advanced NAT settings of the firewall, I can ping over the VPN at 1472 bytes without fragmentation and setting iperf to use smaller MTUs for testing doesn't improve things at all
-
The Phase 2 in this case is using AES-GCM with 128 bits, so should be close to as fast as you can get while still being safe
Any advice would be greatly appreciated, driving myself crazy trying to see if there is something I am doing wrong.
Oh and for a final bit of background, I am asking all this because I am hoping we can get higher throughput than we are currently getting (that roughly 30Mbps) from a device on Box2's subnet, but I think the bottleneck may be somewhere other than the VPN itself (and I have seen spikes from that unit up to 100Mbps), maybe on the device itself. Just doing my due diligence to see if there is more that can be done.
-
-
@planedrop These situations can be really hard to figure out.
What kind of turn around latency are you looking into between the two clients (ping response time between clients)?
Latency can really be detriment to throughput, so perhaps you have something that adds massive latency and thus costs you the throughput. -
@keyser Good question, ran some tests earlier today and I was seeing around 60ms RTT to the Box2 Firewall. So not insane but not super tiny either.
-
@planedrop said in IPSec VPN iperf3 Speeds For Single Stream Slow:
@keyser Good question, ran some tests earlier today and I was seeing around 60ms RTT to the Box2 Firewall. So not insane but not super tiny either.
Well it's enough to likely cause TCP connections to not go much above 50mbits.
I would setup the iperf client/server on the two clients to test with UDP (to see actual troughput regardless of latency), and get a feeling for packetloss ratios.
Packetloss are a killer for TCP throughput because of stops and retransmits - where TCP also scales back the sliding window (dials down possible troughput because it wants more commitment from ACKs before transmitting again).I don't think your numbers are based on wrong IPSec config because even the worst possible settings would be way faster than that. Perhaps wrong NIC hardware assist settings in pfSense? Only hardware check summing should be enabled (And only if you are using Intel NICs).
TCP Segmentation offload and Large recieve offload should not be enabled.You are likely looking at either a link speed negotiation issue or a "relatively" high packetloss issue between the boxes.
-
@keyser I'll try to do some UDP testing and see how things go, definitely worth a shot.
I did some packet capturing during large data transfers and am NOT seeing a lot of TCP retransmits so I'm also hesitant to think that is the cause. And again to be completely clear this is only with a single stream from the iperf3 testing, if I do many streams I get way higher throughput much closer to line speed, so I'm really just trying to find out is a single stream supposed to be that slow?
I'm quite sure I have all the NIC settings configured right, I don't have TCP segmentation offload or large receive offload enabled, and just to be clear I'm quite experienced with setting up VPNs in this manner and have other ones with the same hardware that perform plenty fast (near line speed) EXCEPT with iperf3 single stream testing. So don't think anything in my config is directly wrong unless I'm totally blind and missing something lol.
But without any obvious errors, and with packet loss seemingly not the issue (though I'll do a few more tests in this regard before confirming since it was only 1 pcap I did), AND with multi stream iperf hitting 150Mbps or higher, it's just weird....
-
@keyser Also just FYI I'm very grateful for your help, hopefully I don't come off as grumpy lol, just literally quite sick this last week and trying to figure out what's going on with this has been a little stressful, so apologies if I sound at all angry.
-
@planedrop said in IPSec VPN iperf3 Speeds For Single Stream Slow:
@keyser I'll try to do some UDP testing and see how things go, definitely worth a shot.
I did some packet capturing during large data transfers and am NOT seeing a lot of TCP retransmits so I'm also hesitant to think that is the cause. And again to be completely clear this is only with a single stream from the iperf3 testing, if I do many streams I get way higher throughput much closer to line speed, so I'm really just trying to find out is a single stream supposed to be that slow?
I'm quite sure I have all the NIC settings configured right, I don't have TCP segmentation offload or large receive offload enabled, and just to be clear I'm quite experienced with setting up VPNs in this manner and have other ones with the same hardware that perform plenty fast (near line speed) EXCEPT with iperf3 single stream testing. So don't think anything in my config is directly wrong unless I'm totally blind and missing something lol.
But without any obvious errors, and with packet loss seemingly not the issue (though I'll do a few more tests in this regard before confirming since it was only 1 pcap I did), AND with multi stream iperf hitting 150Mbps or higher, it's just weird....
Yeah, but perhaps latency is the issue then. If the OS is not using modern optimistic TCP scaling mechanisms, a 60ms latency is actually only worth a sustained 8Mbps session. See this test:
https://accedian.com/blog/measuring-network-performance-latency-throughput-packet-loss/
-
@keyser OK wow this actually helps a ton, so I guess in that case then when I am using multiple streams it's multiple TCP connections so I can see nearly linear scaling compared to number of streams (until CPU bottleneck or something along those lines).
It is across the entire country so I guess this makes sense, was hoping I could see faster connectivity through it but if this is what we get then it's what we get. Just wanted to be sure I wasn't seeing something insanely wrong going on.
Thanks for the link and all the help here in general, seems like there isn't a lot I can do to increase things further than that a single TCP stream from iperf3 is performing roughly where I'd expect it based on that latency chart.
-
@planedrop My pleasure.
But be advised that several operating systems do use more optimistic TCP scaling mechanisms by default (windows is one of them), and that should allow you to see up to 3 to 5 times of that test throughput.
The remaining operating systems generally have some settings you can fiddle with to change how the big the sliding windows can become and how aggressive congestion scaling is handled.
Obviously this is hit SEVERELY if the line also features even a little packetloss. -
@keyser Fortunately I don't think we are getting any packet loss here, more testing and I'm seeing absolutely zero retransmits so that's good.
Unfortunately though the specific device that needs the high bandwidth is from a vendor and uses their own OS, it's technically Ubuntu but point is I can't login to make any adjustments to it. Might find out if the vendor can do any tuning on it for this remote setup.
It's an appliance that typically is on a local subnet rather than remote so it's definitely not setup for this, so considering it only operates about half as fast as on site, I'm pretty happy lol, just wanted to double check things.
Thanks again for all the insight here, greatly appreciate it!