Peculiar throughput problem pfSense to pfSense
-
Okay, i have a very strange issue I don’t quite know how to troubleshoot, so I’m looking for input/ideas here:
The setup 2 sites:
Site A - SG6100, Gbit Symmetric WAN
Site B - SG2100, 300 Symmetric WANBoth sites can reach their full bandwidth in any speedtests on Internet sites.
There is a IPsec Site-2-Site tunnel established between the sites (tunnel mode).
OBSERVATIONS:
- Any internal Client in either site can exchange traffic with any other client on the remote site symmetrically at about 270Mbit - Exccellent :-)
- I have made a wellknown workaround on both pfsense boxes by adding a Gateway using its own local LAN address, and then adding a route to the remote sites subnet using that gateway. This makes each pfsense box capable of talking to the other pfsense using the tunnel - works as intended.
PROBLEM: Any traffic going from the 2100 to the 6100 is “capped” at about 300 KB/s - fx. A scp copy of a file. It doesn’t matter if I push it from the 2100 or fetch it from the 6100. It also doesn’t matter if I go through the Tunnel or copy it using public IP to public IP.
The other direction 6100 to the 2100 reaches about 70 Mbit - still far from the speed limit, but much better.Does any one have any idea why there is a limit to how fast the 2100 can talk to (send packets towards) the 6100 - tunneled or not?
-
Can you switch to route based (vti)? That would eliminate the need for such a workaround.
But throttling that severe says some routing issue to me. What states do you see created?
-
@stephenw10 I’ll have a look tomorrow, but it can’t be general routing as clients behind the firewalls works fine.
Also - since it happens on direct wan to wan SSH sessions as well (not over IPsec), I find it hard to believe VTI would change anything. -
Ah, interesting. I misunderstood that.
Hmm, could be a fragmentation issue. It would be pretty bad though! What do you see in a pcap outside the tunnel?
-
Can you show your IPsec config?
Even Advanced Parameter and if you use Intel QA.
Whats your MSS for IPsec Tunnel Mode? -
The fact it also does it outside the tunnel for a connection between the firewalls implies something in the internal rules is normalising the traffic to allow to pass that doesn't get applied to traffic from the host itself.
-
@stephenw10 Did some further investigation while doing a SCP of a large file from the 2100’s CLI on Site B, to the Public IP of the 6100 on Site A.
1: Only one state is present in each firewall. From public IP on Site B to public IP on Site A. No evidence of any funny internal routing on either box or NAT’ing of the traffic.
2: Packet capture on WAN on both boxes shows 1448 byte sizes packets being sent with no fragmentation. One observation is that the 6100 ack’s each and every packet individually which I guess means TCP sliding window does not get to have very many outstanding packets?
Since I do have a 36ms turnaround time between the sites that obviously has a CONSIDERABLE cost in throughput… 1000ms/37ms = 27 packets of 1448 bytes = 39KB/s if the source had to wait out EVERY ack. Since I’m getting about 300KB/s (little less than 3 Mbit) throughput there must be a wee bit of sliding window in play, but slow enough that each packet is ACK’ed individually.
3: There is no CPU load to speak of on the 2100 when this is being done (less than 5% usage)Since clients connections through the tunnel can get full throughput (about 270 Mbit), I don’t think this has anything to do with ISP throttling…
Mystery: Why can Site A 6100 send to Site B 2100 with about 70 Mbit - that is about 1/4 of the bandwidth clients can reach, but almost 25 times more than the other direction? Is that because ACK’s from site B to A are being held back?
Any ideas to what could be at play here?
-
@stephenw10 incidentally: I noticed that when I start the transfer, SCP throughput goes up for half a second to about 300KB/s and then dwindles all the way to 0 over the next 10 secs. It briefly claims the transfer is actually stalled before it kicks in again and now is somewhat stable at around 300KB/s.
It this “stalling” what causes the TCP sliding window to become useless in terms of throughput? -
Yes, the TCP window is likely being significantly affected by.....something.
What happens if you SCP from some client behind the 2100 to the 6100? Or the other way? Is it one end specifically that can be shown to be causing the problem.
Or even better can you test from some third location to each independently?
-
@stephenw10 Okay I have gone to town now on testing. All testing is done to the Public IP on each sites pfsense - aka: no IPsec unless otherwise noted.
It seems I did make one little mistake in my earlier reported throughput numbers concerning client to client throughput.Anyhow: Here's the findings:
Site A Client to Site B 2100 = 12 MB/s
Site A Client to Site B Client = 7 MB/s (Done inside IPsec Tunnel)
Site X Client to Site B 2100 = 13 MB/sSite B Client to Site A 6100 = 7 MB/s
Site B CLient to Site A Client = 25 MB/s (Done inside IPsec Tunnel)
Site X Client to Site A 6100 = 75 MB/sSite A 6100 to Site B 2100 = 7 MB/s
Site B 2100 to Site A 6100 = 300 KB/sHard to draw any conclusions from these numbers except there is something REALLY wrong when the 2100 itself has to send large number of full data packets.
But its noteworthy that site B struggles to reach the 300 Mbit line capacity in either direction during these tests. Thats likely latency playing its part though.
-
Were you able to test from the 2100 to Site X? Is that similarly throttled?
-
@stephenw10 Hmm, I don't have immediate access to such a test, but let me see what I can do...
-
Mmm, since the outlier there seems to be the 2100 itself sending traffic. Which is odd.
How is that 2100 configured? Anything unusual? Still using mvneta0 as WAN?
-
@stephenw10 Did a test from the Site B 2100 to a completely unrelated pfSense on Site X - which has about the same latency on turnaround.
It shows identical behavior to doing a transfer to Site A 6100 - that is: about 300KB/s throughput. I'm quite happy that's the case since we then know its not routing or some other config specific to my Site A setup.
To answer your question. I think the 2100 is very "standard" in the setup apart from WAN being a tagged VLAN (mvneta0.803), AND, the WAN connection is a GPON SFP that bridges ethernet to GPON. Obviously not standard, but working completely as expected from the clients point of view.
Since clients has the expected throughput up/down, and the firewall is doing NAT (one public IP only), the traffic is sourced identically on the outside, so how could the GPON SFP be the culprit? -
Yup that does seem to narrow it to the 2100 or at least something at that site or connection.
Is that a GPON SFP module?
-
@stephenw10 Yes, it a fs.com module. Have been using it for a couple of years without issues - probably apart from this. I just haven't noticed the problem before because I never had the need to transfer large files directly out of the pfsense box itself.
Actually the pfSense config is more or less identical to the Site A 6100 apart from the fact it uses a different VLAN on WAN, and has a BiDi standard Ethernet SFP instead.
What do you think is the next order of business? A packet capture of the SCP file transfer session setup? Perhaps it will show us something when it starts, then stalls before resuming at a steady 300KB/s?
-
Yup, packet capture the throttled traffic. It's so extreme I'd expect to see some pretty obvious issues.
-
@stephenw10 Yup, and there is… Seems I’m suffering an upstream (from 2100) packet loss problem when transmission speed is going up.
Quite interesting that the clients seem to handle that loss with much less consequense for the overall throughput. They are Linux and Windows Clients.
I’ll be looking into my options for tuning or replacing the GPON SFP…..
-
Mmm, the TCP congestion control in pfSense is nothing special because it's tuned for forwading not as a TCP endpoint You may be hitting that in some unusually extreme way!
That might also explain why you see problems across the tunnel too since, presumably, the tunnel is also lossy.
-
@stephenw10 Hmm, well I took a look at the packet loss in general (from my monitoring systems), and there actually is none: < 0.0001%.
The thing is this site rarely uses its upstream bandwidth, and when it does it’s always from WiFi clients. The site has older WiFi 6 AP’s with a best case max bandwidth of slightly less than 400mbps. This is more less what the GPON link is (about 360/360).
So now I’m starting to think: Is the issue really the GPON bridge lacking buffers, but since the WiFi speed is more or less the same as the GPON, buffer drops rarely happens, whereas pfSense itself thinks it’s a Gbit Ethernet link (SFP), so it pushes way to many packets initially causing lots of bufferdrops?
If so, could I create some limiter/bandwidth shaping policy to remediate that?
