Slow TCP connections, fixed by MSS clamp
After installing pfSense box as a main router, I noticed low download speed from some http services.
When connectin directly via PPPoE it is like 2.5MB/s, but when adding pfsense router, the same http link gives me less then 200kB/s.
Problen is clearly related to router, not the endpoint. I tested several web services and differnet devices on my side.
Also, for some services speed loss is not so dramatic, 2-3 times, not 10 times.
So I continued experimenting on the following setup:
<internet>-<pfsense box>-<100mb switch>-<1Gb switch>-<Win7 client>
As a test http server i used cachefly.cachefly.net/100mb.test
Wireshark capture on the client shows a lot of "Dup ACK" and missing segments.
Comparing pfSense capture and client capture shows that some incoming packets show up on the router, but do not reach the client.
I managed to fix the issue by forcing MSS clamp on WAN interface to 1300.
For my PPPoE connection MTU is 1492, so MSS should be 1492-40=1452, but even at 1400 problem still exists.
Does anybody have a clue about the root cause of this issue or a proper solution?
When you connect directly do you see the correct MTU negotiated?
Do you see anything in the PPP logs in pfSense?
Do you still see the issue with a client connected directly to pfSense?
PPPoE connetion is established with MTU 1492 both on the router and directly connected PC.
PPP logs do not show any errors or such.
Also, downloading the same http link on pfsense box itself (with fetch -o /dev/null) gives me clean TCP flow wit no "dup ack" and good speed.
It looks like the problem appears only when NAT is engaged.
PS: I'll also check LAN connection itself
Further investigation showed that the problem is actualy related to all pfsense box outgoing packets longer then 1370 bytes and does not depend on receiving side.
PPPoE and NAT are not the cause also.
Testing with iperf3 UDP gives me 27% packet loss with udp data length 1363 and exactly zero loss with 1362.
High loss shows up only when sending >20Mb/s of UDP traffic.
At lower bandwidth loss is about 0.1% but still present.
At anything more than 20Mb/s exactly 27% is lost
[ 5] 1.00-2.00 sec 6.93 MBytes 58.2 Mbits/sec 0.216 ms 2003/7337 (27%) [ 5] 2.00-3.00 sec 6.94 MBytes 58.2 Mbits/sec 0.222 ms 2001/7337 (27%) [ 5] 3.00-4.00 sec 6.94 MBytes 58.2 Mbits/sec 0.211 ms 2000/7337 (27%) [ 5] 4.00-5.00 sec 6.90 MBytes 57.9 Mbits/sec 0.216 ms 2024/7336 (28%) [ 5] 5.00-6.00 sec 6.93 MBytes 58.1 Mbits/sec 0.223 ms 2007/7337 (27%) [ 5] 6.00-7.00 sec 6.93 MBytes 58.2 Mbits/sec 0.215 ms 2002/7337 (27%)
iperf tcp test in direction router->client shows something like 500kbits/s with default settings and 94Mb/s with forced lower MSS.
Lost packets show up at router`s LAN interface as outgoing, but do dot show at receivers interface.
Same test in reverse direction (client->router) gives no loss up to 90Mb/s with any packet length TCP/UDP.
I tried disabling/enabling hardware offloads with no changes.
Hmm, that sure looks like something hardware offload related. Was that with both switches still in between?
If it was it could be failing negotiate between those switches correctly. I'd have to guess that 100Mb switch is doing something odd. Can you test without it?