Adjust MSS calculation to account for VLANs
-
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
It raises a several questions: Why is the ICMP not making it through? Shouldn't it be allowed as part of the same stream? I'm not too familiar with firewalls.. Why does it need to be fragmented to begin with? The IP datagrams are 1500 bytes each (the 1514 shown in the length field includes the whole Ethernet frame). Why do VLANs affect this at all? As you mentioned the VLAN ID is stripped out, so it shouldn't affect outgoing packets.
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
MSS=1456 when my VLAN MSS is set to 1496
So question for you.. What is the mtu size set on pfsense interfaces and your client on this vlan? the standard 1500?
Can you client ping pfsense with DF set at 1472? example
$ ping -l 1472 -f 192.168.9.253 Pinging 192.168.9.253 with 1472 bytes of data: Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Ping statistics for 192.168.9.253: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms $ ping -l 1473 -f 192.168.9.253 Pinging 192.168.9.253 with 1473 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set.Now do the same test out to say google.com
$ ping -l 1473 -f 8.8.8.8 Pinging 8.8.8.8 with 1473 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Ping statistics for 8.8.8.8: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), $ ping -l 1472 -f 8.8.8.8 Pinging 8.8.8.8 with 1472 bytes of data: Reply from 8.8.8.8: bytes=68 (sent 1472) time=11ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Ping statistics for 8.8.8.8: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 10ms, Maximum = 11ms, Average = 10ms -
I can ping both google and my firewall at 1472. If I turn off the MTU checking flag I can ping my firewall at much larger packet sizes, they will just get fragmented. Note that Google is one of the sites that I don't have issues with.
For duckduckgo (one of the sites I do have problems with) I can ping only up to 64. At 65 it hangs with no MTU warning - although I don't get any fragmentation ICMP replies on a WAN packet capture, so maybe their site is configured to drop any pings larger than 64 bytes.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Why is the ICMP not making it through
Huh.. that is sending you that frag needed.. That mtu needs to be smaller.. What site is that exactly? You saying that happens on all sites?
1514 is normal length that would be shown in wireshark..
Your trying to talk to duckduckgo.com ? I am not having any issues with that site and normal 1460
And yes with the 14 byte header frame in wireshark would be shown as 1514
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
so maybe their site is configured to drop any pings larger than 64 bytes.
I do believe that is the case.. Larger pings to that site are not working for me either.
edit: look in their synack back, they send a mss of 1440
your seeing the same thing on yours..
edit: not sure what you got going on as of yet. But your mss value of 1456 is higher then what they are saying to talk to them at anyway..
-
Yes, duckduckgo is the site I'm trying to get to. It works fine without VLAN - they are identical handshakes but the server doesn't send any fragmentation messages back:
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
How does it know the difference?? Maybe there is a flag somewhere in there that I am overlooking..
I have cleared the MSS setting so I should have 1460 from now on.
It looks like it is just duckduckgo. I recall having more issues in the past but now it is just that site. Setting my PC's MTU to something lower like 1400 gets me through, but I'd prefer the PMTUD to work properly.
-
@virtual-frog PMTUD needs to be supported on all devices in the path. Sadly it isnt as ISPs block icmp.
-
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
It is not blocked by my ISP. You can see the ICMP packets arrive at my WAN but not make it through the firewall for some reason.
-
@virtual-frog The internet is made up of hundreds of carriers. PMTUD to function properly needs to be working end to end. So if it works on one path thats great. My ISP blocks pings so PTMUD would never work for me.
-
So there are two issues here:
Why is it not working on my pfSense instance, where my ISP does not block ICMP?
How would it work when your ISP does block ICMP? Do most routers run MTU black hole detection? MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
The issue pops up all the time. ISPs may misconfigure a link to support a low MTU. Who knows.
PTMTUD relies on ICMP. You cannot force every single router in the world to not block ICMP which is why pmtud often doesnt work.In your case as its already been demonstrated, a lower than 1500 byte MTU in the path with the DF bit set is whats breaking access to certain sites. The only workaround is to make sure that your packet sizes that you send out into the world are less than 1500 bytes.
-
@michmoor said in Adjust MSS calculation to account for VLANs:
The issue pops up all the time.
But not on typical consumer home networks (no PPPoE, IPsec, VLANS, etc). I've set up several "routers" over the years with various locales and ISPs and never once had an issue where a major site fails to connect like this. Is it the result of everyone using 1500 for the MTU by convention, so things "just work"?
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
and some external router was refusing to fragment
Are you getting any ICMP too big messages? The source is supposed to respond to that.
BTW, the world is moving to Path MTU Discovery (PMTUD). It's mandatory on IPv6 and now being used on IPv4. The client should be able to handle those ICMP messages and reduce the packet size.
-
Look at my packet captures earlier in the thread to see the ICMP "fragmentation needed" packets making it to my WAN but not back to the VLAN side.
Yes, we have been discussing PMTUD already. It is not working because the ICMP messages are being blocked by the firewall (not making it back to my client).
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
But internet access only works when I have MSS=1496 on my VLAN interface.
I have the default 1500 on my guest WiFi VLAN. It works fine for me. However, VLANs should have no effect on what goes out to the Internet, as the VLAN tags is discarded by pfSense.
-
Yes, I know VLANs should not affect internet traffic. But somehow they do. Look at my packet captures to see how I can connect on LAN (untagged) but face issues on my VLAN.
-
Just to nip back to basics with MSS - if it set too large the packet is just dropped. That is it, you are done. There is no fragmentation available or offered; effectively it is at Layer 4 (if we slip into the old OSI model, before it became a bit blurry).
The lack of fragmentation or of a negotiated feedback loop is why we sometimes 'clamp' the MSS in a rather blunt manner in order to deliver packets.
This is a fundamental difference between MSS and the outer jacket we call MTU. That can be fragmented or negotiate to a suitable size.
That difference may have been lost in this thread; not aided by the reference to MSS together with VLANs. Those 2 concepts fly together in exactly the same way that bricks don't.
️ -
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.
Please read my bit again, it will help.
Take if from this direction:
If you squeeze your ICMP MTU down to the size of 'stupidly small' you can induce an MSS problem which, as it cannot be fragmented as just explained, means it drops the packet dead. There will be no returning ICMP message.
I suspect this is what duckduckgo is doing. You can get a simple ping from them but go 1 byte larger you just go in the bin. No wasted packets, MTU negotiation or bandwidth wasted replying to some random IP wanting to run a test against them.
That said, I find it really odd that you are getting "fragmentation needed" messages. Are you really sure of this - ie you have checked with Wireshark?
-
(posting this image for the fourth time)
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
I am not sending pings, I am trying to connect to their website with a web browser (TCP). The top picture shows the connection failing from VLAN and the bottom shows it working from LAN (no tag). Same MTU (1500) and MSS (1460) settings for both.
The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?
