Adjust MSS calculation to account for VLANs
-
I created a basic network with pfSense and everything worked fine. After I added VLANs almost every external site stopped working. Eventually I figured out that the TCP Maximum Segment Size was too high- it was overrunning the standard 1500-byte MTU and some external router was refusing to fragment, thus dropping all of my IPv4 internet traffic. The UI field for MSS says "MSS clamping for TCP connections to the value entered above minus 40 for IPv4 (TCP/IPv4 header size) and minus 60 for IPv6 (TCP/IPv6 header size) will be in effect". This does NOT INCLUDE the 4 bytes needed for VLAN tagging. I suggest adding a mechanism to check if VLANs are in use and automatically add the 4-byte offset to this calculation. It is a sneaky issue which caused me a lot of headache.
-
@virtual-frog you fundamentally misunderstand what MSS here. Vlan headers don’t matter when traversing a WAN.
What is likely is you have a segment in the path that has a low MTU -
@michmoor you may be right, I can't explain how it affects the WAN side of things. But internet access only works when I have MSS=1496 on my VLAN interface. This does not occur without VLANs (I can leave the field empty, default 1500). Another confounding factor is that it does not affect all sites, e.g. google is accessible but not duckduckgo.
Plus won't it be a good feature to have regardless? To avoid fragmentation.
-
@virtual-frog no it won’t be a good feature to have because it doesn’t make sense when routing.
Typically a MSS would be 1460 bytes. 40 bytes for the TCP and IP header. That’s 1500 bytes.
If there is a router in the path that has a lower MTU configured and the DF bit is set then the packet gets dropped and a ICMP packet is returned to the sender stating that fragmentation is needed.If DF bit is not set than that packets gets fragmented and off it goes
Again, your vlan tags mean nothing to routers on the internet. -
@virtual-frog said in Adjust MSS calculation to account for VLANs:
works when I have MSS=1496 on my VLAN interface.
What?? The normal mss on a 1500 MTU would be 1460.. Why don't you sniff on your out going traffic out your wan.. What is the mss your sending, it would be in your syn packets as an option
-
To clarify, I've been talking in terms of the pfSense input value, which automatically gets 40 subtracted from it. So the actual MSS would be 1460-4 = 1456.
-
@johnpoz I performed a packet capture on WAN as you suggested. MSS=1456 when my VLAN MSS is set to 1496 (WAN left at default), MSS=1460 when VLAN MSS is on the default (non-working) setting.
-
@virtual-frog again no idea what you talking about... Here input into pfsense as the router..
From a vlan tagged packet..
-
@virtual-frog again you are fundamentally misunderstanding what MSS is and the role MTU plays when it comes to fragmentation.
https://www.cloudflare.com/learning/network-layer/what-is-mss/
https://www.cloudflare.com/learning/network-layer/what-is-mtu/
-
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
It raises a several questions: Why is the ICMP not making it through? Shouldn't it be allowed as part of the same stream? I'm not too familiar with firewalls.. Why does it need to be fragmented to begin with? The IP datagrams are 1500 bytes each (the 1514 shown in the length field includes the whole Ethernet frame). Why do VLANs affect this at all? As you mentioned the VLAN ID is stripped out, so it shouldn't affect outgoing packets.
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
MSS=1456 when my VLAN MSS is set to 1496
So question for you.. What is the mtu size set on pfsense interfaces and your client on this vlan? the standard 1500?
Can you client ping pfsense with DF set at 1472? example
$ ping -l 1472 -f 192.168.9.253 Pinging 192.168.9.253 with 1472 bytes of data: Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Reply from 192.168.9.253: bytes=1472 time<1ms TTL=64 Ping statistics for 192.168.9.253: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms $ ping -l 1473 -f 192.168.9.253 Pinging 192.168.9.253 with 1473 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set.Now do the same test out to say google.com
$ ping -l 1473 -f 8.8.8.8 Pinging 8.8.8.8 with 1473 bytes of data: Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Ping statistics for 8.8.8.8: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), $ ping -l 1472 -f 8.8.8.8 Pinging 8.8.8.8 with 1472 bytes of data: Reply from 8.8.8.8: bytes=68 (sent 1472) time=11ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Reply from 8.8.8.8: bytes=68 (sent 1472) time=10ms TTL=58 Ping statistics for 8.8.8.8: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 10ms, Maximum = 11ms, Average = 10ms -
I can ping both google and my firewall at 1472. If I turn off the MTU checking flag I can ping my firewall at much larger packet sizes, they will just get fragmented. Note that Google is one of the sites that I don't have issues with.
For duckduckgo (one of the sites I do have problems with) I can ping only up to 64. At 65 it hangs with no MTU warning - although I don't get any fragmentation ICMP replies on a WAN packet capture, so maybe their site is configured to drop any pings larger than 64 bytes.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Why is the ICMP not making it through
Huh.. that is sending you that frag needed.. That mtu needs to be smaller.. What site is that exactly? You saying that happens on all sites?
1514 is normal length that would be shown in wireshark..
Your trying to talk to duckduckgo.com ? I am not having any issues with that site and normal 1460
And yes with the 14 byte header frame in wireshark would be shown as 1514
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
so maybe their site is configured to drop any pings larger than 64 bytes.
I do believe that is the case.. Larger pings to that site are not working for me either.
edit: look in their synack back, they send a mss of 1440
your seeing the same thing on yours..
edit: not sure what you got going on as of yet. But your mss value of 1456 is higher then what they are saying to talk to them at anyway..
-
Yes, duckduckgo is the site I'm trying to get to. It works fine without VLAN - they are identical handshakes but the server doesn't send any fragmentation messages back:
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
How does it know the difference?? Maybe there is a flag somewhere in there that I am overlooking..
I have cleared the MSS setting so I should have 1460 from now on.
It looks like it is just duckduckgo. I recall having more issues in the past but now it is just that site. Setting my PC's MTU to something lower like 1400 gets me through, but I'd prefer the PMTUD to work properly.
-
@virtual-frog PMTUD needs to be supported on all devices in the path. Sadly it isnt as ISPs block icmp.
-
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
It is not blocked by my ISP. You can see the ICMP packets arrive at my WAN but not make it through the firewall for some reason.
-
@virtual-frog The internet is made up of hundreds of carriers. PMTUD to function properly needs to be working end to end. So if it works on one path thats great. My ISP blocks pings so PTMUD would never work for me.
-
So there are two issues here:
Why is it not working on my pfSense instance, where my ISP does not block ICMP?
How would it work when your ISP does block ICMP? Do most routers run MTU black hole detection? MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
The issue pops up all the time. ISPs may misconfigure a link to support a low MTU. Who knows.
PTMTUD relies on ICMP. You cannot force every single router in the world to not block ICMP which is why pmtud often doesnt work.In your case as its already been demonstrated, a lower than 1500 byte MTU in the path with the DF bit set is whats breaking access to certain sites. The only workaround is to make sure that your packet sizes that you send out into the world are less than 1500 bytes.
