Adjust MSS calculation to account for VLANs
-
Yes, duckduckgo is the site I'm trying to get to. It works fine without VLAN - they are identical handshakes but the server doesn't send any fragmentation messages back:
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
How does it know the difference?? Maybe there is a flag somewhere in there that I am overlooking..
I have cleared the MSS setting so I should have 1460 from now on.
It looks like it is just duckduckgo. I recall having more issues in the past but now it is just that site. Setting my PC's MTU to something lower like 1400 gets me through, but I'd prefer the PMTUD to work properly.
-
@virtual-frog PMTUD needs to be supported on all devices in the path. Sadly it isnt as ISPs block icmp.
-
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
It is not blocked by my ISP. You can see the ICMP packets arrive at my WAN but not make it through the firewall for some reason.
-
@virtual-frog The internet is made up of hundreds of carriers. PMTUD to function properly needs to be working end to end. So if it works on one path thats great. My ISP blocks pings so PTMUD would never work for me.
-
So there are two issues here:
Why is it not working on my pfSense instance, where my ISP does not block ICMP?
How would it work when your ISP does block ICMP? Do most routers run MTU black hole detection? MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
MTU is not something that a typical home user would be messing with, so I'm wondering how this issue doesn't pop up more often.
The issue pops up all the time. ISPs may misconfigure a link to support a low MTU. Who knows.
PTMTUD relies on ICMP. You cannot force every single router in the world to not block ICMP which is why pmtud often doesnt work.In your case as its already been demonstrated, a lower than 1500 byte MTU in the path with the DF bit set is whats breaking access to certain sites. The only workaround is to make sure that your packet sizes that you send out into the world are less than 1500 bytes.
-
@michmoor said in Adjust MSS calculation to account for VLANs:
The issue pops up all the time.
But not on typical consumer home networks (no PPPoE, IPsec, VLANS, etc). I've set up several "routers" over the years with various locales and ISPs and never once had an issue where a major site fails to connect like this. Is it the result of everyone using 1500 for the MTU by convention, so things "just work"?
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
and some external router was refusing to fragment
Are you getting any ICMP too big messages? The source is supposed to respond to that.
BTW, the world is moving to Path MTU Discovery (PMTUD). It's mandatory on IPv6 and now being used on IPv4. The client should be able to handle those ICMP messages and reduce the packet size.
-
Look at my packet captures earlier in the thread to see the ICMP "fragmentation needed" packets making it to my WAN but not back to the VLAN side.
Yes, we have been discussing PMTUD already. It is not working because the ICMP messages are being blocked by the firewall (not making it back to my client).
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
But internet access only works when I have MSS=1496 on my VLAN interface.
I have the default 1500 on my guest WiFi VLAN. It works fine for me. However, VLANs should have no effect on what goes out to the Internet, as the VLAN tags is discarded by pfSense.
-
Yes, I know VLANs should not affect internet traffic. But somehow they do. Look at my packet captures to see how I can connect on LAN (untagged) but face issues on my VLAN.
-
Just to nip back to basics with MSS - if it set too large the packet is just dropped. That is it, you are done. There is no fragmentation available or offered; effectively it is at Layer 4 (if we slip into the old OSI model, before it became a bit blurry).
The lack of fragmentation or of a negotiated feedback loop is why we sometimes 'clamp' the MSS in a rather blunt manner in order to deliver packets.
This is a fundamental difference between MSS and the outer jacket we call MTU. That can be fragmented or negotiate to a suitable size.
That difference may have been lost in this thread; not aided by the reference to MSS together with VLANs. Those 2 concepts fly together in exactly the same way that bricks don't.
️ -
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).
The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.
Please read my bit again, it will help.
Take if from this direction:
If you squeeze your ICMP MTU down to the size of 'stupidly small' you can induce an MSS problem which, as it cannot be fragmented as just explained, means it drops the packet dead. There will be no returning ICMP message.
I suspect this is what duckduckgo is doing. You can get a simple ping from them but go 1 byte larger you just go in the bin. No wasted packets, MTU negotiation or bandwidth wasted replying to some random IP wanting to run a test against them.
That said, I find it really odd that you are getting "fragmentation needed" messages. Are you really sure of this - ie you have checked with Wireshark?
-
(posting this image for the fourth time)
@virtual-frog said in Adjust MSS calculation to account for VLANs:
Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.
I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:
I am not sending pings, I am trying to connect to their website with a web browser (TCP). The top picture shows the connection failing from VLAN and the bottom shows it working from LAN (no tag). Same MTU (1500) and MSS (1460) settings for both.
The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?
-
With pictures:
Just one ping... ok, 1 byte over the regular ping to duckduckgo:
~ % ping -D -s 57 duckduckgo.com PING duckduckgo.com (52.142.124.215): 57 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 Request timeout for icmp_seq 4 Request timeout for icmp_seq 5The packets are dead, they have ceased to be.
One byte smaller:
~ % ping -D -s 56 duckduckgo.com PING duckduckgo.com (52.142.124.215): 56 data bytes 64 bytes from 52.142.124.215: icmp_seq=0 ttl=116 time=19.785 ms 64 bytes from 52.142.124.215: icmp_seq=1 ttl=116 time=19.598 ms 64 bytes from 52.142.124.215: icmp_seq=2 ttl=116 time=19.523 ms 64 bytes from 52.142.124.215: icmp_seq=3 ttl=116 time=19.381 ms 64 bytes from 52.142.124.215: icmp_seq=4 ttl=116 time=19.540 msLike Lazarus, the ping is back.
Wireshark:
Yep, zero response as soon as you are a single byte over a normal ping. No discovery, no MTU negotiation required, no request for fragmentation. In fact no response at all and nothing being blocked by your firewall. The network link is responding to an MSS limit that happened to be dressed in MTU clothing.
️ -
@virtual-frog said in Adjust MSS calculation to account for VLANs:
The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?
Reply
I assume VLAN2 on the switch, what is the MTU of the switch ports for devices connected to VLAN2?
EDIT:
Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 network
The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?
Do you have a drawing of how this is designed?Firewall: NetGate,Palo Alto-VM,Juniper SRX
Routing: Juniper, Arista, Cisco
Switching: Juniper, Arista, Cisco
Wireless: Unifi, Aruba IAP
JNCIP,CCNP Enterprise -
@michmoor
I'm on VLAN 20. Not sure about MTU settings for this switch (using TP Link Omada SDN so it's a little weird) - there is a jumbo frame option that is set to 1518 so I assume it can at least handle the standard 1500.
I can ping both google and my firewall at 1472. If I turn off the MTU checking flag I can ping my firewall at much larger packet sizes, they will just get fragmented. Note that Google is one of the sites that I don't have issues with.
The connection between my PC and firewall works fine, no fragmentation issues or dropped packets.
-
@virtual-frog said in Adjust MSS calculation to account for VLANs:
- why are they stopped by the firewall?
As described earlier, VLANs are stripped away by the router. They do not leave your network. This is not a firewall issue. You need to look internal.
A successful exchange with duckduckgo:
Zero issues with MSS or MTU with full 1500 (1514) byte packets being exchanged with DDG.
️ -
Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 networkIt's VLAN tag 20, but yes that's the right network.
The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?
Oops, missed the one in the detail pane
Those packet captures are on the WAN side of my firewall, so you're seeing the NAT address. My VLANs are on top of the LAN network, and it all gets NATed to one WAN connection on my firewall.Do you have a drawing of how this is designed?
No graphic but it's a pretty standard setup I think.
ISP modem -> pfSense device -> switch -> PC
Where the firewall/switch connection is a VLAN trunk and the switch is managed with different VLAN config for each port.Edit
I do have Snort intrusion detection running on the WAN interface but I have tried disabling it with the same results.
I stripped down as much as possible on my ISP's router so it is not running its packet filter or anything.
