Adjust MSS calculation to account for VLANs

virtual-frog

@michmoor said in Adjust MSS calculation to account for VLANs:

The issue pops up all the time.

But not on typical consumer home networks (no PPPoE, IPsec, VLANS, etc). I've set up several "routers" over the years with various locales and ISPs and never once had an issue where a major site fails to connect like this. Is it the result of everyone using 1500 for the MTU by convention, so things "just work"?

JKnott

@virtual-frog said in Adjust MSS calculation to account for VLANs:

and some external router was refusing to fragment

Are you getting any ICMP too big messages? The source is supposed to respond to that.

BTW, the world is moving to Path MTU Discovery (PMTUD). It's mandatory on IPv6 and now being used on IPv4. The client should be able to handle those ICMP messages and reduce the packet size.

virtual-frog

@JKnott

Look at my packet captures earlier in the thread to see the ICMP "fragmentation needed" packets making it to my WAN but not back to the VLAN side.

Yes, we have been discussing PMTUD already. It is not working because the ICMP messages are being blocked by the firewall (not making it back to my client).

JKnott

@virtual-frog said in Adjust MSS calculation to account for VLANs:

But internet access only works when I have MSS=1496 on my VLAN interface.

I have the default 1500 on my guest WiFi VLAN. It works fine for me. However, VLANs should have no effect on what goes out to the Internet, as the VLAN tags is discarded by pfSense.

virtual-frog

@JKnott

Yes, I know VLANs should not affect internet traffic. But somehow they do. Look at my packet captures to see how I can connect on LAN (untagged) but face issues on my VLAN.

RobbieTT

@virtual-frog

Just to nip back to basics with MSS - if it set too large the packet is just dropped. That is it, you are done. There is no fragmentation available or offered; effectively it is at Layer 4 (if we slip into the old OSI model, before it became a bit blurry).

The lack of fragmentation or of a negotiated feedback loop is why we sometimes 'clamp' the MSS in a rather blunt manner in order to deliver packets.

This is a fundamental difference between MSS and the outer jacket we call MTU. That can be fragmented or negotiate to a suitable size.

That difference may have been lost in this thread; not aided by the reference to MSS together with VLANs. Those 2 concepts fly together in exactly the same way that bricks don't.

️

virtual-frog

@RobbieTT

@virtual-frog said in Adjust MSS calculation to account for VLANs:

Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).

The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.

RobbieTT

@virtual-frog said in Adjust MSS calculation to account for VLANs:

@RobbieTT

@virtual-frog said in Adjust MSS calculation to account for VLANs:

Sorry for my confusion earlier, I see that MSS was not the real issue and lowering it just happened to alleviate the problem (for TCP anyways).

The issue now is that one specific site (duckduckgo.com) is returning ICMP "fragmentation needed" messages, which are not making it through my firewall back to my PC. I understand they should be unrelated but it only happens on VLANs. You can read the thread for details.

Please read my bit again, it will help.

Take if from this direction:

If you squeeze your ICMP MTU down to the size of 'stupidly small' you can induce an MSS problem which, as it cannot be fragmented as just explained, means it drops the packet dead. There will be no returning ICMP message.

I suspect this is what duckduckgo is doing. You can get a simple ping from them but go 1 byte larger you just go in the bin. No wasted packets, MTU negotiation or bandwidth wasted replying to some random IP wanting to run a test against them.

That said, I find it really odd that you are getting "fragmentation needed" messages. Are you really sure of this - ie you have checked with Wireshark?

virtual-frog

@RobbieTT

(posting this image for the fourth time)

@virtual-frog said in Adjust MSS calculation to account for VLANs:

Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.

I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:

I am not sending pings, I am trying to connect to their website with a web browser (TCP). The top picture shows the connection failing from VLAN and the bottom shows it working from LAN (no tag). Same MTU (1500) and MSS (1460) settings for both.

The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?

RobbieTT

With pictures:

Just one ping... ok, 1 byte over the regular ping to duckduckgo:

~ % ping -D -s 57 duckduckgo.com
PING duckduckgo.com (52.142.124.215): 57 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5

The packets are dead, they have ceased to be.

One byte smaller:

~ % ping -D -s 56 duckduckgo.com
PING duckduckgo.com (52.142.124.215): 56 data bytes
64 bytes from 52.142.124.215: icmp_seq=0 ttl=116 time=19.785 ms
64 bytes from 52.142.124.215: icmp_seq=1 ttl=116 time=19.598 ms
64 bytes from 52.142.124.215: icmp_seq=2 ttl=116 time=19.523 ms
64 bytes from 52.142.124.215: icmp_seq=3 ttl=116 time=19.381 ms
64 bytes from 52.142.124.215: icmp_seq=4 ttl=116 time=19.540 ms

Like Lazarus, the ping is back.

Wireshark:

2023-06-14 at 19.00.42.png

Yep, zero response as soon as you are a single byte over a normal ping. No discovery, no MTU negotiation required, no request for fragmentation. In fact no response at all and nothing being blocked by your firewall. The network link is responding to an MSS limit that happened to be dressed in MTU clothing.

️

michmoor

@virtual-frog said in Adjust MSS calculation to account for VLANs:

The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?

Reply

I assume VLAN2 on the switch, what is the MTU of the switch ports for devices connected to VLAN2?

EDIT:
Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 network
The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?
Do you have a drawing of how this is designed?

virtual-frog

@michmoor
I'm on VLAN 20. Not sure about MTU settings for this switch (using TP Link Omada SDN so it's a little weird) - there is a jumbo frame option that is set to 1518 so I assume it can at least handle the standard 1500.

I can ping both google and my firewall at 1472. If I turn off the MTU checking flag I can ping my firewall at much larger packet sizes, they will just get fragmented. Note that Google is one of the sites that I don't have issues with.

The connection between my PC and firewall works fine, no fragmentation issues or dropped packets.

RobbieTT

@virtual-frog said in Adjust MSS calculation to account for VLANs:

why are they stopped by the firewall?

As described earlier, VLANs are stripped away by the router. They do not leave your network. This is not a firewall issue. You need to look internal.

A successful exchange with duckduckgo:

2023-06-14 at 19.36.00.png

Zero issues with MSS or MTU with full 1500 (1514) byte packets being exchanged with DDG.

️

virtual-frog

@michmoor

Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 network

It's VLAN tag 20, but yes that's the right network.

The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?

Oops, missed the one in the detail pane
Those packet captures are on the WAN side of my firewall, so you're seeing the NAT address. My VLANs are on top of the LAN network, and it all gets NATed to one WAN connection on my firewall.

Do you have a drawing of how this is designed?

No graphic but it's a pretty standard setup I think.
ISP modem -> pfSense device -> switch -> PC
Where the firewall/switch connection is a VLAN trunk and the switch is managed with different VLAN config for each port.

Edit
I do have Snort intrusion detection running on the WAN interface but I have tried disabling it with the same results.
I stripped down as much as possible on my ISP's router so it is not running its packet filter or anything.

virtual-frog

@RobbieTT you're not listening.

The problem is that ICMP messages are arriving on the WAN side of my firewall and not making it to the VLAN side to return to my PC. So my PC just keeps sending packets not knowing someone is unhappy about the MTU.

VLANs are related somehow because I don't get these fragmentation messages without them.

RobbieTT

@virtual-frog

I have no problem knocking on your door with full-sized packets and getting the same in return:

2023-06-14 at 20.20.59.png

Did you try without the switch?

️

virtual-frog

@RobbieTT Right, and I can reach the internet just fine through the switch, on an untagged port. If the switch was a problem, wouldn't the VLAN packet capture still show the ICMP frag messages leaving the firewall? I haven't observed any issues between my PC and firewall.

If nothing else comes up I can try taking out the switch this evening.

michmoor

@virtual-frog

10.0.1.0/24 - LAN1
192.168.20.0/24 - LAN2

For devices on LAN2 network which are having issues

Dont believe this to be the case but are they configured for jumbo frames?
Move devices from LAN2 to LAN1. Are they still having the issue or does it go away?

What is the MTU set on pfSense for each interface.

virtual-frog

@michmoor

I have not configured jumbo frames on any devices, although it seems the switches are configured by default to handle 1518 (the minimum jumbo size).

In your example is LAN1 the parent interface or another VLAN? pfSense MTU and MSS are at defaults now, i.e. 1500/1460.

JKnott

@michmoor said in Adjust MSS calculation to account for VLANs:

I assume VLAN2 on the switch, what is the MTU of the switch ports for devices connected to VLAN2?

Do switches even have an MTU setting? That's a layer 3 concept. Switches have a maximum buffer size, but that's a lot larger than you'll likely find in an MTU. I have a switch with a 16KB buffer size. It will pass any frame up to that size.