Adjust MSS calculation to account for VLANs

[virtual-frog]

@RobbieTT

(posting this image for the fourth time)

@virtual-frog said in Adjust MSS calculation to account for VLANs:

Ok I have found that the ICMP packets are not making it back to negotiate MTU. See this capture with VLAN on the top and WAN on the bottom.

I compared this capture to one from the LAN (no VLAN) and they have the same opening, but on LAN the connection is successful with no complaints from the server:

I am not sending pings, I am trying to connect to their website with a web browser (TCP). The top picture shows the connection failing from VLAN and the bottom shows it working from LAN (no tag). Same MTU (1500) and MSS (1460) settings for both.

The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?

[RobbieTT]

With pictures:

Just one ping... ok, 1 byte over the regular ping to duckduckgo:

~ % ping -D -s 57 duckduckgo.com
PING duckduckgo.com (52.142.124.215): 57 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5

The packets are dead, they have ceased to be.

One byte smaller:

~ % ping -D -s 56 duckduckgo.com
PING duckduckgo.com (52.142.124.215): 56 data bytes
64 bytes from 52.142.124.215: icmp_seq=0 ttl=116 time=19.785 ms
64 bytes from 52.142.124.215: icmp_seq=1 ttl=116 time=19.598 ms
64 bytes from 52.142.124.215: icmp_seq=2 ttl=116 time=19.523 ms
64 bytes from 52.142.124.215: icmp_seq=3 ttl=116 time=19.381 ms
64 bytes from 52.142.124.215: icmp_seq=4 ttl=116 time=19.540 ms

Like Lazarus, the ping is back.

Wireshark:

Yep, zero response as soon as you are a single byte over a normal ping. No discovery, no MTU negotiation required, no request for fragmentation. In fact no response at all and nothing being blocked by your firewall. The network link is responding to an MSS limit that happened to be dressed in MTU clothing.

️

[michmoor]

@virtual-frog said in Adjust MSS calculation to account for VLANs:

The problem is 1) why do I only get fragmentation messages on VLAN? 2) why are they stopped by the firewall?

Reply

I assume VLAN2 on the switch, what is the MTU of the switch ports for devices connected to VLAN2?

EDIT:
Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 network
The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?
Do you have a drawing of how this is designed?

[virtual-frog]

@michmoor
I'm on VLAN 20. Not sure about MTU settings for this switch (using TP Link Omada SDN so it's a little weird) - there is a jumbo frame option that is set to 1518 so I assume it can at least handle the standard 1500.

I can ping both google and my firewall at 1472. If I turn off the MTU checking flag I can ping my firewall at much larger packet sizes, they will just get fragmented. Note that Google is one of the sites that I don't have issues with.

The connection between my PC and firewall works fine, no fragmentation issues or dropped packets.

[RobbieTT]

@virtual-frog said in Adjust MSS calculation to account for VLANs:

2. why are they stopped by the firewall?

As described earlier, VLANs are stripped away by the router. They do not leave your network. This is not a firewall issue. You need to look internal.

A successful exchange with duckduckgo:

Zero issues with MSS or MTU with full 1500 (1514) byte packets being exchanged with DDG.

️

[virtual-frog]

@michmoor

Help us understand your set up
The VLAN2 i assume is your 192.168.20.0/24 network

It's VLAN tag 20, but yes that's the right network.

The non tagged VLAN that you have blurred out but not completely has a public IP of 162.197.52.166. So is this LAN not behind a switch?

Oops, missed the one in the detail pane
Those packet captures are on the WAN side of my firewall, so you're seeing the NAT address. My VLANs are on top of the LAN network, and it all gets NATed to one WAN connection on my firewall.

Do you have a drawing of how this is designed?

No graphic but it's a pretty standard setup I think.
ISP modem -> pfSense device -> switch -> PC
Where the firewall/switch connection is a VLAN trunk and the switch is managed with different VLAN config for each port.

Edit
I do have Snort intrusion detection running on the WAN interface but I have tried disabling it with the same results.
I stripped down as much as possible on my ISP's router so it is not running its packet filter or anything.

[virtual-frog]

@RobbieTT you're not listening.

The problem is that ICMP messages are arriving on the WAN side of my firewall and not making it to the VLAN side to return to my PC. So my PC just keeps sending packets not knowing someone is unhappy about the MTU.

VLANs are related somehow because I don't get these fragmentation messages without them.

[RobbieTT]

@virtual-frog

I have no problem knocking on your door with full-sized packets and getting the same in return:

Did you try without the switch?

️

[virtual-frog]

@RobbieTT Right, and I can reach the internet just fine through the switch, on an untagged port. If the switch was a problem, wouldn't the VLAN packet capture still show the ICMP frag messages leaving the firewall? I haven't observed any issues between my PC and firewall.

If nothing else comes up I can try taking out the switch this evening.

[michmoor]

@virtual-frog

10.0.1.0/24 - LAN1
192.168.20.0/24 - LAN2

For devices on LAN2 network which are having issues

1. Dont believe this to be the case but are they configured for jumbo frames?
2. Move devices from LAN2 to LAN1. Are they still having the issue or does it go away?

What is the MTU set on pfSense for each interface.

[virtual-frog]

@michmoor

I have not configured jumbo frames on any devices, although it seems the switches are configured by default to handle 1518 (the minimum jumbo size).

In your example is LAN1 the parent interface or another VLAN? pfSense MTU and MSS are at defaults now, i.e. 1500/1460.

[JKnott]

@michmoor said in Adjust MSS calculation to account for VLANs:

I assume VLAN2 on the switch, what is the MTU of the switch ports for devices connected to VLAN2?

Do switches even have an MTU setting? That's a layer 3 concept. Switches have a maximum buffer size, but that's a lot larger than you'll likely find in an MTU. I have a switch with a 16KB buffer size. It will pass any frame up to that size.

[JKnott]

@virtual-frog said in Adjust MSS calculation to account for VLANs:

there is a jumbo frame option that is set to 1518

That's barely frame expansion, to allow things like VLAN tags. Jumbo frames are several KB, with 9000 common.

[virtual-frog]

@JKnott said in Adjust MSS calculation to account for VLANs:

Do switches even have an MTU setting? That's a layer 3 concept.

Well, an L3 switch might.. mine is not so it just has the jumbo frame size

[virtual-frog]

@JKnott said in Adjust MSS calculation to account for VLANs:

@virtual-frog said in Adjust MSS calculation to account for VLANs:

there is a jumbo frame option that is set to 1518

That's barely frame expansion, to allow things like VLAN tags. Jumbo frames are several KB, with 9000 common.

It's the same setting; I can adjust it much higher but haven't set up jumbo frame on my devices yet. The default is 1518

[johnpoz]

@virtual-frog said in Adjust MSS calculation to account for VLANs:

haven't set up jumbo frame on my devices yet.

no point to it either.. You think you having issues now - wait til you try and do that when you have devices that don't support jumbo or have different sizes, etc..

if you had a san network running say 10ge and you had storage devices and say server all on this san network ok then it might make sense - but there is really zero point to trying to do it on a normal gig network.. To be honest I would turn that feature off on your switch..

Not sure where you got the idea that those icmp packets saying to frag would or should go to your end device - how would it??

You are sending those.. Not ddg..

[virtual-frog]

@johnpoz
D'oh. My bad. But that makes even less sense. Why does WAN send the frag message for only packets originating on the VLAN?

[johnpoz]

@virtual-frog you have something all f'd up on that vlan would be my guess. So the mtu on the interface is 1500? What about your clients, are they wired or is wireless involved..

No idea about that tp link when it comes to vlans - they don't seem to quite get it ;) A few years back they had an issue where they would not allow you to remove vlan 1 from any ports.. So if you wanted to run say vlan 10 on that port, you were also untagged in vlan 1.. Took them like 2 years to correct it..

Why does WAN send the frag message for only packets originating on the VLAN?

Its not it sending it because what ddg is sending back.. A full packet seems to be too big for your vlan?? Even though the client sent a mss of 1460.. you can see that from the syn you sent.

[virtual-frog]

@johnpoz

Yes, all MTUs are the default 1500.
Mix of wired and wireless, same effect on both.
TP Link seems to have matured somewhat (still not perfect) so I don't think there's some huge flaw in the switch.
It's strange that this only happens with one site, too.
I may reset and reinstall everything.

[johnpoz]

@virtual-frog I am not having any issues with that site, via tagged vlans.. Than again I am not running on tplink ;)

Did you try just turning off jumbo support?