Odd MTU / fragmented packet issue on web GUI and haproxy

shewless · Dec 6, 2021, 7:37 PM

Hi,

Long time pfsense user here. Currently on 2.5.2-RELEASE.

I just replaced a few switches with a managed switch in my network. I was able to change my network assignments in pfsense from 3 individual 1G links to 3 tagged vlans (and 1 native) on a single 10G link. It was shockingly easy.

I noticed a weird problem though... on my new managed switch there is a "oversize" counter on my 10G trunk interface and I noticed it keeps increasing. Actually the "error" counter was also increasing but I was able to "fix" that by increasing the MTU of the switch port from 1500->1550.

The MTU of all interfaces and vlans of pfsense is unchanged (1500). Currently the MTU of my switch ports is 1550 because that at least removes the "error" packets - but the "oversize" packets remain.

After some troubleshooting and a network capture I noticed that the "oversize" packets are coming from pfsense port 443!

The "oversize" packets are length 1518 according to wireshark and they say "TCP segment of a reassembled PDU"

If I disconnect from the management interface of pfsense I don't get any oversize packets. Web browsing the Internet, smb file transfer on my LAN, even my truenas web interface.. no oversize packets.

As soon as I load up the pfsense web interface I can see the oversize counters going up. They all behave the same.. when I take any action on the pfsense web interface the oversize counter increases.

I have pfsense with haproxy acting as an ssl terminator for my nextcloud install and I noticed that it also causes the "oversize" counter on my switch to increase.

I did a test and tried using the physical interface with untagged vlans instead of vlan tags and the "oversize" packet problem goes away... but the web UI fragmentation may still be there (I didn't check).

I should mention that I can't notice any functional issues.. everything seems to be working.. but I would really like to understand why this is happening.

I adjusted the MTU of my VLAN interface on pfsense to 1496 temporarily and that appears to have removed the "oversize" packets issue (I guess because the vlan tag adds 4 bytes to the packet).. but the packet capture I take still shows fragmented packages for the web interface.

Any thoughts?

johnpoz · Dec 6, 2021, 8:53 PM

@shewless seems like cosmetic thing in your switch if you ask me.

1518 would be normal, with 4 byte vlan your at 1522.. But where exactly are you seeing this, wireshark will not include the FCS 4 bytes so in like wireshark it would normally show up as 1514

Also with like windows even if your tagging at the interface it would normally strip the tag before wireshark saw it, etc.

shewless · Dec 6, 2021, 9:03 PM

@johnpoz I am using the pfsense "Packet Capture" functionality to create a capture which is showing the vlan encapsulation.

Your screen capture is very helpful because it shows that you have the same "fragmentation" that I have.. and I guess you're saying that is normal.

With my switch port being set to MTU 1500 I was seeing errors counted as well but the errors went away when I set it to 1550. Could this behaviour be explained by the vlan header?

If that is the case I'd be willing to accept "oversize" counter being cosmetic :)

Thank you

johnpoz · Dec 6, 2021, 9:09 PM

@shewless in wireshark I do not believe it would show the FSC or checksum which would be 4 Bytes.. So if your wireshark is showing you the vlan without the FCS it would be 1518, if showing you everything it would be 1522..

If your switch is saying that is oversized - its a cosmetic issue because a normal size would be 1522 with vlans, and 1518 without. And not showing the FCS like wireshark believe removes then yes a normal untagged full sized mtu you would be looking at 1514 like I am seeing, with vlan 1518..

If your switch is showing them as "oversized" its cosmetic.. Now if it was over 1522 then ok..

shewless · Dec 6, 2021, 9:13 PM

@johnpoz thank you for that explanation. I have another question. Since my physical interface and vlan interface in pfsense are set to MTU 1500 (default) how are the packets allowed to be 1522 in size? Does the MTU size not count FSC, checksum, etc?

johnpoz · Dec 6, 2021, 9:39 PM

@shewless the mtu is the data size.. there is also the headers.. 42-1500

You will notice not all of your frames shown in wireshark show 1518, some will be smaller. And sometimes they will show larger even still when your doing windows scaling..

But that not exactly what went on the wire - that is just wireshark putting it together for you, etc.

As to cosmetic issues with switches - the tplink ones that suppose to do vlans, but when you did to vlans their counters showed all the tagged packets as "errors" ;)

shewless · Dec 6, 2021, 9:46 PM

@johnpoz thx. Consider the matter resolved!

johnpoz · Dec 6, 2021, 9:48 PM

@shewless you mind sharing the specific make and model of the switch what firmware on it, etc. I'm just curious if what your seeing is common info, fixed with a firmware/patch, etc.

shewless · Dec 6, 2021, 10:02 PM

@johnpoz absolutely. It's a S3900-24T4S-R (which seems to operate differently than the S3900-24T4S).

I could not find any newer firmware available for this switch nor could I find much documentation regarding this specific issue.

From what I have read different switches consider MTU differently (Ethernet framesize vs IP MTU, etc)

Fiberstore Co., Limited Internetwork Operating System Software
S3900-24T4S-R Series Software, Version 2.2.0E Build 88393, RELEASE SOFTWARE
Copyright (c) 2021 by FS.COM All Rights Reserved

I'd be happy to submit a bug report for this issue if I could understand it more :)

johnpoz · Dec 7, 2021, 1:34 AM

@shewless do you happen to have an example of the error?

What is odd, is here it shows that 1522 is normal size, and anything larger would be jumbo

https://img-en.fs.com/file/user_manual/s3900-series-configuration-guide.pdf
2.21 Jumbo frames
2.21.1 Introduction
Jumbo frames are Ethernet frames with a frame length greater than 1522 bytes.

Could you show status of one of the interfaces your seeing the errors on?

show interfaces status ethernet

shewless · Dec 7, 2021, 1:34 AM

@johnpoz yeah. The switch seems to be a slightly newer model with a bit different syntax:

switch#show interface tg0/25
TGigaEthernet0/25 is up, line protocol is up
  protocolstatus upTimes 1, downTimes 0, last transition 2000-1-1 0:0:21
  Ifindex is 189, unique port number is 49
  Hardware is 10Giga-FX, address is 649d.9928.4a3d (bia 649d.9928.4a3d)
  MTU 1550 bytes, BW 10000000 kbit, DLY 10 usec
  Encapsulation ARPA
  Full-duplex,  10000Mb/s,  Flow-Control Off
  5 minutes input rate 778815 bits/sec, 117 packets/sec
  5 minutes output rate 119461 bits/sec, 88 packets/sec
  Real time input rate 0%, 327494 bits/sec, 100 packets/sec
  Real time output rate 0%, 145816 bits/sec, 95 packets/sec
     Received 10549650 packets, 9541356381 bytes
     2110 broadcasts, 12068 multicasts, 10535472 ucasts
     0 discard, 0 error, 0 PAUSE
     0 align, 0 FCS, 0 symbol
     0 jabber, 318071 oversize, 0 undersize
     0 carriersense, 0 collision, 0 fragment
     0 L3 packets, 0 discards, 0 Header errors
     Transmitted 7473138 packets, 1683821836 bytes
     313229 broadcasts, 112992 multicasts, 7046917 ucasts
     0 discard, 0 error, 0 PAUSE
     0 sqettest, 0 deferred, 277106 oversize
     0 single, 0 multiple, 0 excessive, 0 late
     0 L3 forwards

I tried it on a lesser used port and verify that as soon as I access the web interface the oversize counter goes up.

Here is the config:

interface TGigaEthernet0/25
 switchport mode trunk
 switchport pvid 100

johnpoz · Dec 7, 2021, 2:40 AM

@shewless that is not showing the status.

show interfaces status ethernet tg0/25

You have the mtu on that port set to 1550..

shewless · Dec 7, 2021, 2:48 AM

@johnpoz that command doesn't work on my switch unless I'm in the wrong mode or something? My switch is the "newer" model: The https://www.fs.com/products/134655.html?attribute=8032&id=289447

switch#show interfaces status ethernet tg0/25
show interfaces status ethernet tg0/25
     ^
Too many parameters

switch#show interface ?
  GigaEthernet      -- GigaEthernet interface
  TGigaEthernet     -- Ten GigaEthernet interface
  Vlan              -- VLAN interface
  Null              -- Null interface
  brief             -- brief information of the interface
  range             -- show interface range
  ifindex           -- show interface based on ifindex
    |         -- Output modifiers
    <cr>

switch#show ethernet ?
  cfm  -- Configure Connection Fault Management protocol(CFM)
  oam  -- Operations, Administration and Maintenance

The MTU was set globally. When I look at the manual I only see a way to do it globally for this switch (the -R version):

switch#show system mtu
System MTU size is 1550 bytes

What am I missing?

johnpoz · Dec 7, 2021, 2:50 AM

@shewless I would set it back to 1500, 1550 mtu is not standard by any means.

did you look over that link to the manual - isn't that your switch?

shewless · Dec 7, 2021, 2:51 AM

@johnpoz I can do that (set the MTU to 1500). I think when it was at 1500 I was seeing both "error" counter increases and "oversize" packet increases... I will change it now and see if that is the case.
I totally have been pouring over the manual. As I mentioned the -R version seems to have a different CLI unfortunately.

shewless · Dec 7, 2021, 2:55 AM

@johnpoz yeah as soon as I set the system mtu to 1500 I see both "error" and "oversize" increasing.. though I haven't noticed any functionality problems.. I'd love to get to the bottom of this.

TGigaEthernet0/25 is up, line protocol is up
  protocolstatus upTimes 1, downTimes 0, last transition 2000-1-1 0:0:21
  Ifindex is 189, unique port number is 49
  Hardware is 10Giga-FX, address is 649d.9928.4a3d (bia 649d.9928.4a3d)
  MTU 1500 bytes, BW 10000000 kbit, DLY 10 usec
  Encapsulation ARPA
  Full-duplex,  10000Mb/s,  Flow-Control Off
  5 minutes input rate 202703 bits/sec, 71 packets/sec
  5 minutes output rate 105389 bits/sec, 69 packets/sec
  Real time input rate 0%, 133720 bits/sec, 64 packets/sec
  Real time output rate 0%, 99284 bits/sec, 67 packets/sec
     Received 14056231 packets, 14473843206 bytes
     2951 broadcasts, 12725 multicasts, 14040555 ucasts
     0 discard, 478 error, 0 PAUSE
     0 align, 0 FCS, 0 symbol
     0 jabber, 340230 oversize, 0 undersize
     0 carriersense, 0 collision, 0 fragment
     0 L3 packets, 0 discards, 0 Header errors
     Transmitted 8803287 packets, 1840177991 bytes
     334995 broadcasts, 121700 multicasts, 8346592 ucasts
     0 discard, 0 error, 0 PAUSE
     0 sqettest, 0 deferred, 287250 oversize
     0 single, 0 multiple, 0 excessive, 0 late
     0 L3 forwards

shewless · Dec 7, 2021, 1:42 PM

@johnpoz I submitted a ticket to the switch vendor. I'll update here if I find out anything useful.

johnpoz · Dec 7, 2021, 1:54 PM

@shewless great - that damn curiosity cat of mine is is always meowing.. Yes please let us know what comes of that.

shewless · Dec 8, 2021, 6:19 PM

@johnpoz update:

I verified that on my ubuntu client that this ping works:
ping -c 10 -M do -s 1472 192.168.120.1
But it results in the oversize counter increasing. wireshark shows a packet size of 1514 for both request and reply.

The size of 1468 is required to avoid oversize counter increasing
ping -c 10 -M do -s 1468 192.168.120.1
This results in a wireshark packet size of 1510.

Likely a cosmetic problem as the packets all seam to go where they are supposed to go... a support case is opened for the switch vendor.

johnpoz · Dec 8, 2021, 6:39 PM

@shewless yeah that is odd! for sure.. 1522 should be max size, even says so in the doc I linked too. I could see anything over 1522 triggering the oversize counter, if you didn't have jumbo enabled..