Odd MTU / fragmented packet issue on web GUI and haproxy

shewless · Dec 6, 2021, 9:13 PM

@johnpoz thank you for that explanation. I have another question. Since my physical interface and vlan interface in pfsense are set to MTU 1500 (default) how are the packets allowed to be 1522 in size? Does the MTU size not count FSC, checksum, etc?

johnpoz · Dec 6, 2021, 9:39 PM

@shewless the mtu is the data size.. there is also the headers.. 42-1500

You will notice not all of your frames shown in wireshark show 1518, some will be smaller. And sometimes they will show larger even still when your doing windows scaling..

But that not exactly what went on the wire - that is just wireshark putting it together for you, etc.

As to cosmetic issues with switches - the tplink ones that suppose to do vlans, but when you did to vlans their counters showed all the tagged packets as "errors" ;)

shewless · Dec 6, 2021, 9:46 PM

@johnpoz thx. Consider the matter resolved!

johnpoz · Dec 6, 2021, 9:48 PM

@shewless you mind sharing the specific make and model of the switch what firmware on it, etc. I'm just curious if what your seeing is common info, fixed with a firmware/patch, etc.

shewless · Dec 6, 2021, 10:02 PM

@johnpoz absolutely. It's a S3900-24T4S-R (which seems to operate differently than the S3900-24T4S).

I could not find any newer firmware available for this switch nor could I find much documentation regarding this specific issue.

From what I have read different switches consider MTU differently (Ethernet framesize vs IP MTU, etc)

Fiberstore Co., Limited Internetwork Operating System Software
S3900-24T4S-R Series Software, Version 2.2.0E Build 88393, RELEASE SOFTWARE
Copyright (c) 2021 by FS.COM All Rights Reserved

I'd be happy to submit a bug report for this issue if I could understand it more :)

johnpoz · Dec 7, 2021, 1:34 AM

@shewless do you happen to have an example of the error?

What is odd, is here it shows that 1522 is normal size, and anything larger would be jumbo

https://img-en.fs.com/file/user_manual/s3900-series-configuration-guide.pdf
2.21 Jumbo frames
2.21.1 Introduction
Jumbo frames are Ethernet frames with a frame length greater than 1522 bytes.

Could you show status of one of the interfaces your seeing the errors on?

show interfaces status ethernet

shewless · Dec 7, 2021, 1:34 AM

@johnpoz yeah. The switch seems to be a slightly newer model with a bit different syntax:

switch#show interface tg0/25
TGigaEthernet0/25 is up, line protocol is up
  protocolstatus upTimes 1, downTimes 0, last transition 2000-1-1 0:0:21
  Ifindex is 189, unique port number is 49
  Hardware is 10Giga-FX, address is 649d.9928.4a3d (bia 649d.9928.4a3d)
  MTU 1550 bytes, BW 10000000 kbit, DLY 10 usec
  Encapsulation ARPA
  Full-duplex,  10000Mb/s,  Flow-Control Off
  5 minutes input rate 778815 bits/sec, 117 packets/sec
  5 minutes output rate 119461 bits/sec, 88 packets/sec
  Real time input rate 0%, 327494 bits/sec, 100 packets/sec
  Real time output rate 0%, 145816 bits/sec, 95 packets/sec
     Received 10549650 packets, 9541356381 bytes
     2110 broadcasts, 12068 multicasts, 10535472 ucasts
     0 discard, 0 error, 0 PAUSE
     0 align, 0 FCS, 0 symbol
     0 jabber, 318071 oversize, 0 undersize
     0 carriersense, 0 collision, 0 fragment
     0 L3 packets, 0 discards, 0 Header errors
     Transmitted 7473138 packets, 1683821836 bytes
     313229 broadcasts, 112992 multicasts, 7046917 ucasts
     0 discard, 0 error, 0 PAUSE
     0 sqettest, 0 deferred, 277106 oversize
     0 single, 0 multiple, 0 excessive, 0 late
     0 L3 forwards

I tried it on a lesser used port and verify that as soon as I access the web interface the oversize counter goes up.

Here is the config:

interface TGigaEthernet0/25
 switchport mode trunk
 switchport pvid 100

johnpoz · Dec 7, 2021, 2:40 AM

@shewless that is not showing the status.

show interfaces status ethernet tg0/25

You have the mtu on that port set to 1550..

shewless · Dec 7, 2021, 2:48 AM

@johnpoz that command doesn't work on my switch unless I'm in the wrong mode or something? My switch is the "newer" model: The https://www.fs.com/products/134655.html?attribute=8032&id=289447

switch#show interfaces status ethernet tg0/25
show interfaces status ethernet tg0/25
     ^
Too many parameters

switch#show interface ?
  GigaEthernet      -- GigaEthernet interface
  TGigaEthernet     -- Ten GigaEthernet interface
  Vlan              -- VLAN interface
  Null              -- Null interface
  brief             -- brief information of the interface
  range             -- show interface range
  ifindex           -- show interface based on ifindex
    |         -- Output modifiers
    <cr>

switch#show ethernet ?
  cfm  -- Configure Connection Fault Management protocol(CFM)
  oam  -- Operations, Administration and Maintenance

The MTU was set globally. When I look at the manual I only see a way to do it globally for this switch (the -R version):

switch#show system mtu
System MTU size is 1550 bytes

What am I missing?

johnpoz · Dec 7, 2021, 2:50 AM

@shewless I would set it back to 1500, 1550 mtu is not standard by any means.

did you look over that link to the manual - isn't that your switch?

shewless · Dec 7, 2021, 2:51 AM

@johnpoz I can do that (set the MTU to 1500). I think when it was at 1500 I was seeing both "error" counter increases and "oversize" packet increases... I will change it now and see if that is the case.
I totally have been pouring over the manual. As I mentioned the -R version seems to have a different CLI unfortunately.

shewless · Dec 7, 2021, 2:55 AM

@johnpoz yeah as soon as I set the system mtu to 1500 I see both "error" and "oversize" increasing.. though I haven't noticed any functionality problems.. I'd love to get to the bottom of this.

TGigaEthernet0/25 is up, line protocol is up
  protocolstatus upTimes 1, downTimes 0, last transition 2000-1-1 0:0:21
  Ifindex is 189, unique port number is 49
  Hardware is 10Giga-FX, address is 649d.9928.4a3d (bia 649d.9928.4a3d)
  MTU 1500 bytes, BW 10000000 kbit, DLY 10 usec
  Encapsulation ARPA
  Full-duplex,  10000Mb/s,  Flow-Control Off
  5 minutes input rate 202703 bits/sec, 71 packets/sec
  5 minutes output rate 105389 bits/sec, 69 packets/sec
  Real time input rate 0%, 133720 bits/sec, 64 packets/sec
  Real time output rate 0%, 99284 bits/sec, 67 packets/sec
     Received 14056231 packets, 14473843206 bytes
     2951 broadcasts, 12725 multicasts, 14040555 ucasts
     0 discard, 478 error, 0 PAUSE
     0 align, 0 FCS, 0 symbol
     0 jabber, 340230 oversize, 0 undersize
     0 carriersense, 0 collision, 0 fragment
     0 L3 packets, 0 discards, 0 Header errors
     Transmitted 8803287 packets, 1840177991 bytes
     334995 broadcasts, 121700 multicasts, 8346592 ucasts
     0 discard, 0 error, 0 PAUSE
     0 sqettest, 0 deferred, 287250 oversize
     0 single, 0 multiple, 0 excessive, 0 late
     0 L3 forwards

shewless · Dec 7, 2021, 1:42 PM

@johnpoz I submitted a ticket to the switch vendor. I'll update here if I find out anything useful.

johnpoz · Dec 7, 2021, 1:54 PM

@shewless great - that damn curiosity cat of mine is is always meowing.. Yes please let us know what comes of that.

shewless · Dec 8, 2021, 6:19 PM

@johnpoz update:

I verified that on my ubuntu client that this ping works:
ping -c 10 -M do -s 1472 192.168.120.1
But it results in the oversize counter increasing. wireshark shows a packet size of 1514 for both request and reply.

The size of 1468 is required to avoid oversize counter increasing
ping -c 10 -M do -s 1468 192.168.120.1
This results in a wireshark packet size of 1510.

Likely a cosmetic problem as the packets all seam to go where they are supposed to go... a support case is opened for the switch vendor.

johnpoz · Dec 8, 2021, 6:39 PM

@shewless yeah that is odd! for sure.. 1522 should be max size, even says so in the doc I linked too. I could see anything over 1522 triggering the oversize counter, if you didn't have jumbo enabled..

shewless · Dec 13, 2021, 2:52 PM

@johnpoz Update:
Switch vendor believes it's okay to have to update the system MTU to a higher value (in order to remove the error counter) and to live with the "oversize" counter increases.
I have since set the system MTU to 9000 (which is the L2 MTU). Any IP Interface I create can have an L3 MTU applied to it of 1500.

Thanks for the help.

johnpoz · Dec 13, 2021, 3:00 PM

@shewless said in Odd MTU / fragmented packet issue on web GUI and haproxy:

it's okay to have to update the system MTU to a higher value (in order to remove the error counter)

Yeah that is not the correct solution.. Mismatched mtu is never a good thing..

shewless · Dec 13, 2021, 3:34 PM

@johnpoz I agree that this is not the correct solution and I have let the switch vendor know. Maybe if other potential customers provide feedback they'll make changes.
In my case I don't think the MTU settings will be harmful. I've already tested that my endpoint devices all have "Layer3" MTU of 1500 and having the switch system MTU set to a higher value doesn't impact the traffic being sent.

johnpoz · Dec 13, 2021, 3:36 PM

@shewless Personally I would just live with the cosmetic errors being listed most likely vs setting a clearly mismatched mtu.

As long as its not actually dropping the traffic, etc.