Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Supermicro A1SRi-2758F Jumbo Frames/MTU Limited to 4078?

    Scheduled Pinned Locked Moved Hardware
    7 Posts 4 Posters 2.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      asayler
      last edited by

      Hi All,

      I'm using a Supermicro A1SRi-2758F running pfSense 2.2. This board contains an Intel Atom C2758 Rangeley chip with 4x Gigabit Ethernet ports supported by a Marvell 88E1543 transceiver. I currently have the system set to use one port for the incoming WAN connection and one-port for a VLAN-tagged connection to my switch:

      
 WAN (wan)       -> igb0       -> v4: [REDACTED]
 MGMT (lan)      -> igb1_vlan3 -> v4: 192.168.3.1/24
 DMZ (opt1)      -> igb1_vlan6 -> v4: 192.168.6.1/24
 LAN (opt2)      -> igb1_vlan9 -> v4: 192.168.9.1/24

      I'm attempting to get everything working with 9K MTU Jumbo frames, but am running into an issue where setting the vlan interfaces to any MTU value larger then 4078 causes the interfaces to loss all connectivity (no ping, etc). MTUs of 4078 or smaller work fine (including the default of 1500), but anything over 4078 and it's no good:

      
[2.2-RELEASE][admin@REDACTED]/root: ifconfig
igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=407bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso>ether [REDACTED]
	inet [REDACTED] netmask [REDACTED] broadcast [REDACTED] 
	inet6 [REDACTED]%igb0 prefixlen 64 scopeid 0x1 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 4078
	options=507bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso>ether [REDACTED]
	inet6 [REDACTED]%igb1 prefixlen 64 scopeid 0x2 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb2: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether [REDACTED]
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
	status: no carrier
igb3: flags=8c02 <broadcast,oactive,simplex,multicast>metric 0 mtu 1500
	options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether [REDACTED]
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
	status: no carrier
...
igb1_vlan3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 4078
	options=303 <rxcsum,txcsum,tso4,tso6>ether [REDACTED]
	inet6 [REDACTED]%igb1_vlan3 prefixlen 64 scopeid 0x9 
	inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 3 vlanpcp: 0 parent interface: igb1
igb1_vlan6: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 4078
	options=303 <rxcsum,txcsum,tso4,tso6>ether [REDACTED]
	inet6 [REDACTED]%igb1_vlan6 prefixlen 64 scopeid 0xa 
	inet 192.168.6.1 netmask 0xffffff00 broadcast 192.168.6.255 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 6 vlanpcp: 0 parent interface: igb1
igb1_vlan9: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 4078
	options=303 <rxcsum,txcsum,tso4,tso6>ether [REDACTED]
	inet6 [REDACTED]%igb1_vlan9 prefixlen 64 scopeid 0xb 
	inet 192.168.9.1 netmask 0xffffff00 broadcast 192.168.9.255 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 9 vlanpcp: 0 parent interface: igb1
...</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,oactive,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,oactive,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwfilter,vlan_hwtso></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,lro,vlan_hwtso></up,broadcast,running,simplex,multicast>

      The above configuration works fine, but any move to a larger MTU causes a complete lack of connectivity on the corresponding ports/vlans:

      
$ ping 192.168.3.1 -M do -s 4050
PING 192.168.3.1 (192.168.3.1) 4050(4078) bytes of data.
4058 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.726 ms
4058 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.750 ms
4058 bytes from 192.168.3.1: icmp_seq=3 ttl=64 time=0.678 ms

      The interesting thing about all of this is that I had previously been playing around with pfSense 2.2 on a Lanner FW-7551 with a processor from the same Rangely line (C2358) and the same Marvell transceiver. That board worked fine with MTUs up to 9K. So the processor/SoC itself seems capable of handling larger MTUs.

      Is this just a limitation of the Supermicro hardware/firmware? Has anyone had luck making the A1SRi-2758F ports works with MTUs above 4078? Maybe it's an issue with using vlans + jumbo frames (again, this worked fine on the Lanner board). I also have the various hardware offload options turned on, but that also worked fine on the C2358-based Lanner board.

      I'm currently using version 1.0 of the A1SRi-2758F firmware, and I saw that v1.1 was recently released. I may try updating to that just in case it's related, but I'm not holding my breath.

      Thoughts?

      1 Reply Last reply Reply Quote 0
      • A
        asayler
        last edited by

        I have a colleague who has this same Supermicro board but who runs Linux on it repeat my tests. He was able to get MTUs up to 9K working fine, which would seem to suggest it's not a hardware limitation:

        
$ ip link
...
6: eth2: <broadcast,multicast,up,lower_up>mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global eth3
24: eth2.100@eth2: <broadcast,multicast,up,lower_up>mtu 9000 qdisc noqueue state UP group default 
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.1/24 brd 10.100.0.255 scope global eth2.100
...</broadcast,multicast,up,lower_up></broadcast,multicast,up,lower_up> 

        
$ ping 10.100.0.2 -M do -s 8972
PING 10.100.0.2 (10.100.0.2) 8972 (9000) bytes of data.
8980 bytes from 10.100.0.2: icmp_seq=1 ttl=64 time=0.841 ms
8980 bytes from 10.100.0.2: icmp_seq=2 ttl=64 time=0.645 ms
8980 bytes from 10.100.0.2: icmp_seq=3 ttl=64 time=0.666 ms

        So it looks like it's either:

        1. An issue with my particular setup/config
        2. An issue with pfSense or the underlying FreeBSD driver/network stack

        The fact that I had the Lanner board working fine with the same config and using the same NICs/driver would seem to rule out both of these options, and yet the issue remains.

        I'm curious to hear if anyone else has gotten larger MTUs working properly on this board under pfSense 2.2.

        As an addendum, my testing setup involves a direct connection between the Supermicro NIC and my desktop NIC, so there shouldn't be any other switches, etc interfering.

        1 Reply Last reply Reply Quote 0
        • H
          heper
          last edited by

          theres an open bug that might be related: https://redmine.pfsense.org/issues/4397

          could you try setting the mtu for the routes manually to see if that fixes it ?

          1 Reply Last reply Reply Quote 0
          • A
            asayler
            last edited by

            @heper:

            theres an open bug that might be related: https://redmine.pfsense.org/issues/4397

            could you try setting the mtu for the routes manually to see if that fixes it ?

            I tried running the manual ifconfig commands suggested in that bug report. They didn't seem to have any effect. Also, the routing table MTUs look correct, even before running the manual ifconfig commands:

            
[2.2-RELEASE][admin@REDACTED]/root: netstat -rW
Routing tables

Internet:
Destination        Gateway            Flags       Use    Mtu      Netif Expire
...
localhost          link#7             UH          212  16384        lo0
192.168.3.0        link#9             U             0   9000 igb1_vlan3
192.168.3.1        link#9             UHS           0  16384        lo0
192.168.6.0        link#10            U          1061   9000 igb1_vlan6
192.168.6.1        link#10            UHS           0  16384        lo0
192.168.9.0        link#11            U            83   9000 igb1_vlan9
192.168.9.1        link#11            UHS           0  16384        lo0
            1 Reply Last reply Reply Quote 0
            • A
              asayler
              last edited by

              As mentioned previously, setting the MTU to any value above 4078 leads to total interface failure, even for packets smaller than the MTU. I.e., when I set the MTU to 9000, all packets fail, even those of smaller size (e.g. 100 bytes).

              I did some more testing and it looks like all incoming traffic to any interface with an MTU > 4078 is getting dropped somewhere. Outgoing traffic seems okay. For example, if I set an interface on pfSense to an MTU of 9000 and try to ping a host reachable via that interface, the host recives the ARP request from pfSense and then sends an ARP response, but pfSense never receives this response:

              Ping from pfSense (192.168.9.1) to Host (192.168.3.3):

              
[2.2-RELEASE][admin@REDACTED]/root: ping 192.168.9.3
PING 192.168.9.3 (192.168.9.3): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down

              Corresponding tcpdump on pfSense:

              
[2.2-RELEASE][admin@REDACTED]/root: tcpdump -i igb1_vlan9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1_vlan9, link-type EN10MB (Ethernet), capture size 65535 bytes
12:52:15.816219 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 28
12:52:16.840044 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 28
12:52:17.861140 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 28

              Corresponding tcpdump on host:

              
$ sudo tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:52:15.807888 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 46
12:52:15.807904 ARP, Reply 192.168.9.3 is-at [REDACTED] (oui Unknown), length 28
12:52:16.831666 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 46
12:52:16.831681 ARP, Reply 192.168.9.3 is-at [REDACTED] (oui Unknown), length 28
12:52:17.852794 ARP, Request who-has 192.168.9.3 tell 192.168.9.1, length 46
12:52:17.852810 ARP, Reply 192.168.9.3 is-at [REDACTED] (oui Unknown), length 28

              Changing the MTU back to a value of 4078 or smaller returns everything to proper working order.

              Any thoughts on why all incoming traffic gets dropped for any interface with an MTU > 4078?

              1 Reply Last reply Reply Quote 0
              • J
                josh4trunks
                last edited by

                I hit this today as well on pfsense 2.2.6. According to the igb manpage, and in my experience with a FreeBSD 10 system, igb interfaces should be able to use an MTU up to 9216. But weirdly anything above 4078 does not work =/

                But, maybe it is a limit with this particular NIC? See here…
                https://bugs.launchpad.net/ubuntu/+source/linux/+bug/715331
                https://bbs.archlinux.org/viewtopic.php?id=107869

                1 Reply Last reply Reply Quote 0
                • ?
                  Guest
                  last edited by

                  @josh4trunks
                  This is a very old threat here and this problem is resolved in or since pfSense version 2.3

                  Redmine Bug #4397
                  The fix above noted in "do control plane MTU tracking" is in 2.3/10-STABLE and works, which fixes this.

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.