Jumbo frame problem with ESXi 5.5 and vmxnet3 driver



  • Hi

    I have just installed a test pfsense to troubleshoot this issue, version 2.1.3.

    Installed perl and vmware tools as teh docs show how to (https://doc.pfsense.org/index.php/VMware_Tools)

    Reboot, vmx3f0 appears.

    Run ifconfig from the shell MTU shows as 1500.

    Now assign interface, throw an ip on it set MTU to be 9000.

    Allow traffic in firewall.

    Run ifconfig from the shell MTU shows as 9000.

    ping with DF set from either windoze or linux box and with size 1472 and lower i get a response, anything higher and no response.

    Any ideas greatly appreciated

    Jellyhead


  • LAYER 8 Global Moderator

    I don't see where you set the mtu on the vswitch in esxi?

    see attached




  • Hi

    A few other points to mention are I am using Virtual Distributed Switches and I have other hosts that are successfully sending large packets (>1500) to each other.

    I am guessing that this is a vmxnet3 driver issue but I'm stumped.

    Thanks

    JH



  • It's not just you. I tried this on 2.1.4 with ESXi 5.5 using the em driver, and I get the same exact result.

    I did set the MTU to 9000 on my vswitch since I'm not using distributed vswitches (its a single host free esxi)

    Other VM's can send and receive large frames. What I noticed is that PFsense shows the MTU 9000 is configured, but when I do a:

    route get HOST (with HOST representing the IP of the other jumbo frame enabled host), I see the path MTU is set to 1500, regardless of the interface being set to 9000.

    PFsense itself won't reply to jumbo frame pings, nor will it forward jumbo frames across, it fragments or drops, as confirmed by tcpdump.

    Regards,


  • LAYER 8 Global Moderator

    Ok - to test this out, since I am running 2.1.4 i386 on 5.5u1 with vmxnet3 drivers.  I set the mtu in the gui not just ifconfig, its shows the path at 9000, and pings to me (minus 28 bytes for headers) just fine.  See attached screenshots

    You only need to set the mtu on the vswitch if there is actual physical nics tied to the switch.  If your just vm to vm, you don't have to set that.  It would set the mtu on the physical nic if you wanted jumbo to the physical world.  My testing is just vm to vm on the same vswitch.






  • Good to see that it works for you. I wonder what might be different in my case. My LAN interface is VLAN tagged and I have multiple networks trunked on that interface. Is your interface tagged?

    What else could be the problem?


  • LAYER 8 Global Moderator

    I read some issues with vlans - mine is not tagged.  Again this is virtual to virtual not in physical world.  Not sure why you would need to tag vm to vm interfaces?  Or even setup vlans?

    you really should clearly spell out your setup from the get go.. If your running vlan interfaces with tagging, etc. Those kind of details can help people narrow down what your issue is.

    I could try and duplicate your setup..  But did you set the mtu on the gui for your vlan interface?  And also on the physical interface via gui?  When you do the route host thing - what does it show you for the mtu?



  • Ok, here goes:

    Asus RS/100-E70P12
    Intel Xeon E3-1230 3.2ghz
    16GB ram
    Intel 82571EB dual port PCI-Express card

    The server is running ESXi 5.5 1746018.

    The PFsense VM has two cores allocated and 4GB Ram.

    The Dual port Intel NIC serves as both WAN and LAN. The LAN is tagged with 3 VLANS - Guest Wifi, IP Phone, DMZ. The main LAN network is untagged. All tagging is done in ESXi, so tagged networks are passed to PFsense as untagged interfaces. (PFsense is not tagging)

    I setup the MTU to 9000 on the GUI for the LAN and DMZ. When I go to the CLI, I can confirm that the 9000 MTU is set by checking ifconfig.

    When I check with the command route get IPADDRESS for a host on the DMZ configured for 9000 MTU, it shows 1500 MTU. Likewise for any host on the LAN, which is configured for 9000 MTU.

    Other VM's inside this ESXi host successfully use 9000 MTU frames, as do other physical hosts. The successful hosts use the onboard Intel NIC in ESXi to communicate, as all other VM traffic is separated physically from PFsense.

    I have a Cisco SG-300 switch doing VLANs on the physical side, and the port to the PFsense host VM is tagged (tags terminated by ESXi).


  • Banned

    HAve you allowed jumbo frames on the physical switch running vlans??



  • Yes, I allow jumbo frames on the switch, up to the limit of 10000.

    Other physical hosts are able to use jumbo frames across the same switch, and also other VM's can communicate to physical hosts using jumbo frames.

    When a physical host tries to ping its Pfsense default gateway address on its respective subnet with jumbo frames, they either get fragmented or dropped. By that I mean, the jumbo frame echo request arrives at PFsense, but the reply is fragmented, or not sent out at all.

    If one host on the LAN tries to ping a host on the DMZ using jumbo frames, the echo request arrives at PFsense in one frame, but when PFsense forwards it to the other segment it fragments it.

    The same behavior happens on the reverse, DMZ to LAN.


  • LAYER 8 Global Moderator

    "When I check with the command route get IPADDRESS for a host on the DMZ configured for 9000 MTU, it shows 1500 MTU"

    I would say that is the problem..

    "so tagged networks are passed to PFsense as untagged interfaces. (PFsense is not tagging)"

    So pfsense has not vlan interfaces - just what it thinks are just normal interfaces? So I just created a vlan and put in on 192.168.4.0/24 and if i do a route get to an IP on that network it shows 9000 as the mtu

    [2.1.4-RELEASE][root@pfsense]/root(16): route get 192.168.4.42
      route to: 192.168.4.42
    destination: 192.168.4.0
          mask: 255.255.255.0
      interface: vmx3f3_vlan10
          flags: <up,done>recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
          0        0        0        0      9000        1        0</up,done>



  • Yes sir, Pfsense thinks these are normal interfaces. All the VLAN tags terminate in ESXi and are exposed to PFsense as additional interfaces with no tag.

    Would that be a problem?

    I did it this way to make it more flexible, as I can add other VM's to the same networks for other purposes such as sniffing traffic. (I have a tcpdump VM that I send mirrored traffic to monitor)


  • LAYER 8 Global Moderator

    I can add any vm I want to any physical network I want via simple giving it an interface int he vswitch that is tied to that physical network.

    My pfsense thinks these are normal interfaces, if your get route command is showing mtu of 1500, that is going to be a problem.  When I set the mtu in the gui of the pfsense interface - this is when it changed the routing info.  If I just edited the mtu via ifconfig - the interface showed the correct mtu, but the get route command still showed only 1500.  Once I edited the interface in the gui - then the route command showed the 9000 mtu and worked fine.



  • Interesting, I've only made changes via the GUI. Never made changes in the CLI.

    I do see that route get shows the 1500 MTU and agree that is the reason for my problem.

    Not sure how to resolve that though.

    Do you have any ideas?


  • LAYER 8 Global Moderator

    try resetting it in the gui, say back to 1500 and then to 9000 again.



  • Well, it definitely doesn't work for me. I don't have the JUMBO_MTU under ifconfig, even though I do show the 9000 MTU.

    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
    options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether redacted
    inet redacted netmask 0xffffff00 broadcast redacted
    inet6 redacted prefixlen 64
    inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2
    nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

    When I do route get for a host, I see 1500 MTU.

    route get redacted
      route to: redacted
    destination: redacted
          mask: 255.255.255.0
      interface: em1
          flags: <up,done>recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
          0        0        0        0      1500        1        0

    I've even rebooted..

    Should I be using the vmx drivers? Does this not work with EM?

    Regards,</up,done></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast>


  • LAYER 8 Global Moderator

    well that might be an issue..  did you install the open vmware tools, ie the package or the actual tools from vmware?

    I am running the actual tools from vmware for the driver of vmxnet3



  • Yes, I had openVMtools from the Pfsense package repository.

    I've just reworked my setup and I'm now running native Vmtools from VMware as well, and using vmxnet3. Same exact results:

    vmx3f2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
            options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether redacted
            inet redacted netmask 0xffffff00 broadcast redacted
            inet6 redacted prefixlen 64
            inet6 fe80::1:1%vmx3f2 prefixlen 64 scopeid 0x3
            nd6 options=1 <performnud>media: Ethernet 10Gbase-T
            status: active

    [2.1.4-RELEASE][root@pfsense]/root(10): route get redacted
      route to: redacted
    destination: redacted
          mask: 255.255.255.0
      interface: vmx3f2
          flags: <up,done>recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
          0        0        0        0      1500        1        0

    The host (a VM on the same host on another NIC) I'm issuing route get against is configured for MTU of 9000 and can use that MTU when communicating with my NAS.</up,done></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>



  • FreeBSD 8.3 has an issue where changing the MTU doesn't update the MTU of any of the affected routes until you either manually change it, or reboot the system.



  • Okay,

    That's good to know. I have rebooted though, I even rebooted the entire VM host to be sure with no change..

    How do I manually change the MTU for a route? Will that change stick through reboots?



  • Well, new quirk.. If I try to set lower values on the GUI, the actually take properly.

    So if I set the MTU to be 2000, it works.
    If I set it to be 8000, it works.
    If I set it to be 8999, it works.

    However, if I set it to be 9000, it won't take and the previous value is preserved.



  • Well, nevermind that.. if I reboot then the MTU for the route goes back to 1500, yet the interface is set to 8000 (for example).

    I give up. There's clearly something wrong, but I can't figure out what or how to fix it permanently.

    I'll try again on a future release.


  • LAYER 8 Global Moderator

    before you were not even using vmx interface you were on em

    interface: em1

    So that would explain why no jumbo_mtu listed for the interface.

    I have not had to reboot anything to get this working.  You running 32 or 64 bit pfsense?



  • It's 64 bit Pfsense. What could explain the behavior I saw where only the 9000 value for MTU won't take? Any other value works.. at least until I reboot, and then it doesn't.

    In all cases the MTU is set on the interface, but the routes are not flagged with the appropriate MTU when it's set to 9000 or after rebooting if set to another value.



  • I think there are multiple issues at play here. FreeBSD 8.3 has issues in not updating the MTU on routes in jumbo frame scenarios at times, potentially both for the link routes and any associated static routes. Scripting a "route change x.x.x.x/x -mtu 9000" at boot time for all required routes is a suitable workaround where you encounter that and the NIC driver properly handles jumbo frames. vmxnet on 8.3 just doesn't work with jumbo frames from what I've seen. Similar for Broadcom NICs, though those are very reliable and well-performing NICs, in 8.3 they don't work with jumbo frames (an oddity of the hardware that wasn't properly accommodated in the driver at that time, it's since been fixed). These issues have all already been confirmed fixed in FreeBSD 10 and hence 2.2.


  • LAYER 8 Global Moderator

    "vmxnet on 8.3 just doesn't work with jumbo frames"

    Looks to be working to me - what driver did you test with?  I don't really see a need for jumbo in my home network.  It would just complicate the connections and not all devices support them.  I could always segment out even more and put my non jumbo on their own segment, etc.  Seems kind of pointless for what I am doing.

    As you can see from thread I was able to ping at the full mtu - but be happy to do some other testing to validate its working.


Log in to reply