Jumbo frame problem with ESXi 5.5 and vmxnet3 driver
-
HAve you allowed jumbo frames on the physical switch running vlans??
-
Yes, I allow jumbo frames on the switch, up to the limit of 10000.
Other physical hosts are able to use jumbo frames across the same switch, and also other VM's can communicate to physical hosts using jumbo frames.
When a physical host tries to ping its Pfsense default gateway address on its respective subnet with jumbo frames, they either get fragmented or dropped. By that I mean, the jumbo frame echo request arrives at PFsense, but the reply is fragmented, or not sent out at all.
If one host on the LAN tries to ping a host on the DMZ using jumbo frames, the echo request arrives at PFsense in one frame, but when PFsense forwards it to the other segment it fragments it.
The same behavior happens on the reverse, DMZ to LAN.
-
"When I check with the command route get IPADDRESS for a host on the DMZ configured for 9000 MTU, it shows 1500 MTU"
I would say that is the problem..
"so tagged networks are passed to PFsense as untagged interfaces. (PFsense is not tagging)"
So pfsense has not vlan interfaces - just what it thinks are just normal interfaces? So I just created a vlan and put in on 192.168.4.0/24 and if i do a route get to an IP on that network it shows 9000 as the mtu
[2.1.4-RELEASE][root@pfsense]/root(16): route get 192.168.4.42
route to: 192.168.4.42
destination: 192.168.4.0
mask: 255.255.255.0
interface: vmx3f3_vlan10
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 9000 1 0</up,done> -
Yes sir, Pfsense thinks these are normal interfaces. All the VLAN tags terminate in ESXi and are exposed to PFsense as additional interfaces with no tag.
Would that be a problem?
I did it this way to make it more flexible, as I can add other VM's to the same networks for other purposes such as sniffing traffic. (I have a tcpdump VM that I send mirrored traffic to monitor)
-
I can add any vm I want to any physical network I want via simple giving it an interface int he vswitch that is tied to that physical network.
My pfsense thinks these are normal interfaces, if your get route command is showing mtu of 1500, that is going to be a problem. When I set the mtu in the gui of the pfsense interface - this is when it changed the routing info. If I just edited the mtu via ifconfig - the interface showed the correct mtu, but the get route command still showed only 1500. Once I edited the interface in the gui - then the route command showed the 9000 mtu and worked fine.
-
Interesting, I've only made changes via the GUI. Never made changes in the CLI.
I do see that route get shows the 1500 MTU and agree that is the reason for my problem.
Not sure how to resolve that though.
Do you have any ideas?
-
try resetting it in the gui, say back to 1500 and then to 9000 again.
-
Well, it definitely doesn't work for me. I don't have the JUMBO_MTU under ifconfig, even though I do show the 9000 MTU.
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether redacted
inet redacted netmask 0xffffff00 broadcast redacted
inet6 redacted prefixlen 64
inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: activeWhen I do route get for a host, I see 1500 MTU.
route get redacted
route to: redacted
destination: redacted
mask: 255.255.255.0
interface: em1
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0I've even rebooted..
Should I be using the vmx drivers? Does this not work with EM?
Regards,</up,done></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast>
-
well that might be an issue.. did you install the open vmware tools, ie the package or the actual tools from vmware?
I am running the actual tools from vmware for the driver of vmxnet3
-
Yes, I had openVMtools from the Pfsense package repository.
I've just reworked my setup and I'm now running native Vmtools from VMware as well, and using vmxnet3. Same exact results:
vmx3f2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether redacted
inet redacted netmask 0xffffff00 broadcast redacted
inet6 redacted prefixlen 64
inet6 fe80::1:1%vmx3f2 prefixlen 64 scopeid 0x3
nd6 options=1 <performnud>media: Ethernet 10Gbase-T
status: active[2.1.4-RELEASE][root@pfsense]/root(10): route get redacted
route to: redacted
destination: redacted
mask: 255.255.255.0
interface: vmx3f2
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0The host (a VM on the same host on another NIC) I'm issuing route get against is configured for MTU of 9000 and can use that MTU when communicating with my NAS.</up,done></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
-
FreeBSD 8.3 has an issue where changing the MTU doesn't update the MTU of any of the affected routes until you either manually change it, or reboot the system.
-
Okay,
That's good to know. I have rebooted though, I even rebooted the entire VM host to be sure with no change..
How do I manually change the MTU for a route? Will that change stick through reboots?
-
Well, new quirk.. If I try to set lower values on the GUI, the actually take properly.
So if I set the MTU to be 2000, it works.
If I set it to be 8000, it works.
If I set it to be 8999, it works.However, if I set it to be 9000, it won't take and the previous value is preserved.
-
Well, nevermind that.. if I reboot then the MTU for the route goes back to 1500, yet the interface is set to 8000 (for example).
I give up. There's clearly something wrong, but I can't figure out what or how to fix it permanently.
I'll try again on a future release.
-
before you were not even using vmx interface you were on em
interface: em1
So that would explain why no jumbo_mtu listed for the interface.
I have not had to reboot anything to get this working. You running 32 or 64 bit pfsense?
-
It's 64 bit Pfsense. What could explain the behavior I saw where only the 9000 value for MTU won't take? Any other value works.. at least until I reboot, and then it doesn't.
In all cases the MTU is set on the interface, but the routes are not flagged with the appropriate MTU when it's set to 9000 or after rebooting if set to another value.
-
I think there are multiple issues at play here. FreeBSD 8.3 has issues in not updating the MTU on routes in jumbo frame scenarios at times, potentially both for the link routes and any associated static routes. Scripting a "route change x.x.x.x/x -mtu 9000" at boot time for all required routes is a suitable workaround where you encounter that and the NIC driver properly handles jumbo frames. vmxnet on 8.3 just doesn't work with jumbo frames from what I've seen. Similar for Broadcom NICs, though those are very reliable and well-performing NICs, in 8.3 they don't work with jumbo frames (an oddity of the hardware that wasn't properly accommodated in the driver at that time, it's since been fixed). These issues have all already been confirmed fixed in FreeBSD 10 and hence 2.2.
-
"vmxnet on 8.3 just doesn't work with jumbo frames"
Looks to be working to me - what driver did you test with? I don't really see a need for jumbo in my home network. It would just complicate the connections and not all devices support them. I could always segment out even more and put my non jumbo on their own segment, etc. Seems kind of pointless for what I am doing.
As you can see from thread I was able to ping at the full mtu - but be happy to do some other testing to validate its working.
