Jumbo frame problem with ESXi 5.5 and vmxnet3 driver
-
It's not just you. I tried this on 2.1.4 with ESXi 5.5 using the em driver, and I get the same exact result.
I did set the MTU to 9000 on my vswitch since I'm not using distributed vswitches (its a single host free esxi)
Other VM's can send and receive large frames. What I noticed is that PFsense shows the MTU 9000 is configured, but when I do a:
route get HOST (with HOST representing the IP of the other jumbo frame enabled host), I see the path MTU is set to 1500, regardless of the interface being set to 9000.
PFsense itself won't reply to jumbo frame pings, nor will it forward jumbo frames across, it fragments or drops, as confirmed by tcpdump.
Regards,
-
Ok - to test this out, since I am running 2.1.4 i386 on 5.5u1 with vmxnet3 drivers. I set the mtu in the gui not just ifconfig, its shows the path at 9000, and pings to me (minus 28 bytes for headers) just fine. See attached screenshots
You only need to set the mtu on the vswitch if there is actual physical nics tied to the switch. If your just vm to vm, you don't have to set that. It would set the mtu on the physical nic if you wanted jumbo to the physical world. My testing is just vm to vm on the same vswitch.
-
Good to see that it works for you. I wonder what might be different in my case. My LAN interface is VLAN tagged and I have multiple networks trunked on that interface. Is your interface tagged?
What else could be the problem?
-
I read some issues with vlans - mine is not tagged. Again this is virtual to virtual not in physical world. Not sure why you would need to tag vm to vm interfaces? Or even setup vlans?
you really should clearly spell out your setup from the get go.. If your running vlan interfaces with tagging, etc. Those kind of details can help people narrow down what your issue is.
I could try and duplicate your setup.. But did you set the mtu on the gui for your vlan interface? And also on the physical interface via gui? When you do the route host thing - what does it show you for the mtu?
-
Ok, here goes:
Asus RS/100-E70P12
Intel Xeon E3-1230 3.2ghz
16GB ram
Intel 82571EB dual port PCI-Express cardThe server is running ESXi 5.5 1746018.
The PFsense VM has two cores allocated and 4GB Ram.
The Dual port Intel NIC serves as both WAN and LAN. The LAN is tagged with 3 VLANS - Guest Wifi, IP Phone, DMZ. The main LAN network is untagged. All tagging is done in ESXi, so tagged networks are passed to PFsense as untagged interfaces. (PFsense is not tagging)
I setup the MTU to 9000 on the GUI for the LAN and DMZ. When I go to the CLI, I can confirm that the 9000 MTU is set by checking ifconfig.
When I check with the command route get IPADDRESS for a host on the DMZ configured for 9000 MTU, it shows 1500 MTU. Likewise for any host on the LAN, which is configured for 9000 MTU.
Other VM's inside this ESXi host successfully use 9000 MTU frames, as do other physical hosts. The successful hosts use the onboard Intel NIC in ESXi to communicate, as all other VM traffic is separated physically from PFsense.
I have a Cisco SG-300 switch doing VLANs on the physical side, and the port to the PFsense host VM is tagged (tags terminated by ESXi).
-
HAve you allowed jumbo frames on the physical switch running vlans??
-
Yes, I allow jumbo frames on the switch, up to the limit of 10000.
Other physical hosts are able to use jumbo frames across the same switch, and also other VM's can communicate to physical hosts using jumbo frames.
When a physical host tries to ping its Pfsense default gateway address on its respective subnet with jumbo frames, they either get fragmented or dropped. By that I mean, the jumbo frame echo request arrives at PFsense, but the reply is fragmented, or not sent out at all.
If one host on the LAN tries to ping a host on the DMZ using jumbo frames, the echo request arrives at PFsense in one frame, but when PFsense forwards it to the other segment it fragments it.
The same behavior happens on the reverse, DMZ to LAN.
-
"When I check with the command route get IPADDRESS for a host on the DMZ configured for 9000 MTU, it shows 1500 MTU"
I would say that is the problem..
"so tagged networks are passed to PFsense as untagged interfaces. (PFsense is not tagging)"
So pfsense has not vlan interfaces - just what it thinks are just normal interfaces? So I just created a vlan and put in on 192.168.4.0/24 and if i do a route get to an IP on that network it shows 9000 as the mtu
[2.1.4-RELEASE][root@pfsense]/root(16): route get 192.168.4.42
route to: 192.168.4.42
destination: 192.168.4.0
mask: 255.255.255.0
interface: vmx3f3_vlan10
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 9000 1 0</up,done> -
Yes sir, Pfsense thinks these are normal interfaces. All the VLAN tags terminate in ESXi and are exposed to PFsense as additional interfaces with no tag.
Would that be a problem?
I did it this way to make it more flexible, as I can add other VM's to the same networks for other purposes such as sniffing traffic. (I have a tcpdump VM that I send mirrored traffic to monitor)
-
I can add any vm I want to any physical network I want via simple giving it an interface int he vswitch that is tied to that physical network.
My pfsense thinks these are normal interfaces, if your get route command is showing mtu of 1500, that is going to be a problem. When I set the mtu in the gui of the pfsense interface - this is when it changed the routing info. If I just edited the mtu via ifconfig - the interface showed the correct mtu, but the get route command still showed only 1500. Once I edited the interface in the gui - then the route command showed the 9000 mtu and worked fine.
-
Interesting, I've only made changes via the GUI. Never made changes in the CLI.
I do see that route get shows the 1500 MTU and agree that is the reason for my problem.
Not sure how to resolve that though.
Do you have any ideas?
-
try resetting it in the gui, say back to 1500 and then to 9000 again.
-
Well, it definitely doesn't work for me. I don't have the JUMBO_MTU under ifconfig, even though I do show the 9000 MTU.
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether redacted
inet redacted netmask 0xffffff00 broadcast redacted
inet6 redacted prefixlen 64
inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2
nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
status: activeWhen I do route get for a host, I see 1500 MTU.
route get redacted
route to: redacted
destination: redacted
mask: 255.255.255.0
interface: em1
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0I've even rebooted..
Should I be using the vmx drivers? Does this not work with EM?
Regards,</up,done></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,simplex,multicast>
-
well that might be an issue.. did you install the open vmware tools, ie the package or the actual tools from vmware?
I am running the actual tools from vmware for the driver of vmxnet3
-
Yes, I had openVMtools from the Pfsense package repository.
I've just reworked my setup and I'm now running native Vmtools from VMware as well, and using vmxnet3. Same exact results:
vmx3f2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 9000
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether redacted
inet redacted netmask 0xffffff00 broadcast redacted
inet6 redacted prefixlen 64
inet6 fe80::1:1%vmx3f2 prefixlen 64 scopeid 0x3
nd6 options=1 <performnud>media: Ethernet 10Gbase-T
status: active[2.1.4-RELEASE][root@pfsense]/root(10): route get redacted
route to: redacted
destination: redacted
mask: 255.255.255.0
interface: vmx3f2
flags: <up,done>recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0The host (a VM on the same host on another NIC) I'm issuing route get against is configured for MTU of 9000 and can use that MTU when communicating with my NAS.</up,done></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
-
FreeBSD 8.3 has an issue where changing the MTU doesn't update the MTU of any of the affected routes until you either manually change it, or reboot the system.
-
Okay,
That's good to know. I have rebooted though, I even rebooted the entire VM host to be sure with no change..
How do I manually change the MTU for a route? Will that change stick through reboots?
-
Well, new quirk.. If I try to set lower values on the GUI, the actually take properly.
So if I set the MTU to be 2000, it works.
If I set it to be 8000, it works.
If I set it to be 8999, it works.However, if I set it to be 9000, it won't take and the previous value is preserved.
-
Well, nevermind that.. if I reboot then the MTU for the route goes back to 1500, yet the interface is set to 8000 (for example).
I give up. There's clearly something wrong, but I can't figure out what or how to fix it permanently.
I'll try again on a future release.
-
before you were not even using vmx interface you were on em
interface: em1
So that would explain why no jumbo_mtu listed for the interface.
I have not had to reboot anything to get this working. You running 32 or 64 bit pfsense?
