Inter-vlan traffic is rate limited as VM
-
You're using VMX NICs in ESXi?
Did you apply the recommended tuning?
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfaces -
@stephenw10, evening. Thanks and yes, set against both vmx0 & vmx1 for the interface carrying the nine Vlans and the other for the WAN.
Pops
-
They were already set or you just set them now? Probably need to reboot to apply if you did.
-
@stephenw10 no, been set as part of the VM transition a week ago.
-
Hmm, do you see the same throttling if you test to or from the firewall directly?
-
@stephenw10 Good question and no, it works normally as expected. There are no traffic issues or any signs of throttling on the management interface or other devices on the same management Vlan. But traversing beyond the L2 domain into another vlan and wham, the problem occurs.
-
Is it a 'hard' limit? If you look at the traffic graphs is it flat or spikes?
It 'feels' like it could be an asymmetric routing issue. If so it would be very spikey.
-
@stephenw10 Hi, very spikey. The snapshot below is of a single device in a DMZ (with everything else shutdown) transferring a 1GB file via SMB from a VM in the management Vlan: -
The traffic path is simply from the VM target -> pfsense -> VM recipient. All VM's are on the same host and use the same aggregated LACP connection. In future, i could separate the VM's into an isolated portgroup so that they do not go over the physical network but this is trivial for the matter at the moment.
Lows of <1MBps and maxing out at best 4MBps. Very unexpected behaviour.
Thanks
pops -
Hmm, I think I'd grab a pcap of that and see what's happening. I'd expect a bunch of retransmits. Could reveal an MTU issue.
-
@stephenw10 Thanks and yes that did show retransmissions but it turned out the solution was to disable hardware large receive and checksum offloads. Not something i disabled before for VM's but occasionally the fix. Clearly something about the hardware i need to investigate.
Thanks again for your efforts.
Pops -
Ah, nice catch!
