Inter-vlan traffic is rate limited as VM
-
@stephenw10 hi
Correct, the router is connected to the core switch and manages all VLAN traffic. I am not seeing any such restrictions on endpoints that are traversing the WAN; they are operating over the WAN at the link's speed. No traffic shaping configured either.
It appears to be affecting traffic traversing the VLANs and not out through the WAN. Curiously, any inbound VPN traffic to those VLANS also appears to be affected and i can see spikes in the ping replies to the VLAN devices when accessing any running http service.
Pops
-
@Popolou said in Inter-vlan traffic is rate limited as VM:
Curiously, any inbound VPN traffic to those VLANS also appears to be affected
How is that routed? From external clients?
-
@stephenw10 Via an OpenVPN instance configured and routed within pfsense.
-
But I mean it's external OpenVPN clients accessing resources on one of the VLANs?
Do you see the throttling in both directions?
-
@stephenw10 Correct, yes and simple ping responses which should be in the low tens of milliseconds are coming back as several hundred of milliseconds. The behaviour does appear to be in both directions.
It’s got me stumped.
-
You're using VMX NICs in ESXi?
Did you apply the recommended tuning?
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfaces -
@stephenw10, evening. Thanks and yes, set against both vmx0 & vmx1 for the interface carrying the nine Vlans and the other for the WAN.
Pops
-
They were already set or you just set them now? Probably need to reboot to apply if you did.
-
@stephenw10 no, been set as part of the VM transition a week ago.
-
Hmm, do you see the same throttling if you test to or from the firewall directly?
-
@stephenw10 Good question and no, it works normally as expected. There are no traffic issues or any signs of throttling on the management interface or other devices on the same management Vlan. But traversing beyond the L2 domain into another vlan and wham, the problem occurs.
-
Is it a 'hard' limit? If you look at the traffic graphs is it flat or spikes?
It 'feels' like it could be an asymmetric routing issue. If so it would be very spikey.
-
@stephenw10 Hi, very spikey. The snapshot below is of a single device in a DMZ (with everything else shutdown) transferring a 1GB file via SMB from a VM in the management Vlan: -
The traffic path is simply from the VM target -> pfsense -> VM recipient. All VM's are on the same host and use the same aggregated LACP connection. In future, i could separate the VM's into an isolated portgroup so that they do not go over the physical network but this is trivial for the matter at the moment.
Lows of <1MBps and maxing out at best 4MBps. Very unexpected behaviour.
Thanks
pops -
Hmm, I think I'd grab a pcap of that and see what's happening. I'd expect a bunch of retransmits. Could reveal an MTU issue.
-
@stephenw10 Thanks and yes that did show retransmissions but it turned out the solution was to disable hardware large receive and checksum offloads. Not something i disabled before for VM's but occasionally the fix. Clearly something about the hardware i need to investigate.
Thanks again for your efforts.
Pops -
Ah, nice catch!
