VLAN Network performance Green to Orange on ESXi setup

AnM

I have some strange issue with my PFSense setup, any hints welcome !

I run my PFSense 2.4.2 in an ESXi environment.

I use VLAN for all networks (Blue, Red, Green and Orange (DMZ))
Green network has Squid transparent proxy setup

Lets assume I have an ESXI setup with 5 VMs

PFSense, O1, O2, G1 and G2
PFSense use VMX3 network adapter type
Other VMs use e1000
All VMs connected to the same vSwitch (Red is on own vSwitch and own network card (identical))

O1 to O2 I get 5 Gbits / s
G1 to G2 I get 5 Gbits / s

G1 to Internet I get 950 Mbits / s (really good ! Have 1Gbps connection)
O1 to internet I get 600 Mbits / s download, and 5 Mbits / s upload (so really bad upload)

G1 to O1 I get about 3 MBits / s (iperf, really bad)
O1 to G1 (with hole in firewall added) I get about 3 MBits / s (iperf, really bad)

I do see a somewhat high %VMWAIT on esxtop (spike 22%, but not continous)
The CPU load in PFSense seems quite normal (low)
I seen no issues with e.g. %wa

I got no clue how to debug further

I use the following settings for my igb network card
(based on info from the forum)

kern.ipc.nmbclusters="1000000"
hw.igb.num_queues=0
kern.cam.boot_delay=10000
hw.igb.rxd=2048
hw.igb.txd=2048
net.pf.states_hashsize=524288
net.pf.source_nodes_hashsize=524288
hw.igb.fc_setting=0
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
net.inet.tcp.syncache.hashsize="2048"
net.inet.tcp.syncache.bucketlimit="16"
net.inet.tcp.syncache.cachelimit="32768"

You disabled offload in the Advanced Tab as suggested?

AnM

No, I did not - because I misunderstood the instructions

Works now, huge thanks !