Pfsense 2.1 + esxi poor network performance
-
Interesting question and useful numbers which is always good. You seem to have missed out some of the iperf results above. Do you have those numbers?
Steve
-
Effectively, my fault, bad copy/paste.
I need to rebuild the vm …So the result for freebsd8.3
e1000
freebsd2# /usr/local/bin/iperf -c 172.17.8.226
Client connecting to 172.17.8.226, TCP port 5001
TCP window size: 32.5 KByte (default)[ 3] local 172.17.8.227 port 21848 connected with 172.17.8.226 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 2.37 GBytes 2.04 Gbits/secand the result with the vmxnet3 driver:
freebsd2# /usr/local/bin/iperf -c 172.17.8.226
–----------------------------------------------------------
Client connecting to 172.17.8.226, TCP port 5001
TCP window size: 32.5 KByte (default)[ 3] local 172.17.8.227 port 56445 connected with 172.17.8.226 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.79 GBytes 3.26 Gbits/sec -
Ah, well the figure that stands out is FreeBSD 8.3 with vmxnet3. I assume it was far higher than pfSense, >5Gbps?
Steve
-
Yes,
but why performance with pfsense are under freebsd on the same bsd version?
-
Had you enabled "pf" (even with a very simple ruleset) when testing stock FreeBSD 8.3 against pfSense ?
-
Nope,
because i was only testing performance without firewalling. I know i will loose a little with firewalling enbled.
I'm very surprised that network performance under freebsd are lower than under debian.
Maybe the ethernet driver are not so good under freebsd.
But i did not expect so awfull performance with pfsense.It is a know issue?
Regards.
-
How did you disable pf?
Steve
-
There is a check box under the System/Advanced/Firewall-NAT called "Disable all packet filtering."
And when i do a pfctl -d under console, i got pfctl: pf not enabled.
So i think it's disabled. -
Fair enough, that seems disabled.
Still routing? NATing stuff?Steve
-
You have to check even the TSO and LRO iirc those get disabled by default on pfSense.
Also worth checking the difference between sysct net.isr between FreeBSD and pfSense.While iperf gives performance for stream it does not generalize the different workload requirements.
Also you are not testing forwarding performance when running iperf on pfsense itself and you have to consider that.
-
Yes i'm already aware of this.
For now it is just a test for test the virtualization.
I was thinking that i get the same result with freebsd and pfsense. -
You've missed ESXi version and hardware you are using.
Is it standalone server or vSphere cluster, do you have all esx host drivers up to date? -
One things that seems to help your workload is polling.
So enable polling and test with it.Also in pfSense the kern.hz is reduced to 100 when VMware is detected might be worth uping that to same value as FreeBSD.
It used to be problematic at the time though if you run with vmware-tools than probably is worth testing that scenario. -
Not sure if this is related but earlier this week using the daily snapshots my download speed went from 250mbit to 12 when i run through the pfsense box, bypassing the box and hooking directly to cable modem gives 250 again. Upload speed does not seem to be affected (15mbit with or without pfsense) Only started to see this earlier this week. Disabled IPv6 but problem still persists to this day.
-
Okay, couple things.
- Which ESXi -exactly-? Version and build number.
- Are you running the vmxnet2 as Flexible?
- Can you please retest with the 'legacy' interface? Preferably in pcn(4) mode over lnc(4) mode. I'm rusty so I forget how to force that behavior. (Hell, I forget if the PCIID changes were committed.)
- How many em(4) (aka e1000) interfaces are you running during testing? Yes, this matters.
I think part of the problem is that 2.1 pulled in a bad em(4) branch - but I haven't had time to test more in depth.
EDIT: Oh, can you also please check to see if you have "calcru: runtime went backwards" messages?
