Virtualized ESXI PFSense can't pass ~1gbit with iPerf3
-
Hi!
So, I have a CARP setup, with two virtualized PFSense (ESXI 7.0).
I tried to run iPerf3 server on my main Pfsense VM (also on my secondary) and as a client I run iperf3 on a ubuntu server machine that lives in the same subnet as my DMZ network.
I can't get over these results:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 179 MBytes 1.50 Gbits/sec 7 567 KBytes
[ 5] 1.00-2.00 sec 185 MBytes 1.55 Gbits/sec 6 563 KBytes
[ 5] 2.00-3.00 sec 160 MBytes 1.34 Gbits/sec 0 749 KBytes
[ 5] 3.00-4.00 sec 175 MBytes 1.47 Gbits/sec 16 732 KBytes
[ 5] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec 10 716 KBytes
[ 5] 5.00-6.00 sec 180 MBytes 1.51 Gbits/sec 1 714 KBytes
[ 5] 6.00-7.00 sec 192 MBytes 1.62 Gbits/sec 10 707 KBytes
[ 5] 7.00-8.00 sec 176 MBytes 1.48 Gbits/sec 10 691 KBytes
[ 5] 8.00-9.00 sec 179 MBytes 1.50 Gbits/sec 11 683 KBytes
[ 5] 9.00-10.00 sec 176 MBytes 1.48 Gbits/sec 14 673 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.74 GBytes 1.50 Gbits/sec 85 sender
[ 5] 0.00-10.00 sec 1.74 GBytes 1.49 Gbits/sec receiverI tried increasing the number o concurrent calls with same results.
I get over 8gbit using iPerf3 between my other VMs.
I use VMXNet3 for all my machines and I have HW Offloading disabled on my PF VM.
It has 8GB of RAM and 4vCPUs
Any hints?
Thank you! -
@fmroeira86 said in Virtualized ESXI PFSense can't pass ~1gbit with iPerf3:
Any hints?
never ever run iperf on pfsense itself.
measure through - not to/from -
@heper Why?
-
@heper said in Virtualized ESXI PFSense can't pass ~1gbit with iPerf3:
measure through - not to/from
Because this is the way.
-
"just because" is not a very illuminating answer.
Can you please clarify why I can't test it "terminating" on pfsense?
Thank you!
-
@fmroeira86 I can't, only I have read that it is like that. But how often do you copy stuff onto your firewall? Just test between your hosts and see yourself...
-
@fmroeira86 said in Virtualized ESXI PFSense can't pass ~1gbit with iPerf3:
@heper Why?
https://docs.netgate.com/pfsense/en/latest/packages/iperf.html?highlight=iperf#usage
iperf running on pfSense software is NOT a suitable way of testing firewall throughput, as there is a significant difference between performance of traffic initiated or terminated on the firewall and traffic traversing the firewall. There are many suitable uses for iperf running on pfSense software, but testing the throughput capabilities of the firewall is not one of them. -
Thank you for your clarification!
-
@bob-dig Thank you!
-
Well, I just did a test as your described with two machines on different subnets, routed by PFSense and I could get more than this:
[SUM] 0.00-10.00 sec 1.46 GBytes 1.25 Gbits/sec 8097 sender
[SUM] 0.00-10.00 sec 1.45 GBytes 1.24 Gbits/sec receiver -
Check the per-core loading shown in
top -HaSPwhen testing throughput. -
@stephenw10 said in Virtualized ESXI PFSense can't pass ~1gbit with iPerf3:
top -HaSP
Ok.
After some reading I added these entries to loader.conf.local
hw.pci.honor_msi_blacklist="0"
dev.vmx.0.iflib.override_ntxds="0,4096"
dev.vmx.0.iflib.override_nrxds="0,2048,0"
dev.vmx.1.iflib.override_ntxds="0,4096"
dev.vmx.1.iflib.override_nrxds="0,2048,0"
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"I also added:
net.isr.dispatch=deferred
and I got:
[SUM] 0.00-10.00 sec 4.65 GBytes 4.00 Gbits/sec 2111 sender
[SUM] 0.00-10.01 sec 4.62 GBytes 3.96 Gbits/sec receiverIt's improving but still far from 10gbit. I would be satisfied with 8Gbits/sec...
During my testing this is the output from top -HaSP command :
-
Update:
New results with:
Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive OffloadingENABLED (meaning= unchecked)
And also removed
net.isr.dispatch=deferred
[SUM] 0.00-60.00 sec 56.8 GBytes 8.14 Gbits/sec 12804 sender
[SUM] 0.00-60.00 sec 56.8 GBytes 8.13 Gbits/sec receiver -
That's probably as good as you're going to get.
You might check he number of queues each vmx NIC is using. It should show in the boot log.
-
In the meantime I just tried to do iperf3 between two servers (with pfsense in the middle) and I only got:
[SUM] 0.00-20.00 sec 10.1 GBytes 4.33 Gbits/sec 12919 sender
[SUM] 0.00-20.01 sec 10.0 GBytes 4.31 Gbits/sec receiverIf I set the pfsense box as a iperf3 server I get the results I told before:
[SUM] 0.00-60.00 sec 56.8 GBytes 8.14 Gbits/sec 12804 sender
[SUM] 0.00-60.00 sec 56.8 GBytes 8.13 Gbits/sec receiver
