[Solved] Different throughput on the same interface
-
Hey Guys,
we have a pfSense (2.3.2…) on a hp ProLiant DL360 Gen9 server with the following networkcards.
-
HP Ethernet 1Gb 4-port 331i
-
HP Ethernet 10Gb 2-port 530T
-
HP FlexFabric 10Gb 2-port 533FLR-T
-
HP Store Fabric 8Gb Single Port PCI-e FC HBA - FC
-
HP Store Fabric 8Gb Single Port PCI-e FC HBA - FC
We only use the 10G interfaces, because we need that bandwidth.
But, we have a strange behavior.We use iperf to test the throughput between the firewall and a virtual machine.
If the firewall is the "Server" and the virtual machine is the "client" we only get a throuput about 3GBit/s. If we send from the firewall to the virtual machine we reach a throughput about 8/9 GBit/s.
It does not matter if it is a virtual or a hardware pfSense.The pfSense and the virtual machine are in a new environment with no traffic or rules.
About the pfSense
Version 2.3.2-RELEASE (amd64) built on Tue Jul 19 12:44:43 CDT 2016 FreeBSD 10.3-RELEASE-p5 CPU Type Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz 32 CPUs: 2 package(s) x 8 core(s) x 2 SMT threads -
-
the only relevant tests would need measurements through the firewall. pfsense is not meant to be an endpoint.
note: its very unlikely to get 10Gbe wire speeds
-
Thank you for that suggestion, i appreciate that.
The problem is, that the firewall is not able to communicate with the same speed bidirectional.
Why should it work when we communicate in a other network?We are not going over the internet. It is a communication in the local lan.
We have already checked all limiters, the kernel parameter and nic paramter.
The tcpdump is also fine.
MTU or tcpWindowSize would have an affact on both directions in communication, i think?The only thing i can imagine, is that in case of sending packets to the firewall, the firewall checks the packages in the fw-ruleset… could that be? Is that the bottleneck?
-
We just activate TSO and LRO on the pfsense.
Now we reach the 8GBit/s on both sides…Anyone an idea or some experience with that features on a pfsense?
-
We use iperf to test the throughput between the firewall and a virtual machine.
Are they both in a VM? I mean pfSense and the virtual server?
If the firewall is the "Server" and the virtual machine is the "client" we only get a throuput about 3GBit/s.
In normal you will be getting something between 2 GBit/s and 3 GBit/s as throughput in real life, from a 10 GBit/s link.
If we send from the firewall to the virtual machine we reach a throughput about 8/9 GBit/s.
Perhaps the virtual machine is able to write the data faster then the pfSense, because there are a RAID in or more
RAM that is acting as buffer for the packets, might this be?It does not matter if it is a virtual or a hardware pfSense.
It does for sure! How many cpu cores are given to the pfSense machine?
We just activate TSO and LRO on the pfsense.
Tunings can be often helping much more then we all would expect from!
high up the mbuf size
shorten down the NIC queues to 4 till 6
and other options or tunings might be helping also, please give them a try out, single or together!Anyone an idea or some experience with that features on a pfsense?
-
Hey BlueKobold,
thank you for your suggestions. We also just recieved an answer from the pfSense-Support.
But i will answer your Questions as good i can :)
@BlueKobold:We use iperf to test the throughput between the firewall and a virtual machine.
Are they both in a VM? I mean pfSense and the virtual server?
We tried both of them. The virtual firewalls most limited by there amount of cpus and often by the featuresets.
After activating TSO and LRO we also reach 5GBit/s with the virtual pfsense.@BlueKobold:
If the firewall is the "Server" and the virtual machine is the "client" we only get a throuput about 3GBit/s.
In normal you will be getting something between 2 GBit/s and 3 GBit/s as throughput in real life, from a 10 GBit/s link.
Yes, of course we are talking about a theoretical throughput, but i would expect a similar throughput in both sides of communication, right?
@BlueKobold:
If we send from the firewall to the virtual machine we reach a throughput about 8/9 GBit/s.
Perhaps the virtual machine is able to write the data faster then the pfSense, because there are a RAID in or more
RAM that is acting as buffer for the packets, might this be?We never send a real amount of data over the cable :) with iperf you send an amount of packets with embedded timestamps and sequence numbers. With this content iperf calculates his statistics.
@BlueKobold:
It does not matter if it is a virtual or a hardware pfSense.
It does for sure! How many cpu cores are given to the pfSense machine?
See my answer above. Of course it matters, because of the amount of cpu - i had to be more specific i think ;)
I mean, it does not matter with the strange behavior of different throughput. But as i said before, when the firewall sends his packets, it expect an ACK after everyone, the vm does not.
So we activate TSO and now the firewall dont expect that anymore - just TSO@BlueKobold:
We just activate TSO and LRO on the pfsense.
Tunings can be often helping much more then we all would expect from!
high up the mbuf size
shorten down the NIC queues to 4 till 6
and other options or tunings might be helping also, please give them a try out, single or together!Anyone an idea or some experience with that features on a pfsense?
I checked that article, everything was okay. Tuning the machine is the first i thought about. Troubleshooting the second ;)
BlueKobold, thank you very much for your help.