Why only ~890Mbit/s on 10Gb network? 7100XG

runevn

I have a 7100XG connected to a Ubiquitous EdgeSwitch 16 XG (10Gb switch) via a DAC cable. My desktop computer, also with a 10G network adapter is also connected to the same switch.

pfSense, the switch and my desktop all have "negotiated" 10Gbps connections.

When I run iperf3 (pfsense is the server and my desktop the client) I get the following results:

>iperf3.exe -c 192.168.60.1
Connecting to host 192.168.60.1, port 5201
[  4] local 192.168.60.92 port 62398 connected to 192.168.60.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  90.2 MBytes   757 Mbits/sec
[  4]   1.00-2.00   sec  93.0 MBytes   780 Mbits/sec
[  4]   2.00-3.00   sec  93.1 MBytes   781 Mbits/sec
[  4]   3.00-4.00   sec   101 MBytes   850 Mbits/sec
[  4]   4.00-5.00   sec   101 MBytes   844 Mbits/sec
[  4]   5.00-6.00   sec   106 MBytes   890 Mbits/sec
[  4]   6.00-7.00   sec   101 MBytes   848 Mbits/sec
[  4]   7.00-8.00   sec  94.9 MBytes   795 Mbits/sec
[  4]   8.00-9.00   sec   106 MBytes   890 Mbits/sec
[  4]   9.00-10.00  sec   106 MBytes   894 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   993 MBytes   833 Mbits/sec                  sender
[  4]   0.00-10.00  sec   993 MBytes   833 Mbits/sec                  receiver

I'm not expecting full 10G connection but it seems that something is constraining it at 1Gbps.

Any help is highly appreciated.

Bob.Dig

@runevn add

-P 5

Also you shouldn't iperf to the firewall but between your real hosts.

runevn

@bob-dig Thanks it works. May I ask why not iperf to the firewall? I can see iperf is a feature in pfSense - I'm just want to learn :-)

bmeeks

@runevn said in Why only ~890Mbit/s on 10Gb network? 7100XG:

@bob-dig Thanks it works. May I ask why not iperf to the firewall? I can see iperf is a feature in pfSense - I'm just want to learn :-)

I will chime in with an answer --

Tools such as iperf themselves require CPU resources to operate. When you are wanting to test for maximum performance, you want pfSense using all of the CPU resources to route packets. That is what it is optimized for. Internally, within the kernel, things are optimized for routing packets between network interfaces.

Things are not as optimized for allocating memory, processing display and keyboard I/O, and doing the other things running applications such as iperf require. That means some of the firewall's horsepower is sidetracked running iperf itself and thus is not available to help route packets. That in turn means the speed test result suffers a little. How much it suffers is determined by the CPU you have.

So when you make the firewall an iperf endpoint, you will see a negative impact on your speed test because some of the packet processing power is instead consumed by the running iperf process. But when you run iperf on two hosts outside the firewall, and make sure they are routing traffic "through" the firewall, you get a valid packet handling test (or speed test) of the firewall.

stephenw10

Yup, that^.

Additionally when running FreeBSD as a server you would use various hardware offloading features that are all disabled by default in pfSense to optimise it for routing.

iperf can still be a use test to run on the firewall for proving the status of a single link as long as you realise the limitations of that test. The result will always be lower than routing through it.

Steve

runevn

@bmeeks Thanks a lot for the thorough answer and it make so much sense - thanks.

bmeeks

@runevn said in Why only ~890Mbit/s on 10Gb network? 7100XG:

@bmeeks Thanks a lot for the thorough answer and it make so much sense - thanks.

There are a couple of other things to remember with speed tests using tools such as iperf.

One is to be sure you launch multiple threads generating independent streams or flows when testing raw performance. This will allow the NIC and firewall to take full advantage of a multi-core CPU. If all traffic is running on a single thread, then you are also going to be running on a single CPU core. That will negatively impact measured performance. Make sure you are sending multiple independent flows to let the RSS stuff do its thing.

The other thing to get set properly is the payload size in the test packets. It takes essentially the same amount of CPU effort to process a packet regardless of the payload size. A better and more accurate measure of firewall performance is the PPS (packets per second) rating. That tells you how many packets the firewall can juggle at full tilt. Looking solely at bits per second can be misleading. To see why, think about 64-byte ICMP packets versus 1500-byte full payload packets. That 1500-byte packet has over 23 times more bits in it than the 64-byte packet. So you can run a firewall into the dirt processing 64-byte payload packets and yet only show a relatively puny bits per second result. On the other hand, send it nothing but 1500-byte (or even 9000-byte jumbo frame) packets and you see a big boost in bits per second just due to the larger payload. But both would in reality be processing about the same number of packets per second.

The true test of real world performance is to generate a mixture of packets with both small, medium, and large payloads. This is the kind of data that will flow during normal network operation, and thus you get a more true indicator of real performance that way.

runevn

@stephenw10 thanks for the addon :-)

runevn

@bmeeks - thanks for your furthering. It makes so much sense. Thanks.