pfSense low throughput from 10GB to 1GB Interfaces
-
Hi,
I have the following HA - Setup:
2x HP Proliant DL360 gen9 with 1 onboard LAN ( HP Ethernet 1GB 331i), 1 PCI-E LAN (HP Ethernet 1GB 331FLR) and 1 PCI-E LAN (HP Ethernet 10Gb 560SFP+).
pfSense Software 2.4.5 p1I have configured different VLANS on these interfaces. The routing, CARP, etc. works so far. Then I wanted to test the speed of each interface. The iPerf3 test gave the following results:
From 1GB interface (HP 331i) to 10GB interface (HP 560):
[ 4] local "IP" port 52811 connected to "IP" port 5201
ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 54.2 MBytes 455 Mbits/sec
[ 4] 1.00-2.00 sec 69.8 MBytes 585 Mbits/sec
[ 4] 2.00-3.00 sec 73.0 MBytes 612 Mbits/sec
[ 4] 3.00-4.00 sec 73.2 MBytes 615 Mbits/sec
4] 4.00-5.00 sec 73.5 MBytes 616 Mbits/sec
[ 4] 5.00-6.00 sec 73.9 MBytes 620 Mbits/sec
[ 4] 6.00-7.00 sec 73.4 MBytes 615 Mbits/sec
[ 4] 7.00-8.00 sec 73.2 MBytes 615 Mbits/sec
[ 4] 8.00-9.00 sec 73.5 MBytes 616 Mbits/sec
[ 4] 9.00-10.00 sec 73.6 MBytes 618 Mbits/sec
ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 711 MBytes 597 Mbits/sec transmitter
4] 0.00-10.00 sec 711 MBytes 597 Mbits/sec receiverFrom 10GB interface (HP 560) to 1GB interface (HP 331i):
[ 4] local "IP" port 59909 connected to "IP" port 5201
ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 51.8 MBytes 428 Mbits/sec
[ 4] 1.01-2.01 sec 39.0 MBytes 328 Mbits/sec
[ 4] 2.01-3.01 sec 51.5 MBytes 433 Mbits/sec
[ 4] 3.01-4.01 sec 40.2 MBytes 338 Mbits/sec
4] 4.01-5.01 sec 38.5 MBytes 323 Mbits/sec
[ 4] 5.01-6.01 sec 43.5 MBytes 365 Mbits/sec
[ 4] 6.01-7.00 sec 48.2 MBytes 405 Mbits/sec
[ 4] 7.00-8.00 sec 41.2 MBytes 347 Mbits/sec
[ 4] 8.00-9.00 sec 39.8 MBytes 334 Mbits/sec
[ 4] 9.00-10.02 sec 38.9 MBytes 322 Mbits/sec
ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 433 MBytes 362 Mbits/sec transmitter
4] 0.00-10.02 sec 433 MBytes 362 Mbits/sec receiverAs you can see, we have significantly worse performance from the 10GB to the 1GB interface than vice versa. However, the speed from 1GB to 10GB does not satisfy us either.
I did an iperf3 test directly from the pfSense to the individual targets and found the following values:
From the pfSense (1GB interface) to the first destination:
[ 5] local "IP" port 56729 connected to "IP" port 5201
ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 947 Mbits/sec 1 208 KBytes
[ 5] 1.00-2.00 sec 113 MBytes 945 Mbits/sec 2 208 KBytes
[ 5] 2.00-3.00 sec 113 MBytes 945 Mbits/sec 2 208 KBytes
[ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
[ 5] 4.00-5.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
5] 5.00-6.00 sec 113 MBytes 945 Mbits/sec 2 153 KBytes
[ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 1 208 KBytes
[ 5] 8.00-9.00 sec 113 MBytes 945 Mbits/sec 1 208 KBytes
[ 5] 9.00-10.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 13 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec receiverFrom pfSense (10GB) to second destination:
[ 5] local "IP" port 42314 connected to "IP" port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 32 209 KBytes
[ 5] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 7 120 KBytes
[ 5] 2.00-3.00 sec 110 MBytes 924 Mbits/sec 0 208 KBytes
[ 5] 3.00-4.00 sec 108 MBytes 907 Mbits/sec 1 159 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes
5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 209 KBytes
[ 5] 6.00-7.00 sec 109 MBytes 913 Mbits/sec 3 192 KBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 209 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 925 Mbits/sec 43 sender
[ 5] 0.00-10.00 sec 1.08 GBytes 925 Mbits/sec receiverDoes anyone have any idea how i can fix this? We would be satisfied with a result close to 1GB.
Thanks in advance.
Kind regards,
Kotty -
@kotty
don't test TO or FROM pfsense ... that is pointless.
it's a router. run the tests THROUGH pfsense with a client/server on each end -
Sorry for the misunderstandings. We have tested the throughput with a client/server.
First test Client/Server -> pfSense in (1GB) -> pfSense out (10GB) -> Client/Server
Second test Client/Server -> pfsense in (10GB) -> pfSense out(1GB) -> Client/Server.
third test iperf pfSense out (1GB) -> Client/Server.
fourth test iperf pfSense out(10GB) -> Client/Server
I hope it is better to understand.
Thank you for your help.
-
@kotty try the tuning for your specific cards/drivers
the 331i appears to be some sort of broadcom chipset
the 560 seems to be an intel chipsethttps://docs.netgate.com/pfsense/en/latest/hardware/tune.html#card-specific-issues
-
Thank you.
I have already gone through this post. This has improved the performance. I had problems with packet loss. After I went through the tuning guide, it got much better. But now there are still the performance problems. Could it be a problem with the network cards?
-
@kotty
I don't have similar hardware.... Might be a lot of things.Cabling/switch/driver/....
-
Hi,
I have some news. After replacing the Intel 10GBE network card with an Intel X722-DA2 (ixL Interface), my performance problems are gone. But there is a new problem. When testing the 10GB performance I only get a speed of 2.7Gigabit. Here is an excerpt from iPerf3:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 276 MBytes 2.31 Gbits/sec 0 687 KBytes
[ 5] 1.00-2.00 sec 273 MBytes 2.29 Gbits/sec 0 687 KBytes
[ 5] 2.00-3.00 sec 276 MBytes 2.32 Gbits/sec 0 687 KBytes
[ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec 0 687 KBytes
[ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec 0 687 KBytes
5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 689 KBytes
[ 5] 6.00-7.00 sec 276 MBytes 2.32 Gbits/sec 0 689 KBytes
5] 7.00-8.00 sec 276 MBytes 2.32 Gbits/sec 0 689 KBytes
5] 8.00-9.00 sec 275 MBytes 2.30 Gbits/sec 0 689 KBytes
[ 5] 9.00-10.00 sec 277 MBytes 2.32 Gbits/sec 0 689 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiverWith the command "top -p CC" I see that the test uses only one CPU Core to 100%. However, this only happens when testing the 10GB connection. 1GB tests are distributed over all cores. As it should be. Is there a command or a way to set the pfSense to multicore for 10GBE connections?
Thanks in advance.
Kind regards,
Kotty