LAN interface performance limited to 400Mbps
-
Hi, i recently started playing with iperf3 and i am getting a 400Mbps max throughput (TCP mode).
I am running pfsense on this box:
Specs:
Intel Quad Core Atom E3845, 64 bit, 1.9GHz, 2MB L2 Cache, AES-NI hardware support
4x Intel Gigabit Ethernet NIC portsWhen i run iperf3 between to machines in my Gigabit LAN i get ~950Mbps. When i run it between the pfsense box and the same client machines i am getting 400Mbps.
The 400 seems to rounded to be a coincidence. it is always between 400 and 403Mbps.
Is there some kind of limitation either in pfsense or the HW config that maybe causing this?
I am good with the local traffic as it hits 900+ Mbps but i have a gigabit connection that somehow is wasted.
Ideas, things to try? i dont think it is a HW limitation, pls chime in
-
Looks like a hardware limitation to me.
That processor is very low power.
-
Have a look at CPU power for the Atom E3845
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+E3845+%40+1.91GHz&id=2225Here's a modern i3 processor
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i3-6100+%40+3.70GHz&id=2617Your single thread performance on the cpu benchmark is 414 vs that i3 processor at 2105
-
You understand pfsense is meant to "route" not provide services right.. If you want to test the performance of pfsense interfaces then you should test through pfsense. So put iperf client on 1 network that routes through pfsense, and a iperf server on another network that pfsense then test from client to server.
So take those 2 machines you were testing between on the same lan, and let pfsense route between them - then test with iperf between those 2 machines.
Here is test just did between 2 boxes that are routed across my sg4860..
$ iperf3.exe -c 192.168.3.80 Connecting to host 192.168.3.80, port 5201 [ 5] local 192.168.9.100 port 50365 connected to 192.168.3.80 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 108 MBytes 909 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec [ 5] 2.00-3.00 sec 113 MBytes 949 Mbits/sec [ 5] 3.00-4.00 sec 113 MBytes 949 Mbits/sec [ 5] 4.00-5.00 sec 113 MBytes 949 Mbits/sec [ 5] 5.00-6.00 sec 113 MBytes 949 Mbits/sec [ 5] 6.00-7.00 sec 113 MBytes 950 Mbits/sec [ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 935 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done.
Here I tested from the 9.100 client to pfsense IP on my lan interface
iperf3.exe -c 192.168.9.253 Connecting to host 192.168.9.253, port 5201 [ 5] local 192.168.9.100 port 50401 connected to 192.168.9.253 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 76.6 MBytes 643 Mbits/sec [ 5] 1.00-2.00 sec 78.4 MBytes 657 Mbits/sec [ 5] 2.00-3.00 sec 76.5 MBytes 642 Mbits/sec [ 5] 3.00-4.00 sec 75.9 MBytes 637 Mbits/sec [ 5] 4.00-5.00 sec 77.4 MBytes 649 Mbits/sec [ 5] 5.00-6.00 sec 74.9 MBytes 628 Mbits/sec [ 5] 6.00-7.00 sec 74.6 MBytes 626 Mbits/sec [ 5] 7.00-8.00 sec 75.0 MBytes 630 Mbits/sec [ 5] 8.00-9.00 sec 78.5 MBytes 658 Mbits/sec [ 5] 9.00-10.00 sec 79.0 MBytes 663 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 767 MBytes 643 Mbits/sec sender [ 5] 0.00-10.00 sec 767 MBytes 643 Mbits/sec receiver iperf Done.
So while to pfsense is slower - routing through pfsense is full wire speed.
Only place you could see issues might be if your having pfsense "proxy" stuff, etc.
-
I did two tests and found some interesting data points:
Test 1:
-iperf3 server in a client on the network
-iperf client on pfsense console
-Result: 809mbps-iperf3 server in pfsense
-iperf3 client in a client on the network
-Result: 409mbpsSo double the bandwidth when not routing i guess..
Test 2:
-uninstalled bandwidthd
-same 2 tests as aboveResults: 928Mbps, and 621mbps
So in conclusion bandwidthd was taking a toll on the throughput. I have no other packages installed.
So the other 300mbps are attributted to the CPU not able to keep up routing at that speed?
-
As was already said (and apparently ignored) An iperf client or server running on pfSense consumes CPU cycles. If you really want to test throughput put an iperf server (known to be able to easily saturate a gigabit link) locally outside the WAN interface and an iperf client (known to be able to easily saturate a gigabit link) locally on the lan and test THROUGH pfSense, not to it or from it.