LAN interface performance limited to 400Mbps

normaluser99

Hi, i recently started playing with iperf3 and i am getting a 400Mbps max throughput (TCP mode).

I am running pfsense on this box:

Specs:
Intel Quad Core Atom E3845, 64 bit, 1.9GHz, 2MB L2 Cache, AES-NI hardware support
4x Intel Gigabit Ethernet NIC ports

When i run iperf3 between to machines in my Gigabit LAN i get ~950Mbps. When i run it between the pfsense box and the same client machines i am getting 400Mbps.

The 400 seems to rounded to be a coincidence. it is always between 400 and 403Mbps.

Is there some kind of limitation either in pfsense or the HW config that maybe causing this?

I am good with the local traffic as it hits 900+ Mbps but i have a gigabit connection that somehow is wasted.

Ideas, things to try? i dont think it is a HW limitation, pls chime in

nzkiwi68

Looks like a hardware limitation to me.

That processor is very low power.

nzkiwi68

Have a look at CPU power for the Atom E3845
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+E3845+%40+1.91GHz&id=2225

Here's a modern i3 processor
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i3-6100+%40+3.70GHz&id=2617

Your single thread performance on the cpu benchmark is 414 vs that i3 processor at 2105

johnpoz

You understand pfsense is meant to "route" not provide services right.. If you want to test the performance of pfsense interfaces then you should test through pfsense. So put iperf client on 1 network that routes through pfsense, and a iperf server on another network that pfsense then test from client to server.

So take those 2 machines you were testing between on the same lan, and let pfsense route between them - then test with iperf between those 2 machines.

Here is test just did between 2 boxes that are routed across my sg4860..

$ iperf3.exe -c 192.168.3.80                                                        
Connecting to host 192.168.3.80, port 5201                                          
[  5] local 192.168.9.100 port 50365 connected to 192.168.3.80 port 5201            
[ ID] Interval           Transfer     Bitrate                                       
[  5]   0.00-1.00   sec   108 MBytes   909 Mbits/sec                                
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec                                
[  5]   2.00-3.00   sec   113 MBytes   949 Mbits/sec                                
[  5]   3.00-4.00   sec   113 MBytes   949 Mbits/sec                                
[  5]   4.00-5.00   sec   113 MBytes   949 Mbits/sec                                
[  5]   5.00-6.00   sec   113 MBytes   949 Mbits/sec                                
[  5]   6.00-7.00   sec   113 MBytes   950 Mbits/sec                                
[  5]   7.00-8.00   sec   113 MBytes   949 Mbits/sec                                
[  5]   8.00-9.00   sec   112 MBytes   935 Mbits/sec                                
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                                
- - - - - - - - - - - - - - - - - - - - - - - - -                                   
[ ID] Interval           Transfer     Bitrate                                       
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  sender        
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver      
                                                                                    
iperf Done.

Here I tested from the 9.100 client to pfsense IP on my lan interface

 iperf3.exe -c 192.168.9.253
Connecting to host 192.168.9.253, port 5201
[  5] local 192.168.9.100 port 50401 connected to 192.168.9.253 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  76.6 MBytes   643 Mbits/sec
[  5]   1.00-2.00   sec  78.4 MBytes   657 Mbits/sec
[  5]   2.00-3.00   sec  76.5 MBytes   642 Mbits/sec
[  5]   3.00-4.00   sec  75.9 MBytes   637 Mbits/sec
[  5]   4.00-5.00   sec  77.4 MBytes   649 Mbits/sec
[  5]   5.00-6.00   sec  74.9 MBytes   628 Mbits/sec
[  5]   6.00-7.00   sec  74.6 MBytes   626 Mbits/sec
[  5]   7.00-8.00   sec  75.0 MBytes   630 Mbits/sec
[  5]   8.00-9.00   sec  78.5 MBytes   658 Mbits/sec
[  5]   9.00-10.00  sec  79.0 MBytes   663 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   767 MBytes   643 Mbits/sec                  sender
[  5]   0.00-10.00  sec   767 MBytes   643 Mbits/sec                  receiver

iperf Done.

So while to pfsense is slower - routing through pfsense is full wire speed.

Only place you could see issues might be if your having pfsense "proxy" stuff, etc.

normaluser99

I did two tests and found some interesting data points:

Test 1:
-iperf3 server in a client on the network
-iperf client on pfsense console
-Result: 809mbps

-iperf3 server in pfsense
-iperf3 client in a client on the network
-Result: 409mbps

So double the bandwidth when not routing i guess..

Test 2:
-uninstalled bandwidthd
-same 2 tests as above

Results: 928Mbps, and 621mbps

So in conclusion bandwidthd was taking a toll on the throughput. I have no other packages installed.

So the other 300mbps are attributted to the CPU not able to keep up routing at that speed?

Derelict

As was already said (and apparently ignored) An iperf client or server running on pfSense consumes CPU cycles. If you really want to test throughput put an iperf server (known to be able to easily saturate a gigabit link) locally outside the WAN interface and an iperf client (known to be able to easily saturate a gigabit link) locally on the lan and test THROUGH pfSense, not to it or from it.