Gigabit Throughput
-
@marvosa said in Gigabit Throughput:
1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no.
Unfortunately I don't have a Gigabit WAN to test with (I can only dream
) but in a local test with iperf3 client LAN side and iperf3 server WAN side, firewalling and NATing:[2.4.4-RELEASE][root@7100.stevew.lan]/root: iperf3 -c 172.21.16.76 Connecting to host 172.21.16.76, port 5201 [ 5] local 192.168.112.10 port 47156 connected to 172.21.16.76 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 940 Mbits/sec 0 160 KBytes [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 3.00-4.00 sec 112 MBytes 936 Mbits/sec 0 160 KBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 6.00-7.00 sec 111 MBytes 928 Mbits/sec 0 160 KBytes [ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 160 KBytes [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 0 sender [ 5] 0.00-10.07 sec 1.09 GBytes 932 Mbits/sec receiver iperf Done.And during that the E8400 box shows:
last pid: 73219; load averages: 1.23, 0.80, 0.56 up 0+00:31:22 17:23:52 233 processes: 6 running, 184 sleeping, 43 waiting CPU: 8.0% user, 0.2% nice, 37.3% system, 37.1% interrupt, 17.5% idle Mem: 348M Active, 243M Inact, 438M Wired, 196M Buf, 888M Free Swap: 1894M Total, 1894M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 351 root 52 0 224M 191M CPU0 0 0:04 48.39% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 56 root -8 - 0K 16K RUN 0 1:38 37.20% [md1] 12 root -92 - 0K 704K CPU0 0 1:53 37.20% [intr{irq261: em1:rx0}] 12 root -92 - 0K 704K WAIT 0 2:02 36.76% [intr{irq257: em0:rx0}] 11 root 155 ki31 0K 32K RUN 1 26:12 18.42% [idle{idle: cpu1}] 11 root 155 ki31 0K 32K RUN 0 25:13 12.20% [idle{idle: cpu0}] 12 root -92 - 0K 704K WAIT 1 0:05 1.52% [intr{irq258: em0:tx0}] 12 root -92 - 0K 704K WAIT 1 0:04 1.46% [intr{irq262: em1:tx0}] 351 root 20 0 224M 191M uwait 1 0:01 1.00% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 351 root 20 0 224M 191M uwait 0 0:00 0.61% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 73735 unbound 20 0 31992K 16828K kqread 0 0:00 0.52% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbo 66520 root 37 0 97732K 37496K accept 1 0:01 0.24% php-fpm: pool nginx (php-fpm){php-fpm} 12 root -60 - 0K 704K WAIT 1 0:04 0.20% [intr{swi4: clock (0)}] 50995 root 20 0 50952K 39972K nanslp 1 0:00 0.13% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfbl 54344 nobody -22 r30 6500K 2608K nanslp 1 0:00 0.11% /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobod 12 root -72 - 0K 704K WAIT 0 0:00 0.08% [intr{swi1: netisr 1}] 11335 root 20 0 9860K 4360K CPU1 1 0:00 0.08% top -aSHSo I'm going to vote yes. And yes even more if I wasn't running Suricata on that box!
Steve
-
As expected an old core2duo can route and nat 1 gigabit of traffic.
But the wan results above are horrible.
Something is completely flawed. Could it be mtu? the wan card?
i'd love to see some iperf3 tests cross box with local wan first. -
@netblues said in Gigabit Throughput:
Could it be mtu?
Normally, you get the best results with the largest MTU. That's usually 1500, but would be 1492 on ADSL.
-
@jknott That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work. I doubt its an mtu issue, however the speed differences via pfsense is huge, and is difficult to believe its the network card unless it is faulty.
A tp link ethernet might not be an intel igb, but still. -
@netblues said in Gigabit Throughput:
That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work.
If that's the case, you should see ICMP "too big" messages. Fragmentation doesn't happen as often as it used to, as it's not allowed on IPv6 and the do not fragment flag is often used on IPv4. With Linux, that flag is set on everything, but only TCP on Windows.
-
Howdy all! Sorry I'm a bit behind. The WAN card was only PCI so I swapped it out with a new PCIx one. So here's what I have now:
em0 00:0f:fe:c7:5e:bc (up) Intel(R) PRO/1000 Network Connection 7.6.1-k
re0 18:a6:f7:01:1e:d0 (up) RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe GigabHere is iperf3 from the pfSense box cmd line with the new PCIx card:
root: iperf3 -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 5] local 68.102.220.240 port 7956 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 5.23 MBytes 43.6 Mbits/sec 1 1.41 KBytes
[ 5] 1.01-2.00 sec 2.02 MBytes 17.0 Mbits/sec 1397 546 KBytes
[ 5] 2.00-3.00 sec 324 KBytes 2.65 Mbits/sec 45 2.85 KBytes
[ 5] 3.00-4.00 sec 912 KBytes 7.49 Mbits/sec 19 272 KBytes
[ 5] 4.00-5.00 sec 3.93 MBytes 33.0 Mbits/sec 0 288 KBytes
[ 5] 5.00-6.00 sec 4.16 MBytes 34.9 Mbits/sec 0 304 KBytes
[ 5] 6.00-7.00 sec 3.04 MBytes 25.5 Mbits/sec 14 157 KBytes
[ 5] 7.00-8.00 sec 2.35 MBytes 19.7 Mbits/sec 0 174 KBytes
[ 5] 8.00-9.00 sec 2.65 MBytes 22.2 Mbits/sec 0 190 KBytes
[ 5] 9.00-10.00 sec 2.82 MBytes 23.6 Mbits/sec 0 206 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 27.4 MBytes 23.0 Mbits/sec 1476 sender
[ 5] 0.00-10.24 sec 25.5 MBytes 20.9 Mbits/sec receiveriperf Done.
I've not tried flipping the interfaces and trying with the Intel card yet.
Here is iperf3 from my desktop:
iperf-3.1.3-win64>iperf3.exe -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 4] local 192.168.5.121 port 63682 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 2.12 MBytes 17.8 Mbits/sec
[ 4] 1.00-2.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 2.00-3.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 3.00-4.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 4.00-5.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 5.00-6.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 6.00-7.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 7.00-8.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 4] 8.00-9.00 sec 3.12 MBytes 26.2 Mbits/sec
[ 4] 9.00-10.00 sec 3.00 MBytes 25.2 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 28.8 MBytes 24.1 Mbits/sec sender
[ 4] 0.00-10.00 sec 28.6 MBytes 24.0 Mbits/sec receiveriperf Done.
Here is my connection from my desktop to the pfSense box:
iperf3.exe -c 192.168.5.1 -p 5201
Connecting to host 192.168.5.1, port 5201
[ 4] local 192.168.5.121 port 63707 connected to 192.168.5.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 105 MBytes 882 Mbits/sec
[ 4] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
[ 4] 2.00-3.00 sec 108 MBytes 910 Mbits/sec
[ 4] 3.00-4.00 sec 110 MBytes 924 Mbits/sec
[ 4] 4.00-5.00 sec 111 MBytes 932 Mbits/sec
[ 4] 5.00-6.00 sec 107 MBytes 896 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
[ 4] 7.00-8.00 sec 111 MBytes 932 Mbits/sec
[ 4] 8.00-9.00 sec 111 MBytes 932 Mbits/sec
[ 4] 9.00-10.00 sec 75.5 MBytes 633 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec receiveriperf Done.
Where should I look next?
-
You have a lot of retries on that first test. Try in the reverse direction. You probably don't need to specify the port, 5201 is the default:
iperf3 -c iperf.scottlinux.com -RRun a test to that server from a client connected directly to the modem without pfSense in play. Make sure you can get a reasonable rate to that server over your WAN at all.
Steve
-
testing out to the internet for iperf not really going to be a valid test... Test from your lan to your wan network.. Put a box on your wan and a box on your lan and run iperf between them..
Once you go out to the internet you have to many variables to rule out just something on net is problem.. Rule out your local hardware first. By testing local!!
C:\tools\iperf3.6_64bit>iperf3.exe -c iperf.scottlinux.com -p 5201 Connecting to host iperf.scottlinux.com, port 5201 [ 5] local 2001:470:<snipped>:7157:1522 port 54061 connected to 2600:3c01::f03c:91ff:fed5:ed33 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.50 MBytes 12.6 Mbits/sec [ 5] 1.00-2.00 sec 3.25 MBytes 27.2 Mbits/sec [ 5] 2.00-3.00 sec 2.50 MBytes 21.0 Mbits/sec [ 5] 3.00-4.00 sec 2.25 MBytes 18.9 Mbits/sec [ 5] 4.00-5.00 sec 2.12 MBytes 17.8 Mbits/sec [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 6.00-7.00 sec 640 KBytes 5.24 Mbits/sec [ 5] 7.00-8.00 sec 896 KBytes 7.34 Mbits/sec [ 5] 8.00-9.00 sec 896 KBytes 7.34 Mbits/sec [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 16.5 MBytes 13.8 Mbits/sec sender [ 5] 0.00-10.09 sec 15.7 MBytes 13.0 Mbits/sec receiver iperf Done. C:\tools\iperf3.6_64bit>Just connected to that scott site you used and the speed was utter crap!!! but I know for a fact my hardware and internet connection are fine.. I see my full pipe all the time downloading and uploading.
-
see my edit... Put a box on your wan network of pfsense and do your testing...
As you can see from that edit - that site is not very reliable for what your speed is or should be..
Here just started download of file from one of my servers in the NL... I have a 500mbps connection seeing 53MBps down... My connection is fine - but that scott iperf showing junk..
-
Depend of your location you should try nearest public iperf3 server.
Choose it here: https://iperf.cc
And try to check your bandwidth again. -
Yes, there are lots of public iperf servers you could use. The one you tested against does not appear to be particularly fast, at least not from where you are.
But the best test you can do is to run your own server locally on the WAN side.
Steve
-
My setup uses a nearly identical Core 2 Due E8500 CPU and I can reliably achieve 900Mbps+ throughput. The network cards make a difference. Also your PC should be fast. I have an older system based on Xeon W3550, and it never gets more than 600Mbps from Internet, testing with iperf or doing an ISP speed test.
-
Howdy all! So I did purchase a PCIe Intel NIC, but don't see a significant change. What is the best way to test the WAN port with iperf? Is a windows laptop ok?
-
It's probably fine.
Before you test the firewall throughput put both the iperf3 server and client machines on the same subnet and test between them directly. Make sure you can see gigabit line rate in both directions.
Then move the server machine into the WAN subet and test against it with the client on the LAN.
If you don't see >900Mbps both ways then look for errors on the interfaces. Try running
top -aSHon the firewall during the test to see the cpu core loading. Be sure not to have the dashboard up in a browser as that can use significant CPU cycles depending on what widgets you have loaded.Steve
-
This is my test :
I run pfSense on Cisco UCS C210 M2 with 2x X5650 CPU and BroadCom QLogic dual port 10G NIC... Maximum load I registered was 11%...
I am pretty sure this result is caused by a speed limitations between me and server instead of my pfSense box... After a few day I will have second 10G line from separate ISP and then I can test again... This machine was released in 2010 so almost 9 years old but works pretty well and I am happy with it ;)
