Gigabit Throughput
-
Howdy! I'm on a gigabit cable connection and can't get much more then 230 meg though my pfSense firewall. I've checked all the interfaces are at 1000baseT full duplex with no errors. I'm only running one VPN that isn't used for internet traffic. Hardware is below and doesn't seem to be taxed in anyway:
Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
Current: 1998 MHz, Max: 3000 MHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: No
4 GB of RAMRunning 2.4.4-RELEASE-p1 (amd64)
I see a bunch of advise to disable snort, traffic shaping, and other services, but I'm just running simple NAT. Where should I look next?
-
Mmm, you should be able to pass 1Gbps of firewall+NAT with that in general. But....
How are you testing?
Are you running any packages?
What NICs do you have?
Steve
-
I've got a TP-Link addon card and the on-motherboard one. As for packages;
apcupsd
bandwithd
iperf
nmap
openvpn-client-exportthanks for the help!!
-
How do those NICs appear in Interfaces > Assignments though?
em0 and em1? re0? Some thing else?
Steve
-
Sorry, I should have included that, WAN (TP-Link) is re0 and LAN is em0.
-
Ok, well the re NIC isn't going to be helping there but I'd still expect far more than 280Mbps.
But how exactly are you testing that?
Really I would like to see an iperf3 test with a client on one side of the firewall and server on the other.
Steve
-
There are several areas to look at... cables, switch, packages, limiters, traffic shaping, are you virtualized, is the testing box even able to push/pull gigabit speeds, etc, etc... but TBH... you lost me at Core 2 Duo and a TP-Link NIC... ;)
The Core 2 Duo E8400 is almost an 11-year-old CPU... which means 11-year-old architecture throughout the box. Let's all have an honest moment of reflection and think about what the average residential internet speeds were at the beginning of 2008... maybe 10, 20, 30 Mbit if you were lucky? Now fast forward ~11 years... where 1, 2, and 10 Gbit speeds are available in residential areas ... which is 35-1000 times faster than what was available to most people back then... are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.
While hardware may not be your only issue... I can almost guarantee it's a contributing factor. Save yourself the dozens of hours (if not multiple weeks) banging your head against the wall and upgrade your hardware (NIC's included)... then re-assess.
-
@marvosa said in Gigabit Throughput:
are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.
the system itself will do that just fine, the TP-link nic will most likely be the culprit
-
@heper Instead of scratching your head, use iperf. You will need two pc with gigabit lan
Boot some dvd linux distro
Mint, debian, fedora, for example
Load/ install iperf and do a few tests
Pc to pc with just a cable
You should get 1g
Pc to pf internal int (run iperf on pf) get 1gPc to pf external int again measure 1g
If you see 1g everywhere and both ways then your options are- Establish plain routing via pf and test
- Establish nat via pf and test
- Setup a bridge with pf and test
Do monitor cpu usage on pf whole testing.
At this point you will have clear indications on what might be the issueApart from that, I would suspect congestion on your provider, but we need to rule out everything else.
-
@heper said in Gigabit Throughput:
the system itself will do that just fine, the TP-link nic will most likely be the culprit
Can it do it mathematically in theory on paper? Possibly... but the question is... can 11-year-old desktop architecture reliably route and sustain 1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no. And that isn't necessarily just my opinion... even the vendor recommends "Server-class Hardware" to push anything over 500 Mbit.
Don't get me wrong... I'm not saying the box won't "work"... but it's all a matter of what each person considers acceptable... will his box route traffic, yes... will it be fast, sure (relatively)... but will it reliably sustain 940+ Mbit/sec of firewalled, real-world traffic over the gigabit WAN he's paying for... my vote is doubtful. And to me, that's unacceptable. Barring a misconfiguration that ends up limiting bandwidth by design, anytime I don't see full bandwidth minus some overhead, I typically start assessing hardware and infrastructure.
-
Morning everyone, thanks for the comments! You'll have to forgive my ignorance as I'm a bit new to measuring throughput. I ran iperf between my desktop computer and the pfSense firewall and got in the 700s:
/root: iperf -c 192.168.5.121 -p 5201
Client connecting to 192.168.5.121, TCP port 5201
TCP window size: 64.2 KByte (default)[ 3] local 192.168.5.1 port 56750 connected with 192.168.5.121 port 5201
write failed: Broken pipe
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 9.5 sec 884 MBytes 783 Mbits/secMy desktop is behind three switches and crap wiring.
Next, I connected a laptop directly to the cable modem and got in the 800s on speedtest.net several times.
Then I attempted to use iperf on pfSense to contact several public servers I found and got results like below several times:
root: iperf -c speedtest.serverius.net -P 10
Client connecting to speedtest.serverius.net, TCP port 5001
TCP window size: 64.2 KByte (default)[ 7] local 68.102.217.178 port 53624 connected with 178.21.16.76 port 5001
[ 11] local 68.102.217.178 port 29834 connected with 178.21.16.76 port 5001
[ 5] local 68.102.217.178 port 24828 connected with 178.21.16.76 port 5001
[ 9] local 68.102.217.178 port 53592 connected with 178.21.16.76 port 5001
[ 3] local 68.102.217.178 port 42250 connected with 178.21.16.76 port 5001
[ 10] local 68.102.217.178 port 44675 connected with 178.21.16.76 port 5001
[ 8] local 68.102.217.178 port 52098 connected with 178.21.16.76 port 5001
[ 12] local 68.102.217.178 port 29426 connected with 178.21.16.76 port 5001
[ 6] local 68.102.217.178 port 50963 connected with 178.21.16.76 port 5001
[ 4] local 68.102.217.178 port 41528 connected with 178.21.16.76 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 2.75 MBytes 2.30 Mbits/sec
[ 8] 0.0-10.1 sec 4.50 MBytes 3.74 Mbits/sec
[ 12] 0.0-10.1 sec 4.38 MBytes 3.64 Mbits/sec
[ 11] 0.0-10.1 sec 4.38 MBytes 3.63 Mbits/sec
[ 9] 0.0-10.1 sec 2.75 MBytes 2.28 Mbits/sec
[ 10] 0.0-10.2 sec 4.50 MBytes 3.72 Mbits/sec
[ 3] 0.0-10.2 sec 4.50 MBytes 3.71 Mbits/sec
[ 6] 0.0-10.2 sec 4.38 MBytes 3.60 Mbits/sec
[ 4] 0.0-10.2 sec 4.38 MBytes 3.61 Mbits/sec
[ 7] 0.0-10.2 sec 2.75 MBytes 2.26 Mbits/sec
[SUM] 0.0-10.2 sec 39.2 MBytes 32.2 Mbits/secThis is way lower than I got with speedtest.net from the desktop behind all the switches and crap wiring. so I'm not sure what to think of all this....
I did keep an eye on the CPU of the pfSense box during all the tests, and it didn't seem to climb to much. What should I try next?
-
With such an old system, is the TP-Link card simple PCI or PCIe?
-
Iperf from cable modem.lan to wan eth?
-
Better to use iperf3. You can install that on pfSense to use from the command line with
pkg install iperf3.
You can test reverse using that among other advantages.Testing either from or to the firewall is not a good test as it's optimised for routing not serving but it can still be useful for proving out some things. With iperf3 you can run that test with -R to get the reverse test to prove out the LAN is at least capable of over 280Mbps.
You should try testing from your laptop using iperf against speedtest.serverius.net to prove that is a good server to test against. It may be limited to that speed anyway. It would be better to test against a local iperf3 server but otherwise find a public server that can do your WAN line rate when connected directly to the modem.
Steve
-
@marvosa said in Gigabit Throughput:
1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no.
Unfortunately I don't have a Gigabit WAN to test with (I can only dream
) but in a local test with iperf3 client LAN side and iperf3 server WAN side, firewalling and NATing:[2.4.4-RELEASE][root@7100.stevew.lan]/root: iperf3 -c 172.21.16.76 Connecting to host 172.21.16.76, port 5201 [ 5] local 192.168.112.10 port 47156 connected to 172.21.16.76 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 940 Mbits/sec 0 160 KBytes [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 3.00-4.00 sec 112 MBytes 936 Mbits/sec 0 160 KBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 6.00-7.00 sec 111 MBytes 928 Mbits/sec 0 160 KBytes [ 5] 7.00-8.00 sec 112 MBytes 939 Mbits/sec 0 160 KBytes [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 160 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 0 sender [ 5] 0.00-10.07 sec 1.09 GBytes 932 Mbits/sec receiver iperf Done.And during that the E8400 box shows:
last pid: 73219; load averages: 1.23, 0.80, 0.56 up 0+00:31:22 17:23:52 233 processes: 6 running, 184 sleeping, 43 waiting CPU: 8.0% user, 0.2% nice, 37.3% system, 37.1% interrupt, 17.5% idle Mem: 348M Active, 243M Inact, 438M Wired, 196M Buf, 888M Free Swap: 1894M Total, 1894M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 351 root 52 0 224M 191M CPU0 0 0:04 48.39% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 56 root -8 - 0K 16K RUN 0 1:38 37.20% [md1] 12 root -92 - 0K 704K CPU0 0 1:53 37.20% [intr{irq261: em1:rx0}] 12 root -92 - 0K 704K WAIT 0 2:02 36.76% [intr{irq257: em0:rx0}] 11 root 155 ki31 0K 32K RUN 1 26:12 18.42% [idle{idle: cpu1}] 11 root 155 ki31 0K 32K RUN 0 25:13 12.20% [idle{idle: cpu0}] 12 root -92 - 0K 704K WAIT 1 0:05 1.52% [intr{irq258: em0:tx0}] 12 root -92 - 0K 704K WAIT 1 0:04 1.46% [intr{irq262: em1:tx0}] 351 root 20 0 224M 191M uwait 1 0:01 1.00% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 351 root 20 0 224M 191M uwait 0 0:00 0.61% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric 73735 unbound 20 0 31992K 16828K kqread 0 0:00 0.52% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbo 66520 root 37 0 97732K 37496K accept 1 0:01 0.24% php-fpm: pool nginx (php-fpm){php-fpm} 12 root -60 - 0K 704K WAIT 1 0:04 0.20% [intr{swi4: clock (0)}] 50995 root 20 0 50952K 39972K nanslp 1 0:00 0.13% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfbl 54344 nobody -22 r30 6500K 2608K nanslp 1 0:00 0.11% /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobod 12 root -72 - 0K 704K WAIT 0 0:00 0.08% [intr{swi1: netisr 1}] 11335 root 20 0 9860K 4360K CPU1 1 0:00 0.08% top -aSHSo I'm going to vote yes. And yes even more if I wasn't running Suricata on that box!
Steve
-
As expected an old core2duo can route and nat 1 gigabit of traffic.
But the wan results above are horrible.
Something is completely flawed. Could it be mtu? the wan card?
i'd love to see some iperf3 tests cross box with local wan first. -
@netblues said in Gigabit Throughput:
Could it be mtu?
Normally, you get the best results with the largest MTU. That's usually 1500, but would be 1492 on ADSL.
-
@jknott That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work. I doubt its an mtu issue, however the speed differences via pfsense is huge, and is difficult to believe its the network card unless it is faulty.
A tp link ethernet might not be an intel igb, but still. -
@netblues said in Gigabit Throughput:
That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work.
If that's the case, you should see ICMP "too big" messages. Fragmentation doesn't happen as often as it used to, as it's not allowed on IPv6 and the do not fragment flag is often used on IPv4. With Linux, that flag is set on everything, but only TCP on Windows.
-
Howdy all! Sorry I'm a bit behind. The WAN card was only PCI so I swapped it out with a new PCIx one. So here's what I have now:
em0 00:0f:fe:c7:5e:bc (up) Intel(R) PRO/1000 Network Connection 7.6.1-k
re0 18:a6:f7:01:1e:d0 (up) RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe GigabHere is iperf3 from the pfSense box cmd line with the new PCIx card:
root: iperf3 -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 5] local 68.102.220.240 port 7956 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 5.23 MBytes 43.6 Mbits/sec 1 1.41 KBytes
[ 5] 1.01-2.00 sec 2.02 MBytes 17.0 Mbits/sec 1397 546 KBytes
[ 5] 2.00-3.00 sec 324 KBytes 2.65 Mbits/sec 45 2.85 KBytes
[ 5] 3.00-4.00 sec 912 KBytes 7.49 Mbits/sec 19 272 KBytes
[ 5] 4.00-5.00 sec 3.93 MBytes 33.0 Mbits/sec 0 288 KBytes
[ 5] 5.00-6.00 sec 4.16 MBytes 34.9 Mbits/sec 0 304 KBytes
[ 5] 6.00-7.00 sec 3.04 MBytes 25.5 Mbits/sec 14 157 KBytes
[ 5] 7.00-8.00 sec 2.35 MBytes 19.7 Mbits/sec 0 174 KBytes
[ 5] 8.00-9.00 sec 2.65 MBytes 22.2 Mbits/sec 0 190 KBytes
[ 5] 9.00-10.00 sec 2.82 MBytes 23.6 Mbits/sec 0 206 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 27.4 MBytes 23.0 Mbits/sec 1476 sender
[ 5] 0.00-10.24 sec 25.5 MBytes 20.9 Mbits/sec receiveriperf Done.
I've not tried flipping the interfaces and trying with the Intel card yet.
Here is iperf3 from my desktop:
iperf-3.1.3-win64>iperf3.exe -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 4] local 192.168.5.121 port 63682 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 2.12 MBytes 17.8 Mbits/sec
[ 4] 1.00-2.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 2.00-3.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 3.00-4.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 4.00-5.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 5.00-6.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 6.00-7.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 7.00-8.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 4] 8.00-9.00 sec 3.12 MBytes 26.2 Mbits/sec
[ 4] 9.00-10.00 sec 3.00 MBytes 25.2 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 28.8 MBytes 24.1 Mbits/sec sender
[ 4] 0.00-10.00 sec 28.6 MBytes 24.0 Mbits/sec receiveriperf Done.
Here is my connection from my desktop to the pfSense box:
iperf3.exe -c 192.168.5.1 -p 5201
Connecting to host 192.168.5.1, port 5201
[ 4] local 192.168.5.121 port 63707 connected to 192.168.5.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 105 MBytes 882 Mbits/sec
[ 4] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
[ 4] 2.00-3.00 sec 108 MBytes 910 Mbits/sec
[ 4] 3.00-4.00 sec 110 MBytes 924 Mbits/sec
[ 4] 4.00-5.00 sec 111 MBytes 932 Mbits/sec
[ 4] 5.00-6.00 sec 107 MBytes 896 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
[ 4] 7.00-8.00 sec 111 MBytes 932 Mbits/sec
[ 4] 8.00-9.00 sec 111 MBytes 932 Mbits/sec
[ 4] 9.00-10.00 sec 75.5 MBytes 633 Mbits/sec
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec receiveriperf Done.
Where should I look next?
-
You have a lot of retries on that first test. Try in the reverse direction. You probably don't need to specify the port, 5201 is the default:
iperf3 -c iperf.scottlinux.com -RRun a test to that server from a client connected directly to the modem without pfSense in play. Make sure you can get a reasonable rate to that server over your WAN at all.
Steve
-
testing out to the internet for iperf not really going to be a valid test... Test from your lan to your wan network.. Put a box on your wan and a box on your lan and run iperf between them..
Once you go out to the internet you have to many variables to rule out just something on net is problem.. Rule out your local hardware first. By testing local!!
C:\tools\iperf3.6_64bit>iperf3.exe -c iperf.scottlinux.com -p 5201 Connecting to host iperf.scottlinux.com, port 5201 [ 5] local 2001:470:<snipped>:7157:1522 port 54061 connected to 2600:3c01::f03c:91ff:fed5:ed33 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.50 MBytes 12.6 Mbits/sec [ 5] 1.00-2.00 sec 3.25 MBytes 27.2 Mbits/sec [ 5] 2.00-3.00 sec 2.50 MBytes 21.0 Mbits/sec [ 5] 3.00-4.00 sec 2.25 MBytes 18.9 Mbits/sec [ 5] 4.00-5.00 sec 2.12 MBytes 17.8 Mbits/sec [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 6.00-7.00 sec 640 KBytes 5.24 Mbits/sec [ 5] 7.00-8.00 sec 896 KBytes 7.34 Mbits/sec [ 5] 8.00-9.00 sec 896 KBytes 7.34 Mbits/sec [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 16.5 MBytes 13.8 Mbits/sec sender [ 5] 0.00-10.09 sec 15.7 MBytes 13.0 Mbits/sec receiver iperf Done. C:\tools\iperf3.6_64bit>Just connected to that scott site you used and the speed was utter crap!!! but I know for a fact my hardware and internet connection are fine.. I see my full pipe all the time downloading and uploading.
-
see my edit... Put a box on your wan network of pfsense and do your testing...
As you can see from that edit - that site is not very reliable for what your speed is or should be..
Here just started download of file from one of my servers in the NL... I have a 500mbps connection seeing 53MBps down... My connection is fine - but that scott iperf showing junk..
-
Depend of your location you should try nearest public iperf3 server.
Choose it here: https://iperf.cc
And try to check your bandwidth again. -
Yes, there are lots of public iperf servers you could use. The one you tested against does not appear to be particularly fast, at least not from where you are.
But the best test you can do is to run your own server locally on the WAN side.
Steve
-
My setup uses a nearly identical Core 2 Due E8500 CPU and I can reliably achieve 900Mbps+ throughput. The network cards make a difference. Also your PC should be fast. I have an older system based on Xeon W3550, and it never gets more than 600Mbps from Internet, testing with iperf or doing an ISP speed test.
-
Howdy all! So I did purchase a PCIe Intel NIC, but don't see a significant change. What is the best way to test the WAN port with iperf? Is a windows laptop ok?
-
It's probably fine.
Before you test the firewall throughput put both the iperf3 server and client machines on the same subnet and test between them directly. Make sure you can see gigabit line rate in both directions.
Then move the server machine into the WAN subet and test against it with the client on the LAN.
If you don't see >900Mbps both ways then look for errors on the interfaces. Try running
top -aSHon the firewall during the test to see the cpu core loading. Be sure not to have the dashboard up in a browser as that can use significant CPU cycles depending on what widgets you have loaded.Steve
-
This is my test :
I run pfSense on Cisco UCS C210 M2 with 2x X5650 CPU and BroadCom QLogic dual port 10G NIC... Maximum load I registered was 11%...
I am pretty sure this result is caused by a speed limitations between me and server instead of my pfSense box... After a few day I will have second 10G line from separate ISP and then I can test again... This machine was released in 2010 so almost 9 years old but works pretty well and I am happy with it ;)
