Gigabit Throughput



  • Howdy! I'm on a gigabit cable connection and can't get much more then 230 meg though my pfSense firewall. I've checked all the interfaces are at 1000baseT full duplex with no errors. I'm only running one VPN that isn't used for internet traffic. Hardware is below and doesn't seem to be taxed in anyway:

    Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
    Current: 1998 MHz, Max: 3000 MHz
    2 CPUs: 1 package(s) x 2 core(s)
    AES-NI CPU Crypto: No
    4 GB of RAM

    Running 2.4.4-RELEASE-p1 (amd64)

    I see a bunch of advise to disable snort, traffic shaping, and other services, but I'm just running simple NAT. Where should I look next?


  • Netgate Administrator

    Mmm, you should be able to pass 1Gbps of firewall+NAT with that in general. But....

    How are you testing?

    Are you running any packages?

    What NICs do you have?

    Steve



  • I've got a TP-Link addon card and the on-motherboard one. As for packages;

    apcupsd
    bandwithd
    iperf
    nmap
    openvpn-client-export

    thanks for the help!!


  • Netgate Administrator

    How do those NICs appear in Interfaces > Assignments though?

    em0 and em1? re0? Some thing else?

    Steve



  • Sorry, I should have included that, WAN (TP-Link) is re0 and LAN is em0.


  • Netgate Administrator

    Ok, well the re NIC isn't going to be helping there but I'd still expect far more than 280Mbps.

    But how exactly are you testing that?

    Really I would like to see an iperf3 test with a client on one side of the firewall and server on the other.

    Steve



  • There are several areas to look at... cables, switch, packages, limiters, traffic shaping, are you virtualized, is the testing box even able to push/pull gigabit speeds, etc, etc... but TBH... you lost me at Core 2 Duo and a TP-Link NIC... ;)

    The Core 2 Duo E8400 is almost an 11-year-old CPU... which means 11-year-old architecture throughout the box. Let's all have an honest moment of reflection and think about what the average residential internet speeds were at the beginning of 2008... maybe 10, 20, 30 Mbit if you were lucky? Now fast forward ~11 years... where 1, 2, and 10 Gbit speeds are available in residential areas ... which is 35-1000 times faster than what was available to most people back then... are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.

    While hardware may not be your only issue... I can almost guarantee it's a contributing factor. Save yourself the dozens of hours (if not multiple weeks) banging your head against the wall and upgrade your hardware (NIC's included)... then re-assess.



  • @marvosa said in Gigabit Throughput:

    are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.

    the system itself will do that just fine, the TP-link nic will most likely be the culprit



  • @heper Instead of scratching your head, use iperf. You will need two pc with gigabit lan
    Boot some dvd linux distro
    Mint, debian, fedora, for example
    Load/ install iperf and do a few tests
    Pc to pc with just a cable
    You should get 1g
    Pc to pf internal int (run iperf on pf) get 1g

    Pc to pf external int again measure 1g
    If you see 1g everywhere and both ways then your options are

    1. Establish plain routing via pf and test
    2. Establish nat via pf and test
    3. Setup a bridge with pf and test

    Do monitor cpu usage on pf whole testing.
    At this point you will have clear indications on what might be the issue

    Apart from that, I would suspect congestion on your provider, but we need to rule out everything else.



  • @heper said in Gigabit Throughput:

    the system itself will do that just fine, the TP-link nic will most likely be the culprit

    Can it do it mathematically in theory on paper? Possibly... but the question is... can 11-year-old desktop architecture reliably route and sustain 1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no. And that isn't necessarily just my opinion... even the vendor recommends "Server-class Hardware" to push anything over 500 Mbit.

    Don't get me wrong... I'm not saying the box won't "work"... but it's all a matter of what each person considers acceptable... will his box route traffic, yes... will it be fast, sure (relatively)... but will it reliably sustain 940+ Mbit/sec of firewalled, real-world traffic over the gigabit WAN he's paying for... my vote is doubtful. And to me, that's unacceptable. Barring a misconfiguration that ends up limiting bandwidth by design, anytime I don't see full bandwidth minus some overhead, I typically start assessing hardware and infrastructure.



  • Morning everyone, thanks for the comments! You'll have to forgive my ignorance as I'm a bit new to measuring throughput. I ran iperf between my desktop computer and the pfSense firewall and got in the 700s:

    /root: iperf -c 192.168.5.121 -p 5201

    Client connecting to 192.168.5.121, TCP port 5201
    TCP window size: 64.2 KByte (default)

    [ 3] local 192.168.5.1 port 56750 connected with 192.168.5.121 port 5201

    write failed: Broken pipe
    [ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 9.5 sec 884 MBytes 783 Mbits/sec

    My desktop is behind three switches and crap wiring.

    Next, I connected a laptop directly to the cable modem and got in the 800s on speedtest.net several times.

    Then I attempted to use iperf on pfSense to contact several public servers I found and got results like below several times:

    root: iperf -c speedtest.serverius.net -P 10

    Client connecting to speedtest.serverius.net, TCP port 5001
    TCP window size: 64.2 KByte (default)

    [ 7] local 68.102.217.178 port 53624 connected with 178.21.16.76 port 5001
    [ 11] local 68.102.217.178 port 29834 connected with 178.21.16.76 port 5001
    [ 5] local 68.102.217.178 port 24828 connected with 178.21.16.76 port 5001
    [ 9] local 68.102.217.178 port 53592 connected with 178.21.16.76 port 5001
    [ 3] local 68.102.217.178 port 42250 connected with 178.21.16.76 port 5001
    [ 10] local 68.102.217.178 port 44675 connected with 178.21.16.76 port 5001
    [ 8] local 68.102.217.178 port 52098 connected with 178.21.16.76 port 5001
    [ 12] local 68.102.217.178 port 29426 connected with 178.21.16.76 port 5001
    [ 6] local 68.102.217.178 port 50963 connected with 178.21.16.76 port 5001
    [ 4] local 68.102.217.178 port 41528 connected with 178.21.16.76 port 5001
    [ ID] Interval Transfer Bandwidth
    [ 5] 0.0-10.0 sec 2.75 MBytes 2.30 Mbits/sec
    [ 8] 0.0-10.1 sec 4.50 MBytes 3.74 Mbits/sec
    [ 12] 0.0-10.1 sec 4.38 MBytes 3.64 Mbits/sec
    [ 11] 0.0-10.1 sec 4.38 MBytes 3.63 Mbits/sec
    [ 9] 0.0-10.1 sec 2.75 MBytes 2.28 Mbits/sec
    [ 10] 0.0-10.2 sec 4.50 MBytes 3.72 Mbits/sec
    [ 3] 0.0-10.2 sec 4.50 MBytes 3.71 Mbits/sec
    [ 6] 0.0-10.2 sec 4.38 MBytes 3.60 Mbits/sec
    [ 4] 0.0-10.2 sec 4.38 MBytes 3.61 Mbits/sec
    [ 7] 0.0-10.2 sec 2.75 MBytes 2.26 Mbits/sec
    [SUM] 0.0-10.2 sec 39.2 MBytes 32.2 Mbits/sec

    This is way lower than I got with speedtest.net from the desktop behind all the switches and crap wiring. so I'm not sure what to think of all this....

    I did keep an eye on the CPU of the pfSense box during all the tests, and it didn't seem to climb to much. What should I try next?


  • Banned

    With such an old system, is the TP-Link card simple PCI or PCIe?



  • Iperf from cable modem.lan to wan eth?


  • Netgate Administrator

    Better to use iperf3. You can install that on pfSense to use from the command line with pkg install iperf3.
    You can test reverse using that among other advantages.

    Testing either from or to the firewall is not a good test as it's optimised for routing not serving but it can still be useful for proving out some things. With iperf3 you can run that test with -R to get the reverse test to prove out the LAN is at least capable of over 280Mbps.

    You should try testing from your laptop using iperf against speedtest.serverius.net to prove that is a good server to test against. It may be limited to that speed anyway. It would be better to test against a local iperf3 server but otherwise find a public server that can do your WAN line rate when connected directly to the modem.

    Steve


  • Netgate Administrator

    @marvosa said in Gigabit Throughput:

    1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no.

    Unfortunately I don't have a Gigabit WAN to test with (I can only dream 😢 ) but in a local test with iperf3 client LAN side and iperf3 server WAN side, firewalling and NATing:

    [2.4.4-RELEASE][root@7100.stevew.lan]/root: iperf3 -c 172.21.16.76
    Connecting to host 172.21.16.76, port 5201
    [  5] local 192.168.112.10 port 47156 connected to 172.21.16.76 port 5201
    [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
    [  5]   0.00-1.00   sec   112 MBytes   940 Mbits/sec    0    160 KBytes       
    [  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    [  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    [  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec    0    160 KBytes       
    [  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    [  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    [  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    0    160 KBytes       
    [  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    160 KBytes       
    [  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    [  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate         Retr
    [  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec    0             sender
    [  5]   0.00-10.07  sec  1.09 GBytes   932 Mbits/sec                  receiver
    
    iperf Done.
    

    And during that the E8400 box shows:

    last pid: 73219;  load averages:  1.23,  0.80,  0.56                                               up 0+00:31:22  17:23:52
    233 processes: 6 running, 184 sleeping, 43 waiting
    CPU:  8.0% user,  0.2% nice, 37.3% system, 37.1% interrupt, 17.5% idle
    Mem: 348M Active, 243M Inact, 438M Wired, 196M Buf, 888M Free
    Swap: 1894M Total, 1894M Free
    
      PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
      351 root        52    0   224M   191M CPU0    0   0:04  48.39% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
       56 root        -8    -     0K    16K RUN     0   1:38  37.20% [md1]
       12 root       -92    -     0K   704K CPU0    0   1:53  37.20% [intr{irq261: em1:rx0}]
       12 root       -92    -     0K   704K WAIT    0   2:02  36.76% [intr{irq257: em0:rx0}]
       11 root       155 ki31     0K    32K RUN     1  26:12  18.42% [idle{idle: cpu1}]
       11 root       155 ki31     0K    32K RUN     0  25:13  12.20% [idle{idle: cpu0}]
       12 root       -92    -     0K   704K WAIT    1   0:05   1.52% [intr{irq258: em0:tx0}]
       12 root       -92    -     0K   704K WAIT    1   0:04   1.46% [intr{irq262: em1:tx0}]
      351 root        20    0   224M   191M uwait   1   0:01   1.00% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
      351 root        20    0   224M   191M uwait   0   0:00   0.61% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
    73735 unbound     20    0 31992K 16828K kqread  0   0:00   0.52% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbo
    66520 root        37    0 97732K 37496K accept  1   0:01   0.24% php-fpm: pool nginx (php-fpm){php-fpm}
       12 root       -60    -     0K   704K WAIT    1   0:04   0.20% [intr{swi4: clock (0)}]
    50995 root        20    0 50952K 39972K nanslp  1   0:00   0.13% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfbl
    54344 nobody     -22  r30  6500K  2608K nanslp  1   0:00   0.11% /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobod
       12 root       -72    -     0K   704K WAIT    0   0:00   0.08% [intr{swi1: netisr 1}]
    11335 root        20    0  9860K  4360K CPU1    1   0:00   0.08% top -aSH
    

    So I'm going to vote yes. And yes even more if I wasn't running Suricata on that box! 😉

    Steve



  • As expected an old core2duo can route and nat 1 gigabit of traffic.
    But the wan results above are horrible.
    Something is completely flawed. Could it be mtu? the wan card?
    i'd love to see some iperf3 tests cross box with local wan first.



  • @netblues said in Gigabit Throughput:

    Could it be mtu?

    Normally, you get the best results with the largest MTU. That's usually 1500, but would be 1492 on ADSL.



  • @jknott That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work. I doubt its an mtu issue, however the speed differences via pfsense is huge, and is difficult to believe its the network card unless it is faulty.
    A tp link ethernet might not be an intel igb, but still.



  • @netblues said in Gigabit Throughput:

    That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work.

    If that's the case, you should see ICMP "too big" messages. Fragmentation doesn't happen as often as it used to, as it's not allowed on IPv6 and the do not fragment flag is often used on IPv4. With Linux, that flag is set on everything, but only TCP on Windows.



  • Howdy all! Sorry I'm a bit behind. The WAN card was only PCI so I swapped it out with a new PCIx one. So here's what I have now:

    em0 00:0f:fe:c7:5e:bc (up) Intel(R) PRO/1000 Network Connection 7.6.1-k
    re0 18:a6:f7:01:1e:d0 (up) RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigab

    Here is iperf3 from the pfSense box cmd line with the new PCIx card:

    root: iperf3 -c iperf.scottlinux.com -p 5201
    Connecting to host iperf.scottlinux.com, port 5201
    [ 5] local 68.102.220.240 port 7956 connected to 45.33.39.39 port 5201
    [ ID] Interval Transfer Bitrate Retr Cwnd
    [ 5] 0.00-1.01 sec 5.23 MBytes 43.6 Mbits/sec 1 1.41 KBytes
    [ 5] 1.01-2.00 sec 2.02 MBytes 17.0 Mbits/sec 1397 546 KBytes
    [ 5] 2.00-3.00 sec 324 KBytes 2.65 Mbits/sec 45 2.85 KBytes
    [ 5] 3.00-4.00 sec 912 KBytes 7.49 Mbits/sec 19 272 KBytes
    [ 5] 4.00-5.00 sec 3.93 MBytes 33.0 Mbits/sec 0 288 KBytes
    [ 5] 5.00-6.00 sec 4.16 MBytes 34.9 Mbits/sec 0 304 KBytes
    [ 5] 6.00-7.00 sec 3.04 MBytes 25.5 Mbits/sec 14 157 KBytes
    [ 5] 7.00-8.00 sec 2.35 MBytes 19.7 Mbits/sec 0 174 KBytes
    [ 5] 8.00-9.00 sec 2.65 MBytes 22.2 Mbits/sec 0 190 KBytes
    [ 5] 9.00-10.00 sec 2.82 MBytes 23.6 Mbits/sec 0 206 KBytes


    [ ID] Interval Transfer Bitrate Retr
    [ 5] 0.00-10.00 sec 27.4 MBytes 23.0 Mbits/sec 1476 sender
    [ 5] 0.00-10.24 sec 25.5 MBytes 20.9 Mbits/sec receiver

    iperf Done.

    I've not tried flipping the interfaces and trying with the Intel card yet.

    Here is iperf3 from my desktop:

    iperf-3.1.3-win64>iperf3.exe -c iperf.scottlinux.com -p 5201
    Connecting to host iperf.scottlinux.com, port 5201
    [ 4] local 192.168.5.121 port 63682 connected to 45.33.39.39 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 2.12 MBytes 17.8 Mbits/sec
    [ 4] 1.00-2.00 sec 3.00 MBytes 25.1 Mbits/sec
    [ 4] 2.00-3.00 sec 3.00 MBytes 25.2 Mbits/sec
    [ 4] 3.00-4.00 sec 3.00 MBytes 25.2 Mbits/sec
    [ 4] 4.00-5.00 sec 2.88 MBytes 24.1 Mbits/sec
    [ 4] 5.00-6.00 sec 3.00 MBytes 25.1 Mbits/sec
    [ 4] 6.00-7.00 sec 2.88 MBytes 24.1 Mbits/sec
    [ 4] 7.00-8.00 sec 2.75 MBytes 23.1 Mbits/sec
    [ 4] 8.00-9.00 sec 3.12 MBytes 26.2 Mbits/sec
    [ 4] 9.00-10.00 sec 3.00 MBytes 25.2 Mbits/sec


    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-10.00 sec 28.8 MBytes 24.1 Mbits/sec sender
    [ 4] 0.00-10.00 sec 28.6 MBytes 24.0 Mbits/sec receiver

    iperf Done.

    Here is my connection from my desktop to the pfSense box:

    iperf3.exe -c 192.168.5.1 -p 5201
    Connecting to host 192.168.5.1, port 5201
    [ 4] local 192.168.5.121 port 63707 connected to 192.168.5.1 port 5201
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-1.00 sec 105 MBytes 882 Mbits/sec
    [ 4] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
    [ 4] 2.00-3.00 sec 108 MBytes 910 Mbits/sec
    [ 4] 3.00-4.00 sec 110 MBytes 924 Mbits/sec
    [ 4] 4.00-5.00 sec 111 MBytes 932 Mbits/sec
    [ 4] 5.00-6.00 sec 107 MBytes 896 Mbits/sec
    [ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
    [ 4] 7.00-8.00 sec 111 MBytes 932 Mbits/sec
    [ 4] 8.00-9.00 sec 111 MBytes 932 Mbits/sec
    [ 4] 9.00-10.00 sec 75.5 MBytes 633 Mbits/sec


    [ ID] Interval Transfer Bandwidth
    [ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec sender
    [ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec receiver

    iperf Done.

    Where should I look next?


  • Netgate Administrator

    You have a lot of retries on that first test. Try in the reverse direction. You probably don't need to specify the port, 5201 is the default:
    iperf3 -c iperf.scottlinux.com -R

    Run a test to that server from a client connected directly to the modem without pfSense in play. Make sure you can get a reasonable rate to that server over your WAN at all.

    Steve


  • LAYER 8 Global Moderator

    testing out to the internet for iperf not really going to be a valid test... Test from your lan to your wan network.. Put a box on your wan and a box on your lan and run iperf between them..

    Once you go out to the internet you have to many variables to rule out just something on net is problem.. Rule out your local hardware first. By testing local!!

    C:\tools\iperf3.6_64bit>iperf3.exe -c iperf.scottlinux.com -p 5201
    Connecting to host iperf.scottlinux.com, port 5201
    [  5] local 2001:470:<snipped>:7157:1522 port 54061 connected to 2600:3c01::f03c:91ff:fed5:ed33 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec  1.50 MBytes  12.6 Mbits/sec
    [  5]   1.00-2.00   sec  3.25 MBytes  27.2 Mbits/sec
    [  5]   2.00-3.00   sec  2.50 MBytes  21.0 Mbits/sec
    [  5]   3.00-4.00   sec  2.25 MBytes  18.9 Mbits/sec
    [  5]   4.00-5.00   sec  2.12 MBytes  17.8 Mbits/sec
    [  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec
    [  5]   6.00-7.00   sec   640 KBytes  5.24 Mbits/sec
    [  5]   7.00-8.00   sec   896 KBytes  7.34 Mbits/sec
    [  5]   8.00-9.00   sec   896 KBytes  7.34 Mbits/sec
    [  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec  16.5 MBytes  13.8 Mbits/sec                  sender
    [  5]   0.00-10.09  sec  15.7 MBytes  13.0 Mbits/sec                  receiver
    
    iperf Done.
    
    C:\tools\iperf3.6_64bit>
    

    Just connected to that scott site you used and the speed was utter crap!!! but I know for a fact my hardware and internet connection are fine.. I see my full pipe all the time downloading and uploading.


  • LAYER 8 Global Moderator

    see my edit... Put a box on your wan network of pfsense and do your testing...

    As you can see from that edit - that site is not very reliable for what your speed is or should be..

    Here just started download of file from one of my servers in the NL... I have a 500mbps connection seeing 53MBps down... My connection is fine - but that scott iperf showing junk..

    0_1544626409419_speed.png



  • Depend of your location you should try nearest public iperf3 server.
    Choose it here: https://iperf.cc
    And try to check your bandwidth again.


  • Netgate Administrator

    Yes, there are lots of public iperf servers you could use. The one you tested against does not appear to be particularly fast, at least not from where you are.

    But the best test you can do is to run your own server locally on the WAN side.

    Steve



  • My setup uses a nearly identical Core 2 Due E8500 CPU and I can reliably achieve 900Mbps+ throughput. The network cards make a difference. Also your PC should be fast. I have an older system based on Xeon W3550, and it never gets more than 600Mbps from Internet, testing with iperf or doing an ISP speed test.



  • Howdy all! So I did purchase a PCIe Intel NIC, but don't see a significant change. What is the best way to test the WAN port with iperf? Is a windows laptop ok?


  • Netgate Administrator

    It's probably fine.

    Before you test the firewall throughput put both the iperf3 server and client machines on the same subnet and test between them directly. Make sure you can see gigabit line rate in both directions.

    Then move the server machine into the WAN subet and test against it with the client on the LAN.

    If you don't see >900Mbps both ways then look for errors on the interfaces. Try running top -aSH on the firewall during the test to see the cpu core loading. Be sure not to have the dashboard up in a browser as that can use significant CPU cycles depending on what widgets you have loaded.

    Steve



  • This is my test :

    0_1545093184825_3Gbit iPerf.PNG

    I run pfSense on Cisco UCS C210 M2 with 2x X5650 CPU and BroadCom QLogic dual port 10G NIC... Maximum load I registered was 11%...
    I am pretty sure this result is caused by a speed limitations between me and server instead of my pfSense box... After a few day I will have second 10G line from separate ISP and then I can test again... This machine was released in 2010 so almost 9 years old but works pretty well and I am happy with it ;)


Log in to reply