I run pfSense on Cisco UCS C210 M2 with 2x X5650 CPU and BroadCom QLogic dual port 10G NIC... Maximum load I registered was 11%...
I am pretty sure this result is caused by a speed limitations between me and server instead of my pfSense box... After a few day I will have second 10G line from separate ISP and then I can test again... This machine was released in 2010 so almost 9 years old but works pretty well and I am happy with it ;)

Before you test the firewall throughput put both the iperf3 server and client machines on the same subnet and test between them directly. Make sure you can see gigabit line rate in both directions.

Then move the server machine into the WAN subet and test against it with the client on the LAN.

If you don't see >900Mbps both ways then look for errors on the interfaces. Try running top -aSH on the firewall during the test to see the cpu core loading. Be sure not to have the dashboard up in a browser as that can use significant CPU cycles depending on what widgets you have loaded.

Steve

But the best test you can do is to run your own server locally on the WAN side.

Steve

As you can see from that edit - that site is not very reliable for what your speed is or should be..

Here just started download of file from one of my servers in the NL... I have a 500mbps connection seeing 53MBps down... My connection is fine - but that scott iperf showing junk..

Once you go out to the internet you have to many variables to rule out just something on net is problem.. Rule out your local hardware first. By testing local!!

C:\tools\iperf3.6_64bit>iperf3.exe -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[  5] local 2001:470:<snipped>:7157:1522 port 54061 connected to 2600:3c01::f03c:91ff:fed5:ed33 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.50 MBytes  12.6 Mbits/sec
[  5]   1.00-2.00   sec  3.25 MBytes  27.2 Mbits/sec
[  5]   2.00-3.00   sec  2.50 MBytes  21.0 Mbits/sec
[  5]   3.00-4.00   sec  2.25 MBytes  18.9 Mbits/sec
[  5]   4.00-5.00   sec  2.12 MBytes  17.8 Mbits/sec
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec
[  5]   6.00-7.00   sec   640 KBytes  5.24 Mbits/sec
[  5]   7.00-8.00   sec   896 KBytes  7.34 Mbits/sec
[  5]   8.00-9.00   sec   896 KBytes  7.34 Mbits/sec
[  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  16.5 MBytes  13.8 Mbits/sec                  sender
[  5]   0.00-10.09  sec  15.7 MBytes  13.0 Mbits/sec                  receiver

iperf Done.

C:\tools\iperf3.6_64bit>

Just connected to that scott site you used and the speed was utter crap!!! but I know for a fact my hardware and internet connection are fine.. I see my full pipe all the time downloading and uploading.

Run a test to that server from a client connected directly to the modem without pfSense in play. Make sure you can get a reasonable rate to that server over your WAN at all.

Steve

em0 00:0f:fe:c7:5e:bc (up) Intel(R) PRO/1000 Network Connection 7.6.1-k
re0 18:a6:f7:01:1e:d0 (up) RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigab

Here is iperf3 from the pfSense box cmd line with the new PCIx card:

root: iperf3 -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 5] local 68.102.220.240 port 7956 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 5.23 MBytes 43.6 Mbits/sec 1 1.41 KBytes
[ 5] 1.01-2.00 sec 2.02 MBytes 17.0 Mbits/sec 1397 546 KBytes
[ 5] 2.00-3.00 sec 324 KBytes 2.65 Mbits/sec 45 2.85 KBytes
[ 5] 3.00-4.00 sec 912 KBytes 7.49 Mbits/sec 19 272 KBytes
[ 5] 4.00-5.00 sec 3.93 MBytes 33.0 Mbits/sec 0 288 KBytes
[ 5] 5.00-6.00 sec 4.16 MBytes 34.9 Mbits/sec 0 304 KBytes
[ 5] 6.00-7.00 sec 3.04 MBytes 25.5 Mbits/sec 14 157 KBytes
[ 5] 7.00-8.00 sec 2.35 MBytes 19.7 Mbits/sec 0 174 KBytes
[ 5] 8.00-9.00 sec 2.65 MBytes 22.2 Mbits/sec 0 190 KBytes
[ 5] 9.00-10.00 sec 2.82 MBytes 23.6 Mbits/sec 0 206 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 27.4 MBytes 23.0 Mbits/sec 1476 sender
[ 5] 0.00-10.24 sec 25.5 MBytes 20.9 Mbits/sec receiver

iperf Done.

I've not tried flipping the interfaces and trying with the Intel card yet.

Here is iperf3 from my desktop:

iperf-3.1.3-win64>iperf3.exe -c iperf.scottlinux.com -p 5201
Connecting to host iperf.scottlinux.com, port 5201
[ 4] local 192.168.5.121 port 63682 connected to 45.33.39.39 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 2.12 MBytes 17.8 Mbits/sec
[ 4] 1.00-2.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 2.00-3.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 3.00-4.00 sec 3.00 MBytes 25.2 Mbits/sec
[ 4] 4.00-5.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 5.00-6.00 sec 3.00 MBytes 25.1 Mbits/sec
[ 4] 6.00-7.00 sec 2.88 MBytes 24.1 Mbits/sec
[ 4] 7.00-8.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 4] 8.00-9.00 sec 3.12 MBytes 26.2 Mbits/sec
[ 4] 9.00-10.00 sec 3.00 MBytes 25.2 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 28.8 MBytes 24.1 Mbits/sec sender
[ 4] 0.00-10.00 sec 28.6 MBytes 24.0 Mbits/sec receiver

iperf Done.

Here is my connection from my desktop to the pfSense box:

iperf3.exe -c 192.168.5.1 -p 5201
Connecting to host 192.168.5.1, port 5201
[ 4] local 192.168.5.121 port 63707 connected to 192.168.5.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 105 MBytes 882 Mbits/sec
[ 4] 1.00-2.00 sec 111 MBytes 931 Mbits/sec
[ 4] 2.00-3.00 sec 108 MBytes 910 Mbits/sec
[ 4] 3.00-4.00 sec 110 MBytes 924 Mbits/sec
[ 4] 4.00-5.00 sec 111 MBytes 932 Mbits/sec
[ 4] 5.00-6.00 sec 107 MBytes 896 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec
[ 4] 7.00-8.00 sec 111 MBytes 932 Mbits/sec
[ 4] 8.00-9.00 sec 111 MBytes 932 Mbits/sec
[ 4] 9.00-10.00 sec 75.5 MBytes 633 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec sender
[ 4] 0.00-10.00 sec 1.04 GBytes 891 Mbits/sec receiver

iperf Done.

Where should I look next?

That is competely true, however you get very bad results if mtu is needed to be less and fragmentation doesn;t work.

If that's the case, you should see ICMP "too big" messages. Fragmentation doesn't happen as often as it used to, as it's not allowed on IPv6 and the do not fragment flag is often used on IPv4. With Linux, that flag is set on everything, but only TCP on Windows.

Could it be mtu?

Normally, you get the best results with the largest MTU. That's usually 1500, but would be 1492 on ADSL.

1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no.

Unfortunately I don't have a Gigabit WAN to test with (I can only dream ) but in a local test with iperf3 client LAN side and iperf3 server WAN side, firewalling and NATing:

[2.4.4-RELEASE][root@7100.stevew.lan]/root: iperf3 -c 172.21.16.76
Connecting to host 172.21.16.76, port 5201
[  5] local 192.168.112.10 port 47156 connected to 172.21.16.76 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   940 Mbits/sec    0    160 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec    0    160 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec    0    160 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    160 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    160 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec    0             sender
[  5]   0.00-10.07  sec  1.09 GBytes   932 Mbits/sec                  receiver

iperf Done.

And during that the E8400 box shows:

last pid: 73219;  load averages:  1.23,  0.80,  0.56                                               up 0+00:31:22  17:23:52
233 processes: 6 running, 184 sleeping, 43 waiting
CPU:  8.0% user,  0.2% nice, 37.3% system, 37.1% interrupt, 17.5% idle
Mem: 348M Active, 243M Inact, 438M Wired, 196M Buf, 888M Free
Swap: 1894M Total, 1894M Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
  351 root        52    0   224M   191M CPU0    0   0:04  48.39% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
   56 root        -8    -     0K    16K RUN     0   1:38  37.20% [md1]
   12 root       -92    -     0K   704K CPU0    0   1:53  37.20% [intr{irq261: em1:rx0}]
   12 root       -92    -     0K   704K WAIT    0   2:02  36.76% [intr{irq257: em0:rx0}]
   11 root       155 ki31     0K    32K RUN     1  26:12  18.42% [idle{idle: cpu1}]
   11 root       155 ki31     0K    32K RUN     0  25:13  12.20% [idle{idle: cpu0}]
   12 root       -92    -     0K   704K WAIT    1   0:05   1.52% [intr{irq258: em0:tx0}]
   12 root       -92    -     0K   704K WAIT    1   0:04   1.46% [intr{irq262: em1:tx0}]
  351 root        20    0   224M   191M uwait   1   0:01   1.00% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
  351 root        20    0   224M   191M uwait   0   0:00   0.61% /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suric
73735 unbound     20    0 31992K 16828K kqread  0   0:00   0.52% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbo
66520 root        37    0 97732K 37496K accept  1   0:01   0.24% php-fpm: pool nginx (php-fpm){php-fpm}
   12 root       -60    -     0K   704K WAIT    1   0:04   0.20% [intr{swi4: clock (0)}]
50995 root        20    0 50952K 39972K nanslp  1   0:00   0.13% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfbl
54344 nobody     -22  r30  6500K  2608K nanslp  1   0:00   0.11% /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobod
   12 root       -72    -     0K   704K WAIT    0   0:00   0.08% [intr{swi1: netisr 1}]
11335 root        20    0  9860K  4360K CPU1    1   0:00   0.08% top -aSH

So I'm going to vote yes. And yes even more if I wasn't running Suricata on that box!

Steve

Testing either from or to the firewall is not a good test as it's optimised for routing not serving but it can still be useful for proving out some things. With iperf3 you can run that test with -R to get the reverse test to prove out the LAN is at least capable of over 280Mbps.

You should try testing from your laptop using iperf against speedtest.serverius.net to prove that is a good server to test against. It may be limited to that speed anyway. It would be better to test against a local iperf3 server but otherwise find a public server that can do your WAN line rate when connected directly to the modem.

Steve

/root: iperf -c 192.168.5.121 -p 5201

Client connecting to 192.168.5.121, TCP port 5201
TCP window size: 64.2 KByte (default)

[ 3] local 192.168.5.1 port 56750 connected with 192.168.5.121 port 5201

write failed: Broken pipe
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 9.5 sec 884 MBytes 783 Mbits/sec

My desktop is behind three switches and crap wiring.

Next, I connected a laptop directly to the cable modem and got in the 800s on speedtest.net several times.

Then I attempted to use iperf on pfSense to contact several public servers I found and got results like below several times:

root: iperf -c speedtest.serverius.net -P 10

Client connecting to speedtest.serverius.net, TCP port 5001
TCP window size: 64.2 KByte (default)

[ 7] local 68.102.217.178 port 53624 connected with 178.21.16.76 port 5001
[ 11] local 68.102.217.178 port 29834 connected with 178.21.16.76 port 5001
[ 5] local 68.102.217.178 port 24828 connected with 178.21.16.76 port 5001
[ 9] local 68.102.217.178 port 53592 connected with 178.21.16.76 port 5001
[ 3] local 68.102.217.178 port 42250 connected with 178.21.16.76 port 5001
[ 10] local 68.102.217.178 port 44675 connected with 178.21.16.76 port 5001
[ 8] local 68.102.217.178 port 52098 connected with 178.21.16.76 port 5001
[ 12] local 68.102.217.178 port 29426 connected with 178.21.16.76 port 5001
[ 6] local 68.102.217.178 port 50963 connected with 178.21.16.76 port 5001
[ 4] local 68.102.217.178 port 41528 connected with 178.21.16.76 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 2.75 MBytes 2.30 Mbits/sec
[ 8] 0.0-10.1 sec 4.50 MBytes 3.74 Mbits/sec
[ 12] 0.0-10.1 sec 4.38 MBytes 3.64 Mbits/sec
[ 11] 0.0-10.1 sec 4.38 MBytes 3.63 Mbits/sec
[ 9] 0.0-10.1 sec 2.75 MBytes 2.28 Mbits/sec
[ 10] 0.0-10.2 sec 4.50 MBytes 3.72 Mbits/sec
[ 3] 0.0-10.2 sec 4.50 MBytes 3.71 Mbits/sec
[ 6] 0.0-10.2 sec 4.38 MBytes 3.60 Mbits/sec
[ 4] 0.0-10.2 sec 4.38 MBytes 3.61 Mbits/sec
[ 7] 0.0-10.2 sec 2.75 MBytes 2.26 Mbits/sec
[SUM] 0.0-10.2 sec 39.2 MBytes 32.2 Mbits/sec

This is way lower than I got with speedtest.net from the desktop behind all the switches and crap wiring. so I'm not sure what to think of all this....

I did keep an eye on the CPU of the pfSense box during all the tests, and it didn't seem to climb to much. What should I try next?

the system itself will do that just fine, the TP-link nic will most likely be the culprit

Can it do it mathematically in theory on paper? Possibly... but the question is... can 11-year-old desktop architecture reliably route and sustain 1 Gbit firewalled throughput (930-940+ Mbit) over a gigabit WAN?... my vote is no. And that isn't necessarily just my opinion... even the vendor recommends "Server-class Hardware" to push anything over 500 Mbit.

Don't get me wrong... I'm not saying the box won't "work"... but it's all a matter of what each person considers acceptable... will his box route traffic, yes... will it be fast, sure (relatively)... but will it reliably sustain 940+ Mbit/sec of firewalled, real-world traffic over the gigabit WAN he's paying for... my vote is doubtful. And to me, that's unacceptable. Barring a misconfiguration that ends up limiting bandwidth by design, anytime I don't see full bandwidth minus some overhead, I typically start assessing hardware and infrastructure.

Pc to pf external int again measure 1g
If you see 1g everywhere and both ways then your options are

1. Establish plain routing via pf and test
2. Establish nat via pf and test
3. Setup a bridge with pf and test

Do monitor cpu usage on pf whole testing.
At this point you will have clear indications on what might be the issue

Apart from that, I would suspect congestion on your provider, but we need to rule out everything else.

are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.

the system itself will do that just fine, the TP-link nic will most likely be the culprit

The Core 2 Duo E8400 is almost an 11-year-old CPU... which means 11-year-old architecture throughout the box. Let's all have an honest moment of reflection and think about what the average residential internet speeds were at the beginning of 2008... maybe 10, 20, 30 Mbit if you were lucky? Now fast forward ~11 years... where 1, 2, and 10 Gbit speeds are available in residential areas ... which is 35-1000 times faster than what was available to most people back then... are we really expecting a 10+ year old desktop with a cheap TP-Link NIC to reliably route and filter traffic @ 1 Gbit throughput? To me, the logical answer to that question is no.

While hardware may not be your only issue... I can almost guarantee it's a contributing factor. Save yourself the dozens of hours (if not multiple weeks) banging your head against the wall and upgrade your hardware (NIC's included)... then re-assess.

But how exactly are you testing that?

Really I would like to see an iperf3 test with a client on one side of the firewall and server on the other.

Steve