Weird WAN performance issue. PfSense slower than clients.

AndyD

I have an issue where clients connecting to the internet via PfSense get better up/down speeds than the PfSense unit itself. 400-500Mbit/s on the client, vs. 4-5Mbit/s on PfSense.

How is this possible?

Hardware: Netgate XG1537, Intel Xeon 1,7 GHz, 8GB RAM
Network: PfSense is the network gateway, static IPv4; Single LAN subnet with two clients.

iperf3 output of client:

Connecting to host ping.online.net, port 5202
[  5] local 172.17.0.21 port 40404 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  61.4 MBytes   515 Mbits/sec  777   1.04 MBytes       
[  5]   1.00-2.00   sec  47.5 MBytes   398 Mbits/sec    0   1.12 MBytes       
[  5]   2.00-3.00   sec  50.0 MBytes   419 Mbits/sec    0   1.17 MBytes       
[  5]   3.00-4.00   sec  52.5 MBytes   440 Mbits/sec    0   1.20 MBytes       
[  5]   4.00-5.00   sec  52.5 MBytes   440 Mbits/sec    0   1.22 MBytes       
[  5]   5.00-6.00   sec  53.8 MBytes   451 Mbits/sec    0   1.24 MBytes       
[  5]   6.00-7.00   sec  53.8 MBytes   451 Mbits/sec    0   1.24 MBytes       
[  5]   7.00-8.00   sec  53.8 MBytes   451 Mbits/sec    0   1.25 MBytes       
[  5]   8.00-9.00   sec  55.0 MBytes   461 Mbits/sec    0   1.29 MBytes       
[  5]   9.00-10.00  sec  57.5 MBytes   482 Mbits/sec    0   1.32 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   538 MBytes   451 Mbits/sec  777             sender
[  5]   0.00-10.04  sec   536 MBytes   447 Mbits/sec                  receiver

iperf Done.

iperf3 output when running directly on pfsense:

Connecting to host ping.online.net, port 5202
[  5] local PUBLIC_IPv4_OF_PFSENSE port 31368 connected to 51.158.1.21 port 5202
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   384 KBytes  3.12 Mbits/sec   17   12.7 KBytes
[  5]   1.01-2.06   sec   768 KBytes  5.97 Mbits/sec   19   9.28 KBytes
[  5]   2.06-3.00   sec   384 KBytes  3.35 Mbits/sec    6   10.0 KBytes
[  5]   3.00-4.01   sec   640 KBytes  5.22 Mbits/sec   12   18.5 KBytes
[  5]   4.01-5.06   sec   512 KBytes  3.97 Mbits/sec   18   9.37 KBytes
[  5]   5.06-6.04   sec   512 KBytes  4.30 Mbits/sec   19   9.09 KBytes
[  5]   6.04-7.01   sec   896 KBytes  7.57 Mbits/sec   26   7.10 KBytes
[  5]   7.01-8.00   sec   384 KBytes  3.16 Mbits/sec   10   11.4 KBytes
[  5]   8.00-9.05   sec   384 KBytes  3.00 Mbits/sec    5   8.85 KBytes
[  5]   9.05-10.04  sec   640 KBytes  5.29 Mbits/sec   13   11.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  5.38 MBytes  4.49 Mbits/sec  145             sender
[  5]   0.00-10.06  sec  5.35 MBytes  4.46 Mbits/sec                  receiver

iperf Done.

AndyD

To answer my own question: The problem is due to TCP packet reordering, which the default TCP stack of freeBSD 15 does not handle very well.

The solution would be to activate the RACK TCP stack available in freeBSD. However, pfSense+ has this feature of stock freeBSD disabled.
https://freebsdfoundation.org/our-work/journal/browser-based-edition/networking-10th-anniversary/rack-and-alternate-tcp-stacks-for-freebsd/

I created an issue on the PfSense redmine and ask anyone experiencing similar issues to support it: https://redmine.pfsense.org/issues/15813