Download speed capped at 6xx Mbps on Gigabit link with Intel NIC

nicolae

Hello.

I have a pfSense box (MB+CPU combo AsRock QC5000M (Quad Core) + 4GB RAM + 2 x Intel Gigabit CT EXPI9301CTBLK) that i use as my primary router (I disabled the onboard Realtek NIC from the BIOS).

I have one ISP connected via ethernet to my home and PPPoE and a 1Gbps plan.

When i use the computer directly into the ISP ethernet cable i get around 9xx DOWN and 7xx UP Mbps speeds.

When i get routed via my pfSense box (pc plugged directly into the LAN NIC, no switches etc, and i tested and tried multiple cables, all are good and are Cat6 or Cat5E, and got the same results roughly) i never get pass 600 Mbps and the upload is always around 100Mbps higher (strange for me, i always see better download speeds)

As a side note about computers in my LAN routed via pfSense box, the CPU load of pfSense is always 20% when using speedtest on one PC. If i use it on 3 PCs at the same time with a gigabit dumb switch, CPU load goes over 50%.

I did a final test with speedtest-cli directly via ssh on the pfSense box, and the results are the same, somehow it looks capped at 600 Mbps DOWN and 700 Mbps UP.

P.S. All tests are made with speedtest and speedtest-cli with the same server (i provide specifically the id of the ISP server which is just 1KM away)

P.P.S. I never had this issue with my Mikrotik Router which i had before this pfSense build.

Any ideas on what might be wrong?

I don't have any custom tuning flags, just stock pfSense with minimal configuration.

stephenw10

Probably this: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Try the suggested loader variable.

Steve

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

Probably this: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Try the suggested loader variable.

Steve

Thanks for the suggestion, i tried some of the suggestions on that page.

I now have the following custom System Tunables:

kern.ipc.nmbclusters="1000000"
net.isr.dispatch="deferred"
hw.igb.num_queues="1"

Rebooted, and still half speeds.

stephenw10

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

hw.igb.num_queues="1"

You definitely don't want that, it will kill your throughput everywhere else.

Setting net.isr.dispatch to deferred should give you some improvement if that's the issue you were hitting. You saw nothing?

Is it actually loaded? sysctl net.isr.dispatch

Unfortunately the CPU in that system appears to have a pretty terrible single thread performance. It's never going to be great with PPPoE in pfSense/FreeBSD.

Steve

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

You definitely don't want that, it will kill your throughput everywhere else.

Removed it, thanks.

Yup i checked, it is loaded (set to deferred).

The download speed never goes over 550~ Mbps, i will try to test with other hardware, got some old mb+cpu combos in the house. But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

stephenw10

Did you really see no difference in throughput setting net.isr.dispatch to deferred?

If that's the case you may not be hitting that.

Try running top -aSH at the command line while testing. See if you are hitting one core at 100%.

Steve

bigsy

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

WireGuard uses ChaCha20 as its encryption cipher, not AES.

nicolae

@bigsy said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

WireGuard uses ChaCha20 as its encryption cipher, not AES.

Thanks for pointing that out, unfortunately i still need AES because i mainly use OpenVPN, i have some MikroTik clients that need to connect to my home LAN. WireGuard is for my own personal use, which is not that heavy, only when i'm away from home.

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

Did you really see no difference in throughput setting net.isr.dispatch to deferred?

If that's the case you may not be hitting that.

Try running top -aSH at the command line while testing. See if you are hitting one core at 100%.

Steve

So i tested again but from a Windows PC in the LAN, and now the DOWNLOAD goes all the way to almost 850 Mbps and the UPLOAD is close behind at 770~.

From my point of view i think the speedtest-cli on the pfSense box was the issue, that is what i was testing with, and now when i reach the above mentioned speeds on the LAN Windows PC, the speedtest-cli on the pfSense box never goes over 550 Mbps, although i am using the same server to test with.

Here are some videos with both speed tests, one from the Windows PC and one from the pfSense box, and top monitoring when they are in progress.

PC : http://s.go.ro/agzpjfwi | password: 566419
pfSense: http://s.go.ro/w9pzpy3w | password: 688860

stephenw10

Ah, yes speedtest from the firewall itself is a terrible way to test absolute speeds. It will always return a lower result that testing though the firewall.
850Mbps is probably as good as you will see over PPPoE using that CPU.

Steve

nicolae

@stephenw10 @bigsy

I think the speed is enough for me, i will consider buying or building new hardware.

Thanks for helping me sort this out!