Download speed capped at 6xx Mbps on Gigabit link with Intel NIC

stephenw10

Probably this: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Try the suggested loader variable.

Steve

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

Probably this: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Try the suggested loader variable.

Steve

Thanks for the suggestion, i tried some of the suggestions on that page.

I now have the following custom System Tunables:

kern.ipc.nmbclusters="1000000"
net.isr.dispatch="deferred"
hw.igb.num_queues="1"

Rebooted, and still half speeds.

stephenw10

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

hw.igb.num_queues="1"

You definitely don't want that, it will kill your throughput everywhere else.

Setting net.isr.dispatch to deferred should give you some improvement if that's the issue you were hitting. You saw nothing?

Is it actually loaded? sysctl net.isr.dispatch

Unfortunately the CPU in that system appears to have a pretty terrible single thread performance. It's never going to be great with PPPoE in pfSense/FreeBSD.

Steve

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

You definitely don't want that, it will kill your throughput everywhere else.

Removed it, thanks.

Yup i checked, it is loaded (set to deferred).

The download speed never goes over 550~ Mbps, i will try to test with other hardware, got some old mb+cpu combos in the house. But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

stephenw10

Did you really see no difference in throughput setting net.isr.dispatch to deferred?

If that's the case you may not be hitting that.

Try running top -aSH at the command line while testing. See if you are hitting one core at 100%.

Steve

bigsy

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

WireGuard uses ChaCha20 as its encryption cipher, not AES.

nicolae

@bigsy said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

@nicolae said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

But i preferred this box because it's the only one with AES extension on the CPU and i need it for OpenVPN and WireGuard.

WireGuard uses ChaCha20 as its encryption cipher, not AES.

Thanks for pointing that out, unfortunately i still need AES because i mainly use OpenVPN, i have some MikroTik clients that need to connect to my home LAN. WireGuard is for my own personal use, which is not that heavy, only when i'm away from home.

nicolae

@stephenw10 said in Download speed capped at 6xx Mbps on Gigabit link with Intel NIC:

Did you really see no difference in throughput setting net.isr.dispatch to deferred?

If that's the case you may not be hitting that.

Try running top -aSH at the command line while testing. See if you are hitting one core at 100%.

Steve

So i tested again but from a Windows PC in the LAN, and now the DOWNLOAD goes all the way to almost 850 Mbps and the UPLOAD is close behind at 770~.

From my point of view i think the speedtest-cli on the pfSense box was the issue, that is what i was testing with, and now when i reach the above mentioned speeds on the LAN Windows PC, the speedtest-cli on the pfSense box never goes over 550 Mbps, although i am using the same server to test with.

Here are some videos with both speed tests, one from the Windows PC and one from the pfSense box, and top monitoring when they are in progress.

PC : http://s.go.ro/agzpjfwi | password: 566419
pfSense: http://s.go.ro/w9pzpy3w | password: 688860

stephenw10

Ah, yes speedtest from the firewall itself is a terrible way to test absolute speeds. It will always return a lower result that testing though the firewall.
850Mbps is probably as good as you will see over PPPoE using that CPU.

Steve

nicolae

@stephenw10 @bigsy

I think the speed is enough for me, i will consider buying or building new hardware.

Thanks for helping me sort this out!