@epiclper
There will be perhaps three things you could try out to
gain the throughput a bit more. But with 870 MBit/s
plus TCP overheat you will normally reaching the
range of 900 + something MBit/s and this with a
older 4 core CPU!!!
First point:
Install the last firmware 4.19.0.1 according to this HowTo.
APU Bios upgrade
PC Engines APU BIOS depot
Set up in the /boot/loader.conf.local the following entries;
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
hint.acpi_perf.0.disabled=1
Now your cpu will be not running anymore between
600MHz and 1000MHz, it is able to "run" from
1000MHz to 1400MHz, you should watch out
the entire CPU temperature too please!
Second point:
Since pfSense version 2.6 the entire WAN load will be
pulled over several queues, if you are not nailed to the
1 CPU core usage using PPPoE, you will be benefitting
from the 1 queue = 1 CPU core. That means in theoretic
more queues = more throughput. There are three
different numbers for the queues as I know it;
queue amount
queue length
queue size
Third point
The mbufsize can be tuned also, not even needed but also
nice to know. If you are size them up you could get a gain
from, with point of view towards to the throughput.
A tip from me, if you are installing a fresh pfSense 2.6
please install it and then test it out without any packages
installed and configured, your rules should be in place for
sure, but no packages please installed. So you will see the
entire throughput and you see then also what packages
are narrow down the entire speed later! I was setting up
at the installation using ZFS and size up the swap partition
to 4 GB, since that I am not using 60% -90% of my onboard soldered ram, I am using 39% ram and ~35% swap, so it free me a bit of ram for more headspace.
A side note, all available tunings can be single solve the
problem, but often it is a together working game play
of them, and to find out the bets option you must
perhaps do some more tests in either different configuration to get the most out for you.