APU2 uses 100% CPU while Gigabit Down/Up causing various issues

EpicLPer

Heya!

I'm currently in the process of reinstalling my APU2 with a fresh install of pfSense, but I'm kind of curious if someone has some ideas as to why this issue happened in the first place so I can avoid making the same mistake accidentally again :)
My knowledge about firewalls is a bit limited, but I'll try my best to provide as much info as I can!

Quick Overview:

APU2
coreboot v4.12.0.4
pfSense 2.6.0 CE
Synchronous Gigabit Fiber connection via VLAN to provider

The Issue:
Whenever I'm stressing my network quite a bit, as an example trying to download with full Gigabit speeds, the APU2 CPU spikes to an almost constant 100% usage and then becomes immensely sluggish to respond, even via SSH. As an example, I've moved quite a lot of files around via rclone recently, had 3 tabs á 20 connections open and with that entirely "disabled" other devices on the network until I stopped most rclone operations. It becomes so bad that UniFi can't assign DHCP addresses quick enough anymore, defaults to a random IP (like 10.0.0.10) which is outside the pfSense DHCP range and already used by another device. I've also never managed to use the full Gigabit connection, but I think this is simply down to the APU2 not being fully capable of doing so, max. I reach is ~870MBit/s down and ~750MBit/s up.
In short, trying to pull a Gigabit stresses the APU2 out so much it becomes unresponsive.

I'm not too familiar with network stuff or firewalls in general, I've had this setup for a few years now and finally narrowed the DHCP issue I had down to that being the case.

Enabled Services/Installed Packages:

avahi
bsnmpd
openvpn
A few others which I have disabled for now, though the issue is still occurring

APU Tweaks:
I've applied a few APU(2) tweaks I found both in the forums here and on other sites, this is my current /boot/loader.conf.local:

legal.intel_igb.license_ack="1"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
hw.em.rx_process_limit=-1
dev.igb.0.iflib.tx_abdicate=1
dev.igb.1.iflib.tx_abdicate=1
dev.igb.2.iflib.tx_abdicate=1

I've also disabled "Hardware Checksum Offloading", "Hardware TCP Segmentation Offloading" and "Hardware Large Receive Offloading" in System -> Advanced -> Networking , added net.inet6.ip6.redirect=0 and net.inet.ip.redirect=0 in System Tunables, and tried some other tweaks and things that I probably forgot over the years.

I'm not sure what else to look into anymore. I've talked with a friend of mine who then talked me into just trying to reinstall the whole system, which I'm currently in the process of doing so. But out of curiosity's sake (and to not just make a similar mistake on the new setup), I'd love to hear what kind of suggestions you guys have on this, and I already thank you for your time! :)

stephenw10

It looks like you've probably already read this: https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
If not that's probably the best source of info for that device.

Yes, you're probably just hitting the limits of the hardware. Not much more you can do there.

Steve

EpicLPer

@stephenw10 said in APU2 uses 100% CPU while Gigabit Down/Up causing various issues:

It looks like you've probably already read this: https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
If not that's probably the best source of info for that device.

Yes, you're probably just hitting the limits of the hardware. Not much more you can do there.

Steve

Yeah, that's where I got most my info from so far. Tho seen others hit "only" like 400-600MBit/s, that I'm getting around 900 is already a win to me lol.

Dobby_

@epiclper

There will be perhaps three things you could try out to
gain the throughput a bit more. But with 870 MBit/s
plus TCP overheat you will normally reaching the
range of 900 + something MBit/s and this with a
older 4 core CPU!!!

First point:
Install the last firmware 4.19.0.1 according to this HowTo.
APU Bios upgrade
PC Engines APU BIOS depot

Set up in the /boot/loader.conf.local the following entries;

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
hint.acpi_perf.0.disabled=1

Now your cpu will be not running anymore between
600MHz and 1000MHz, it is able to "run" from
1000MHz to 1400MHz, you should watch out
the entire CPU temperature too please!

Second point:
Since pfSense version 2.6 the entire WAN load will be
pulled over several queues, if you are not nailed to the
1 CPU core usage using PPPoE, you will be benefitting
from the 1 queue = 1 CPU core. That means in theoretic
more queues = more throughput. There are three
different numbers for the queues as I know it;

queue amount
queue length
queue size

Third point
The mbufsize can be tuned also, not even needed but also
nice to know. If you are size them up you could get a gain
from, with point of view towards to the throughput.

A tip from me, if you are installing a fresh pfSense 2.6
please install it and then test it out without any packages
installed and configured, your rules should be in place for
sure, but no packages please installed. So you will see the
entire throughput and you see then also what packages
are narrow down the entire speed later! I was setting up
at the installation using ZFS and size up the swap partition
to 4 GB, since that I am not using 60% -90% of my onboard soldered ram, I am using 39% ram and ~35% swap, so it free me a bit of ram for more headspace.

A side note, all available tunings can be single solve the
problem, but often it is a together working game play
of them, and to find out the bets option you must
perhaps do some more tests in either different configuration to get the most out for you.