PC Engines apu2 experiences

saltandpepper

Hi

Currently im running in APU2. Till 2 weeks ago I was able to get 1gibt trough my Box. 1Gbit is on the edge but it worked. Around 2 weeks ago I startet to play with traffic shapping. Unfortunatly since then I was not able to revert back to a setup that gets nearly close to my previous performance.

To check if the issue us on my side or my ISP i changed the router, which gave me my Gbit back.

Since im quite sertain that the issue is in my network i startet to test with iperf. No matter how much connection i open i only get around 400mbit trough. If only one connection is, 20 or 100. Around 400 - 420mbit is the limit. Previous i got 940mbit as expected with iperf. I already startet to test my switches, but I get 940mbit to the last switch.
After i tested all my equitment, I reinstalled Pfsense and did my confing manualy. Still 400mbit, in up and download. The settings that i use are the same as mentioned by dugeem.

@dugeem said in PC Engines apu2 experiences:

My current APU2 performance tweak summary:

Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)

Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local

There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more

~~The only thing that i have not done yet is a bios rollback to 4.11.04~~
Still 400mbit

Maybe someone has an idea whats currently is wrong with my setup.

Cheers

fireodo

Update APU2 Bios from v4.11.0.6 to v4.12.0.1 on 2 boxes - until now without issues.

Regards,
fireodo

Qinn

@fireodo Will try tomorrow, thanks for reporting!

FLOK

Hi folks,

since yesterday i have fiber at home with 300MBits Up and down.
But i cant get a real good performance....okay....its really poor.
So i made an Bios upgrade on my APU2D4 from 4.07 to 4.12.01.

Then created the /boot/loader.conf.local file an inserted (with nano):
hw.igb.rx_process_limit="-1 ".

TSO and LRO are disabled.
Reboot...

But still a poor performance: Iperf3 to an public Server ~30MBits/s.
Here a my tunables (are they all necessary ??)

Any hints ?

Qinn

@FLOK I hope this might helps...

A reboot after an bios update is not enough, it is recommended to shutdown the APU2 and make it powerless, this can be done gracefully using the GUI and go to Diagnostics -> Halt system and wait for the lights on the APU2 to go down, than remove the powerplug and wait for 30 sec and put it back in, hope it helps.

FLOK

@Qinn
Okay. Will try tomorrow...

stephenw10

But 30Mbps is very poor for a 300Mbps connection. It should not require any special tuning to get 300Mbps, the APU1 can pass that.
It seems like you have a more fundamental problem in play somewhere.

Steve

Qinn

Not that it explains te very low speed (about 10% of the attainable), but why do you use 7 DNS servers?

DaddyGo

with these tun / mod., they work stably up to 500Mbps (ISP!!!)

/boot/loader.conf.local

++++ edit: APU Core Performance Boost
https://github.com/pcengines/apu2-documentation/blob/master/docs/apu_CPU_boost.md

FLOK

@DaddyGo
Are there more tunables, or is this your complete list ?

DaddyGo

that's all for the APU tun

since Coreboot development is very unstable, I have stopped here for now: v4.10.0.1 /perhaps v4.10.0.4

+++the 30 Mbps is still very suspicious of this...., I think there is another problem too
we use it with 500/200 FTTH in several places without any issue
(true, this is the end of APU)

Veldkornet

What’s the hardware offload settings like?

Web panel -> System -> Advanced -> Networking -> Scroll to the bottom.

Make sure that all 3 first checkboxes under "Network Interfaces" are unchecked.

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading

https://teklager.se/media/filer_public_thumbnails/filer_public/45/9a/459a5e96-ec62-4090-9b1c-55dcf60166e9/pfsense_1gbit_apu2_config_panel.png__2280x1024_q85_subsampling-2.png

saltandpepper

Offloading is used to execute functions of the router using the hardware directly, instead of a process of software functions.
The post of @dugeem has some information about the performance impact of those functions and the tweaks from teklager.

@dugeem said in PC Engines apu2 experiences:

@kevindd992002

Unfortunately some of the advice in the link is incorrect.

Firstly TSO & LRO should always be disabled on routers. Netgate recommend this (hence pfSense defaults) as do others. BSDRP have even tested this and found routing performance drop negligible from enabling TSO & LRO (see link below).

In terms of loader.conf.local suggestions:

hw.igb.rx_process_limit=-1 is a standard tweak for Intel igb NICs. Performance boost on APU2 though is only ~1%.

hw.igb.tx_process_limit already defaults to -1 so no need to change this.

hw.igb.num_queues=1 is not for APU2 as stated. Default 0 allows driver to allocate maximum queues across CPU cores (i210AT has 4 queues; i211AT has 2 queues)

kern.ipc.nmbclusters=1000000 is unnecessary - default on APU2 with 4GB RAM is ~250000 (mbuf is 2kB - so this represents maximum 12% of RAM). Possibly for high bandwidth routers 500000 mbufs would be prudent. However use the command netstat -m to verify mbuf use prior to changing.

net.pf.states_hashsize=2097152 is ridculous for an APU2. If you need to be tweaking this then you'll likely need better hardware.

hw.igb.rxd=4096 & hw.igb.txd=4096. Increasing NIC descriptors on APU2 will actually decrease performance by 20%. And likely worsens buffer bloat. Default of 1024 is fine.

net.inet.tcp.* sysctl tuning is for end clients (ie not routers).

Reference performance data for some of the above: https://bsdrp.net/documentation/technical_docs/performance#nic_drivers_tuning

Even longer version: https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf

The only caveat is that these BSDRP performance numbers were compiled in 2018 before the AMD CPB was enabled in APU2 BIOS - so performance should now exceed this.

My current APU2 performance tweak summary:

Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)

Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local

There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more

kevindd992002

How's BIOS v4.12.0.1 doing for the one's who tested already?

fireodo

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I cannot find any issue ...

kevindd992002

@fireodo said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I can find any issue ...

Sorry, you can or you can't?

fireodo

@kevindd992002 said in PC Engines apu2 experiences:

@fireodo said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I can find any issue ...

Sorry, you can or you can't?

I cannot find any issue :-)

kinch

Hello
I bought an APU2E4 I have tried to increase the throughput with the help of this website. Supposedly a speed with only 1 connection at about 800Mbit/s and with 2 connections should be about 940 Mbit/s. But I reach with a measurement in LAN (iperf3) with one connection about 350Mbit/s.

The following is in loader.conf.local:

hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"

offloading is unchecked under advanced networking.

website:
https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
https://teklager.se/en/knowledge-base/apu-pfsense-throughput-bios-comparison/

BIOS: 4.12.0.1
pfsense: 2.4.5

What could be the problem here?

DaddyGo

this has long been out of date https://teklager.se/ !!!

This is the right @kevindd992002 !!!

@kinch Pls, check all offloading (because i210AT = multi - queue ethernet controller
(4 receive queues and 4 transmit queues))

Thanks @saltandpepper

edit:
the APU board Coreboot development team (https://3mdeb.com/) isn't specifically focused on FreeBSD, it's more Linux, so wait longer and don't update the BIOS right away

Think about how long it has not been recommended the Mainline releases for FreeBSD (pfSense) only Lagacy was the right way.

kevindd992002

@DaddyGo said in PC Engines apu2 experiences:

this has long been out of date https://teklager.se/ !!!

This is the right @kevindd992002 !!!

@kinch Pls, check all offloading (because i210AT = multi - queue ethernet controller
(4 receive queues and 4 transmit queues))

Thanks @saltandpepper

edit:
the APU board Coreboot development team (https://3mdeb.com/) isn't specifically focused on FreeBSD, it's more Linux, so wait longer and don't update the BIOS right away

Think about how long it has not been recommended the Mainline releases for FreeBSD (pfSense) only Lagacy was the right way.

I'm sorry but which info were you trying to tell me? I don't seem to understand your mention.