PC Engines apu2 experiences

Veldkornet

What’s the hardware offload settings like?

Web panel -> System -> Advanced -> Networking -> Scroll to the bottom.

Make sure that all 3 first checkboxes under "Network Interfaces" are unchecked.

Hardware Checksum Offloading
Hardware TCP Segmentation Offloading
Hardware Large Receive Offloading

https://teklager.se/media/filer_public_thumbnails/filer_public/45/9a/459a5e96-ec62-4090-9b1c-55dcf60166e9/pfsense_1gbit_apu2_config_panel.png__2280x1024_q85_subsampling-2.png

saltandpepper

Offloading is used to execute functions of the router using the hardware directly, instead of a process of software functions.
The post of @dugeem has some information about the performance impact of those functions and the tweaks from teklager.

@dugeem said in PC Engines apu2 experiences:

@kevindd992002

Unfortunately some of the advice in the link is incorrect.

Firstly TSO & LRO should always be disabled on routers. Netgate recommend this (hence pfSense defaults) as do others. BSDRP have even tested this and found routing performance drop negligible from enabling TSO & LRO (see link below).

In terms of loader.conf.local suggestions:

hw.igb.rx_process_limit=-1 is a standard tweak for Intel igb NICs. Performance boost on APU2 though is only ~1%.

hw.igb.tx_process_limit already defaults to -1 so no need to change this.

hw.igb.num_queues=1 is not for APU2 as stated. Default 0 allows driver to allocate maximum queues across CPU cores (i210AT has 4 queues; i211AT has 2 queues)

kern.ipc.nmbclusters=1000000 is unnecessary - default on APU2 with 4GB RAM is ~250000 (mbuf is 2kB - so this represents maximum 12% of RAM). Possibly for high bandwidth routers 500000 mbufs would be prudent. However use the command netstat -m to verify mbuf use prior to changing.

net.pf.states_hashsize=2097152 is ridculous for an APU2. If you need to be tweaking this then you'll likely need better hardware.

hw.igb.rxd=4096 & hw.igb.txd=4096. Increasing NIC descriptors on APU2 will actually decrease performance by 20%. And likely worsens buffer bloat. Default of 1024 is fine.

net.inet.tcp.* sysctl tuning is for end clients (ie not routers).

Reference performance data for some of the above: https://bsdrp.net/documentation/technical_docs/performance#nic_drivers_tuning

Even longer version: https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf

The only caveat is that these BSDRP performance numbers were compiled in 2018 before the AMD CPB was enabled in APU2 BIOS - so performance should now exceed this.

My current APU2 performance tweak summary:

Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)

Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local

There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more

kevindd992002

How's BIOS v4.12.0.1 doing for the one's who tested already?

fireodo

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I cannot find any issue ...

kevindd992002

@fireodo said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I can find any issue ...

Sorry, you can or you can't?

fireodo

@kevindd992002 said in PC Engines apu2 experiences:

@fireodo said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

How's BIOS v4.12.0.1 doing for the one's who tested already?

Until now I can find any issue ...

Sorry, you can or you can't?

I cannot find any issue :-)

kinch

Hello
I bought an APU2E4 I have tried to increase the throughput with the help of this website. Supposedly a speed with only 1 connection at about 800Mbit/s and with 2 connections should be about 940 Mbit/s. But I reach with a measurement in LAN (iperf3) with one connection about 350Mbit/s.

The following is in loader.conf.local:

hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"

offloading is unchecked under advanced networking.

website:
https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
https://teklager.se/en/knowledge-base/apu-pfsense-throughput-bios-comparison/

BIOS: 4.12.0.1
pfsense: 2.4.5

What could be the problem here?

DaddyGo

this has long been out of date https://teklager.se/ !!!

This is the right @kevindd992002 !!!

@kinch Pls, check all offloading (because i210AT = multi - queue ethernet controller
(4 receive queues and 4 transmit queues))

Thanks @saltandpepper

edit:
the APU board Coreboot development team (https://3mdeb.com/) isn't specifically focused on FreeBSD, it's more Linux, so wait longer and don't update the BIOS right away

Think about how long it has not been recommended the Mainline releases for FreeBSD (pfSense) only Lagacy was the right way.

kevindd992002

@DaddyGo said in PC Engines apu2 experiences:

this has long been out of date https://teklager.se/ !!!

This is the right @kevindd992002 !!!

@kinch Pls, check all offloading (because i210AT = multi - queue ethernet controller
(4 receive queues and 4 transmit queues))

Thanks @saltandpepper

edit:
the APU board Coreboot development team (https://3mdeb.com/) isn't specifically focused on FreeBSD, it's more Linux, so wait longer and don't update the BIOS right away

Think about how long it has not been recommended the Mainline releases for FreeBSD (pfSense) only Lagacy was the right way.

I'm sorry but which info were you trying to tell me? I don't seem to understand your mention.

kinch

@DaddyGo

what do you mean, outdated?
How does his test differ from now?

DaddyGo

@kinch

Hi,

since then the guys (F.E. - @kevindd992002 ) who actively use the pfSense with APU boards overwritten this tweak ( https://teklager.se/)
our system has nearly 40 pcs. APU4d4s, so I speak from experience as well

what you read here ( https://teklager.se/) is just a test and suggestion, since then the experience gained is important, not a blog statement

DaddyGo

@kevindd992002 I'm just saying that your work is the appropriate guidance

kinch

@DaddyGo
unfortunately, none of the guys you mentioned posted a speed test. Also the question is not answered, how the boy from https://teklager.se/ could reach these values.

kevindd992002

You're probably referring to @dugeem's post here. It wasn't "my work" per se :)

DaddyGo

@kevindd992002

yes, I know but you were mentioned in this post and I passed it on
I didn't want to complicate the thread further with the question of author......... (ERROR on my part)
but you are right, the laurel is his: @dugeem

edit: you see how many misconceptions there are about APU

DaddyGo

@kinch
pls. don't get me wrong, I want to help

do you want to get first hand results, why is speed the most important thing?
(what kind of speed ISP connection do you have at home, how big is the internal network?)
is it a business or SOHO system you operate?

I'm happy to help you, because I know the limits of APU stuff as well (we've been using them for 4 -5 years for "homeworker" colleagues' endpoints)

FLOK

Hi again !
I did every trick you gave me...
Flashed back to Bios V4.10.0.1 - Halt System - Disconnect Power for 30sec.

Made changes is loader.conf.local
2020-06-04 17_34_18-Window.png

Made a few tunables (like DaddyGo)
2020-06-04 17_35_18-Window.png

Disabled the offloads
2020-06-04 17_36_59-Window.png

Reboot

Speedtest
2020-06-04 17_38_24-Window.png

Shit

DaddyGo

Never give up

it’s time to review your entire configuration
can we get a drawing of your system and more information

(F.E. ISP, WAN interface type, installed packages, LAN enviroment)
believe me the APU knows a lot in the SOHO category

the place of the joke:
"the error is in your device"
does not matter anyway, this is a European slang

edit: it was missed

DaddyGo

watch this, this is an APU 4d4 performance (I apologize for the Hungarian language)
yes, I live in lisbon now because

and

what is the lesson
is that it is not impossible

a small dictionary:

letöltés = download
feltöltés = upload
késleltetés = latency
átlag = average
remegés = jitter

Veldkornet

Personally, I got better performance by enabling hardware offload (clear the check boxes).

iPerf on local LAN from pfSense (client) to my NAS (server):

iperf3 -c SERVER_IP_HERE -P 4

With Hardware Offload: ~950Mbit/s

Without Hardware Offload: ~450Mbit/s