PC Engines apu2 experiences

stephenw10

A Former User

@kevindd992002

What is an alternative to the apu2c4 if I need to
upgrade mine to handle 1Gbps (or more) Internet speeds?

Since version 2.6 of pfSense it is not so easy to answer
if you are not using PPPoE the entire WAN connection
will be running over several queues and if your device is sorted with many CPU core, you can run more queues
over more CPU core for sure. So it is also pending on
the entire rest and/or other circumstances, as I see it
right. By the way if you are using PPPoE and you play
around with some settings here and there over a
weekend or so you may be see here and there also
something around 800 MBit/s - 900 MBit/s throughput
and with the overheat on top counting you will be
more near to your real 1 GBit/s as you could be.

Is also not that bad as I see it right, or?

As @stephenw10 wrote the Netgate 4100 or 6100 will be
nice to do so. Or if you need some stuff to realize what the
Netgate might be not able to serve you, I will be pretty sure the entire Supermicro Intel Atom C3x58 line will
be able to route a real 1 GBit/s at the WAN for you.

But with the Netgate ones you will be ensure that all
is running fine and compatible. And a pfSense+ (Plus)
or an TSNR option will be available too.

I personally owns three of the APU devices, one for
pfSense+ (Plus) with lab or home license, one with MikroTik RouterOS and one with OpenWRT. For my
home lap and /or home network it is enough and I
am happy with it.

dugeem

Hello all,

Some of you may recall a number of APU2 tweaks I posted in this thread back in April 2020. I promised that when my home internet evolved to 500Mb/s I'd worry some more. Well it's happened - I now have an asymmetric HFC 500/50 service up & running (for only a modest price increment on my previous 250/25 service).

For multiple connections the APU2 throughput is fine for home use but single connections are a bit constrained - typically maximum download is around ~370Mb/s. This seems a bit lower than expected.

As always there is YAT = Yet Another Tunable In this case the ability to change the FreeBSD 12 iflib interface TX tasking via tx_abdicate tunable (see FreeBSD iflib(4) man page)

It turns out that our BSDRP friends documented the FreeBSD 12 tx_abdicate sysctl back in 2019 (not useful for the pfSense community back then as pfSense was still using FreeBSD 11) ... https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_pc_engines_apu2#enabling_tx_abdicate_iflib_drivers

Caveats: The only known issue with tx_abdicate is if IPSEC is in use - enabling tx_abdicate may result in lower IPSEC throughput. In this case leave it disabled (ie default).

Enabling this tunable on the LAN interface yields a useful ~20% throughput boost - in my case for single connections I'm now seeing ~450 Mb/s (with no other config changes).

So a quick summary of my current APU2 tweaks:

Upgrade APU2 BIOS to enable CPB (mainline v4.9.0.2 or later - suggest v4.16 or later)
Add sysctl hw.em.rx_process_limit=-1 to /boot/loader.conf.local (note the prefix change from hw.igb to hw.em)
Add following sysctls to either /boot/loader.conf.local or /etc/sysctl.conf

dev.igb.0.iflib.tx_abdicate=1
dev.igb.1.iflib.tx_abdicate=1
dev.igb.2.iflib.tx_abdicate=1

NB1 yes there is sysctl device naming inconsistency ... hw.em & dev.igb
NB2 some of you might notice the suggestion in the linked BSDRP report to disable IP redirects - this is no longer necessary due to subsequent enhancements to FreeBSD 12-STABLE

Anyway for me the APU2 continues on - let's see what pfSense CE 2.7 / pfSense+ 23.01 (based on FreeBSD 14) will bring.

Enjoy!

stephenw10

Hm, that's interesting. I don't have an APU2 but they have i210 NICs which I expect to be recognised as igb. I wouldn't expect that hw.em sysctl to do anything. They are both the e1000 driver under iflib though.

Steve

dugeem

@stephenw10 Yes it's confusing. I am merely following advice from FreeBSD UPDATING

20170109:
	The igb(4), em(4) and lem(4) ethernet drivers are now implemented via
	IFLIB.  If you have a custom kernel configuration that excludes em(4)
	but you use igb(4), you need to re-add em(4) to your custom
	configuration.

In any case the rx_process_limit tweak was only a 1-2% improvement (at least with previous versions).

dugeem

In their usual style PC Engines have released a long term APU2 EOL statement.

Appears that the APU2 CPU - the AMD Embedded G series GX-412TC - will move to EOL status later this year. PC Engines are intending to do a final buy of AMD GX-412TC CPUs and associated components with APU2 board availability running into 2024.

No successor is currently planned by PC Engines.

PC Engines EOL notice: https://www.pcengines.ch/eol.htm

logan5247

@dugeem sad to see this, love my APU2.

Thoughts on a future replacement that is small, low-power, and fanless? I haven't checked the NICs on these, but these pop into mind right away...

Protectli
Fitlet3
Seed Studio Odyssey Blue
ZimaBoard
ODROID H3
A fanless SuperMicro?
Netgate 1100 or 2100
Some no-name AliExpress boxes?

stephenw10

What actual features would you be looking for?

logan5247

@stephenw10 I'm a home user, so pretty basic:

1Gbps
Handful of VLANs
pfBlockerNG

Just added the Netgate 1100 and 2100 to my list above.

stephenw10

1Gbps throughput or 1G NICs? Or both I guess...

Other hardware requirements? mPCIe slots? Intel NICs?

Stewart

@logan5247 I've looked for a while. I suppose PC Engines isn't exactly a well established brand but we've been using them since Netgate used them so they've always been a great reliable product. I don't know of any others that have a whitebox at that price that has an established US presence and distribution. Netgate directly has some good hardware as well as Protectli but both are quite a bit more expensive. Quotom seems to have intriguing models but no US distribution and many models appear only through AliExpress.

Also, ISPs are moving beyond 1Gb connections so we are moving up to have 2.5Gb ports. A few weeks ago I had a Spectrum rep tell me they are looking to roll out symmetrical coax which would be a big jump. All together I would think that the J4125 units would be under-powered in a year or two, even if it's 3x the processing of an APU2. Then all these newer units are using the i226-v chips now which isn't even available in mainstream pfSense yet. (I believe 2.7 will have support for that.)

I wish I had a recommendation for you. I've been hoping that there would be a successor to the APU2 built on a newer Zen platform with i-225V. I think those would sell incredibly well but it looks like that won't happen. I've been looking at a HUNSN model but was kinda waiting for the NIC support in pfSense. But then again, who is HUNSN and will they even be around in 3 years?

Dobby_

I only "know" that there is a Board called or named APU7 based on APU2 but with 2,5 GBit/s ports (Intel i225), but I
am not knowing when and where it will be sold, or even
if this will be the or a chance to get hands on for us all.

During the search fot the latest BIOS for my both APU4
and APU6 I was finding out that the latest BIOS was also
tested on a board that called APU7, that´s all I know about.

Pleas go to the bottom line of that link and choose APU7
to view the tests about and with it. PC Engines Regression test results

Pleas e have a look over the latest BIOS description:
PC Engines GitHub.io BIOS files

v4.19.0.1
Release date: 2023-02-02

Changed:
Rebased with official coreboot repository commit 2ccbcc5
Removed configuration and mainboard files for apu1 due to the board being dropped from upstream coreboot

*Known issues:
APU7 iPXE network boot with i225 NICs does not work

apuled driver doesn't work in FreeBSD.
Check the GPIOs document for workaround.

some PCIe cards are not detected on certain OSes and/or in certain mPCIe slots. Check the mPCIe modules document for solution/workaround.

booting with 2 USB 3.x sticks plugged in apu4 sometimes results in detecting only 1 stick certain USB 3.x sticks happen to not appear in boot menu

booting Xen is unstable

Rumors about PC Engines board here in Germany
were also talking about an device from PC Engines
that will be named or called IPU and comes out
after the APU series will be set really to EoL status,
but who knows if this will be all real?

VAMike

The handwriting was on the wall for a while. I don't think they ever had the volume to compete with the various multi-port boxes coming out of china on the high end (which can be configured with much more powerful processors than could fit in the APU form factor) nor with stuff like the raspberry pi on the low end. Soekris ran into the same problems but threw in the towel earlier. The niche they filled was just getting too small.

Stewart

@dobby_ The APU2 can barely do gigabit. Not sure how it was expected to go higher.

Stewart

@vamike Did they do any marketing or outreach? The only place I know them from is that we used to purchase them through Netgate before they took over pfSense. When they stopped using them we tracked them down and began purchasing and using them on our own. For the last several years they've just been our go-to box dozens of times. I hate to move on from them.

VAMike

@stewart to whom? for what purpose? I really just don't think there's enough of a market that doesn't want either 1) cheaper, 2) smaller/lower wattage, or 3) more powerful. And there are existing players in each of those segments. We used to buy a good number of APUs as well, but for the past couple of years we've been going with faster higher-core intel boxes with more interfaces, m.2, and expandable RAM, because we can use the extra capacity for not much more money and not much of a larger device---we just didn't need the specific constraints of the APU form factor. On the flip side, small ARM boards can be found cheaper, smaller, and lower power than the APU. Sure, there are people for whom the APU line is a better fit than anything else, the question is whether there's enough of them. Scale is a thing, and the lower the volume the more expensive each part becomes until it's not cost competitive at all. There are probably disproportionately more of them here given the limited architecture options for pfsense, but I don't think you can build a sustainable business on the set of people who want to run pfsense but don't want to buy hardware from netgate.

Stewart

@vamike I understand all of that. But as a company, if they want to increase volume, what did they do? We are the market they are/were building for. They aren't in Amazon to see them in searches. I've never seen any advertisement for them. They didn't come into forums for outreach from what I've ever seen. How were they going to increase volume if nobody knew about them? From their statement it looks like they never received the vendor support to make something better, though, so they kept making variations of the same unit that's just kinda grown out of relevance.

VAMike

@stewart this wasn't the market, it's too small to support a company. their target was larger commercial/oem customers, who generally don't hang out in forums or buy stuff based on amazon ads. e.g., at one point netgate was a customer. they dabbled in small volume consumer sales, but it was never their focus.

kevindd992002

@dugeem are the tweaks in the link I provided before still applicable for 2.7?

https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/

stephenw10

The link suggests no tweaks are required in 2.5 or 2.6 and I would expect the same in 2.7. Though I don't have an APU2 to test with.