PC Engines apu2 experiences

dugeem

Yes the PCIe power management feature is now disabled by default (ie maximum performance) and ACPI change reverted (ie sysctl dev.cpu works).

Detailed APU* Coreboot v4.11.0.6 release notes:

Rebased with official coreboot repository commit d6f7ec5.
Updated sortbootorder to v4.6.18 bringing the PCI Express power management features runtime option. For details refer to sortbootorder documentation. When PCI Express power management features features are enabled, the network controllers (NICs and WIFi cards) may have reduced performance at the cost of reduced power consumption. By default this option will be disabled to not impact the network performance.
Reverted changes to ACPI CPU definitions causing BSD systems to not probe CPU frequency driver. The ACPI compliance of current BSD systems is not up to date, the situation should improve when the distribution will start to use FreeBSD 12.x, which works well with most recent rules of defining processors in ACPI.
Reverted changes with PCIe reset logic causing mPCIe2 slot connected modules to not appear in OS. The change did more harm than good. We are working to improve the PCIe modules detection in firmware, which is dependent on the AGESA.
Added IOMMU IVRS generation expanded with IVHD type 11h for newer Xen. This change should allow newer Xen images to utilize more IOMMU features.
Fixed memtest hang on apu1.
Fixed TPM2 detection on FreeBSD 12.1. Since FreeBSD 12.1 the TPM2 support is available along with FreeBSD ports offering TPM2 tools. We will provide documentation how to install and utilize those tools on FreeBSD systems soon.
Fixed a problem where SD 3.0 mode could not be disabled.

Applied to apu2c4 test system. All good so far.

kevindd992002

@dugeem said in PC Engines apu2 experiences:

Yes the PCIe power management feature is now disabled by default (ie maximum performance) and ACPI change reverted (ie sysctl dev.cpu works).

Detailed APU* Coreboot v4.11.0.6 release notes:

Rebased with official coreboot repository commit d6f7ec5.

Updated sortbootorder to v4.6.18 bringing the PCI Express power management features runtime option. For details refer to sortbootorder documentation. When PCI Express power management features features are enabled, the network controllers (NICs and WIFi cards) may have reduced performance at the cost of reduced power consumption. By default this option will be disabled to not impact the network performance.

Reverted changes to ACPI CPU definitions causing BSD systems to not probe CPU frequency driver. The ACPI compliance of current BSD systems is not up to date, the situation should improve when the distribution will start to use FreeBSD 12.x, which works well with most recent rules of defining processors in ACPI.

Reverted changes with PCIe reset logic causing mPCIe2 slot connected modules to not appear in OS. The change did more harm than good. We are working to improve the PCIe modules detection in firmware, which is dependent on the AGESA.

Added IOMMU IVRS generation expanded with IVHD type 11h for newer Xen. This change should allow newer Xen images to utilize more IOMMU features.

Fixed memtest hang on apu1.

Fixed TPM2 detection on FreeBSD 12.1. Since FreeBSD 12.1 the TPM2 support is available along with FreeBSD ports offering TPM2 tools. We will provide documentation how to install and utilize those tools on FreeBSD systems soon.

Fixed a problem where SD 3.0 mode could not be disabled.

Applied to apu2c4 test system. All good so far.

Do you happen to know the what the recommendations are for the APU2C4 v4.11.0.6 BIOS settings to achieve max performance?

dugeem

@kevindd992002 said in PC Engines apu2 experiences:

Do you happen to know the what the recommendations are for the APU2C4 v4.11.0.6 BIOS settings to achieve max performance?

Defaults are fine. Only two that matter are:

Core Performance Boost (CPB) enabled (default)
PCIe Power Management disabled (now default)

Cheers

daemonix

@dugeem good to know. Im planning to update too.

kevindd992002

@dugeem said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

Do you happen to know the what the recommendations are for the APU2C4 v4.11.0.6 BIOS settings to achieve max performance?

Defaults are fine. Only two that matter are:

Core Performance Boost (CPB) enabled (default)

PCIe Power Management disabled (now default)

Cheers

Got it. I know CPB enabled is the default but was disabled in the older BIOS'es where it was initially introduced. Just for the heck of it, is there a way to know if CPB is enabled aside from checking it during runtime? I currently don't have physical access to the pfsense boxes.

Qinn

@kevindd992002 I only know

dmidecode

on the command line, but sadly it doesn't show the CPB

kevindd992002

@Qinn said in PC Engines apu2 experiences:

/boot/loader.conf.local

@cysiacom and others

Not sure why but my OpenVPN remote access server only 55-60Mbps speed when I'm trying to saturate the link. I'm not sure if my test is right but I connect my local laptop (same network as the OpenVPN remote access server) to the OpenVPN server and download a game in steam. My Internet connection is 300Mbps symmetrical speed, so I'm expecting much higher speed. As soon as I disconnect the VPN connection of the laptop, the download speed soars up to saturate the 300Mbps link.

Here are my OpenVPN settings:

When my laptop is NOT connected to the VPN server, here are the traffic graphs:

As soon as I connect it to the VPN server:

See the sudden drop in speed? When I disconnect from the VPN server, the speed goes back up to the saturation point. Any ideas?

DaddyGo

We use more than 20 APU4 boards in our system with OpenVPN.
The value will not be higher! (cca. 55 Mbps)
This is what the APU board can screw out of of itself, since OpenVPN uses only 1 CPU core and in this case is1 .0 GHz or 1.4 GHz (1.4 if you do the tuning, but that only applies to one core).
We've been experimenting for a long time to achieve higher speeds, but low CPU clock can do just that with OpenVPN.

Under the same conditions, a Supermicro M11SDV-4C-LN4F-based pfSense 310 to 340 Mbps can be accessed in the same location with the same installation.
This is proof of the above.

BTW, your setting is completely correct!
Regarding OpenVPN, the CPU clock is your best friend.

kevindd992002

That's what I thought. Does that mean that the CPU usage in the pfsense dashboard means overall CPU usage and not just single core? I just don't see it peaking at 100% while running the test. I tried it again with speedtest and got it to be higher:

https://www.speedtest.net/result/9415734802

I don't understand how it got higher though. The battle.net game download I had earlier was saturating the link without the VPN connection so I was expecting it to be more or less the same as the speedtest result.

Does that mean that it's better to do IPsec road warrior server then?

DaddyGo

@kevindd992002 said in PC Engines apu2 experiences:

https://www.speedtest.net

Exactly! if you want better values then IPsec. (multithreading)
Since we use ExpressVPN and our own OpenVPN tunnels, we can't switch to IPsec, we just accepted the fact that, these motherboards can do just that.
The measurements depend on a lot of things, the current load and what else is running on the APU under pfSense, such as Squid, pfBlocketNG, Snort / Suricata.

The graphs show aggregate values and can be suggestive.
Measurements can also be performed in several ways, such as https://fast.com/ and https://speedof.me/ and https://www.meter.net/ping-test/.
I personally don't like the https://www.speedtest.net

stephenw10

Run top -aSH at the command line while testing to see how the CPU cores are loaded.

You have NCP enabled and you have AES-CBC and AES-GCM set as NCP ciphers. Which is it actually connecting with?

I would disable NCP there to force it to use AES-128-GCM which you have selected. I would expect that to be fastest.

Steve

DaddyGo

NCP is a good idea Steve, but it won’t help much.
Compare these two settings with one NCP and the other W/O NCP.
(We work with each provider and have experience with these settings.)

or

There is no significant difference between the obtained velocities.
We have been using APU boards for 5 years and these are long-term experiences.

kevindd992002

@DaddyGo said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

https://www.speedtest.net

Exactly! if you want better values then IPsec. (multithreading)
Since we use ExpressVPN and our own OpenVPN tunnels, we can't switch to IPsec, we just accepted the fact that, these motherboards can do just that.
The measurements depend on a lot of things, the current load and what else is running on the APU under pfSense, such as Squid, pfBlocketNG, Snort / Suricata.

The graphs show aggregate values and can be suggestive.
Measurements can also be performed in several ways, such as https://fast.com/ and https://speedof.me/ and https://www.meter.net/ping-test/.
I personally don't like the https://www.speedtest.net

I see. I have to get my head around using IPsec then. I tried establishing an Ipsec connection as described here but failed.

Yes, I understand that they show aggregate values but I only have one test client connected to the OpenVPN server when I tested that. Here's what I see with the other test sites:

fast.com = 82 Mbps
speedof.me = 60 Mbps
meter.net = 68 Mbps

For me, usually fast.com and speedtest.net almost always saturates my link.

@stephenw10 said in PC Engines apu2 experiences:

Run top -aSH at the command line while testing to see how the CPU cores are loaded.

You have NCP enabled and you have AES-CBC and AES-GCM set as NCP ciphers. Which is it actually connecting with?

I would disable NCP there to force it to use AES-128-GCM which you have selected. I would expect that to be fastest.

Steve

Ok, yeah I see that the process is only using one CPU while running the test.

The connection is using the AES-128-GCM as expected but disabling NCP to force AES-128-GCM makes sense.

kevindd992002

@DaddyGo said in PC Engines apu2 experiences:

NCP is a good idea Steve, but it won’t help much.
Compare these two settings with one NCP and the other W/O NCP.
(We work with each provider and have experience with these settings.)

or

There is no significant difference between the obtained velocities.
We have been using APU boards for 5 years and these are long-term experiences.

I don't think he's saying that NCP helps. If anything, he's suggesting to disable NCP on my settings to force the AES-128-GCM cipher.

DaddyGo

Of the above two providers, it uses one CBC and the other GCM, due to the finite performance of the APU board, no significant difference is seen.

I know he doesn't say it helps ;-) (NCP)

IPsec requires a little more care to set up, there are several good descriptions as I have seen in the forum in the past.

What I can tell you for sure is that I know APU boards very well, and we love them very much.
Only as long as the Chihuahua is a good lap dog, he is a very bad Caucasian bear killer. :-)

kevindd992002

@DaddyGo said in PC Engines apu2 experiences:

Of the above two providers, it uses one CBC and the other GCM, due to the finite performance of the APU board, no significant difference is seen.

I know he doesn't say it helps ;-) (NCP)

IPsec requires a little more care to set up, there are several good descriptions as I have seen in the forum in the past.

What I can tell you for sure is that I know APU boards very well, and we love them very much.
Only as long as the Chihuahua is a good lap dog, he is a very bad Caucasian bear killer. :-)

My bad then :)

Yeah, there's too many variables with IPsec and I hate that FreeBSD doesn't support reply-to's but hey if it's faster then I'm all for making it work. This is just for home user between two sites anyway.

DaddyGo

Yeeeppp, then they will....do
The guys at Netgate love IPsec (let’s say I understand), but they also support OpenVPN very well.
We’ll see what the future holds, just thinking about TNSR and IPsec, hmmmm

stephenw10

Indeed. Enabling NCP merely allows the two ends to negotiate a cipher and even then they both have to have ciphers set that match.
I don't have an APU2 to test with but IIRC the updated CPU is aes-ni capable. That should be measurably faster using AES-GCM over AES-CBC+SHA1/256.

Alternatively if the restriction is at the server end it should use less CPU to pass the same bandwidth.

Steve

DaddyGo

@stephenw10

Yes, you remember correctly MOBO has aes-ni ability, based on AMD Embedded G series GX-412TC CPU,
low power consumption (12W) like SOHO router category, 4 CPU core (1400 Core Performance Boost (CPB) /1000/1000/1000), 4GB DDR3-1333 with ECC and 4x Intel i211 characterizes the device (APU4).
In light of the above, the MOBO can't do more with OpenVPN either, unfortunately. (cca. 50 - 60 Mbps)

We tried it with a completely clean pfSense installation, only an OpenVPN connection was installed and we used an ISP 1000/1000 with a business subscription with a fixed IP without filters on FTTB.

Absolutely true:
Alternatively if the restriction is at the server end it should use less CPU to pass the same bandwidth.

In the tests mentioned, our own OpenVPN server was in a data center rented in BIX (4x10Gig) and the APU was at the end of the FTTB (1Gig).

This should not scare anyone from using APU boards, which is a great good little tool for external colleagues, smaller sites endpoints.

saltandpepper

Hi

Currently im running in APU2. Till 2 weeks ago I was able to get 1gibt trough my Box. 1Gbit is on the edge but it worked. Around 2 weeks ago I startet to play with traffic shapping. Unfortunatly since then I was not able to revert back to a setup that gets nearly close to my previous performance.

To check if the issue us on my side or my ISP i changed the router, which gave me my Gbit back.

Since im quite sertain that the issue is in my network i startet to test with iperf. No matter how much connection i open i only get around 400mbit trough. If only one connection is, 20 or 100. Around 400 - 420mbit is the limit. Previous i got 940mbit as expected with iperf. I already startet to test my switches, but I get 940mbit to the last switch.
After i tested all my equitment, I reinstalled Pfsense and did my confing manualy. Still 400mbit, in up and download. The settings that i use are the same as mentioned by dugeem.

@dugeem said in PC Engines apu2 experiences:

My current APU2 performance tweak summary:

Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)

Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local

There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more

~~The only thing that i have not done yet is a bios rollback to 4.11.04~~
Still 400mbit

Maybe someone has an idea whats currently is wrong with my setup.

Cheers