PC Engines apu2 experiences

kevindd992002

@dugeem said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled

The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).

Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343

Ok and for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct?

dugeem

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

dugeem

@kevindd992002 said in PC Engines apu2 experiences:

for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct

Yes. ICMP Redirects are generally only used where there are two or more routers accessible on an IP network.

Qinn

@dugeem said in PC Engines apu2 experiences:

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

btw do these tweaks need a reboot?

kevindd992002

@Qinn said in PC Engines apu2 experiences:

@dugeem said in PC Engines apu2 experiences:

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

btw do these tweaks need a reboot?

You need to create the file if it isn't there (by default it isn't).

Qinn

@kevindd992002 Done it!

Cheers Qinn

stefanl

I've changed my settings as @dugeem mentioned and I see a somewhat lower CPU load and my network stays usable even when my Tor relay is doing 300mbit in/out.

Next step is to see if I can hookup my fiber connection (PPPoE) directly to pfSense and maintain those speeds.

Qinn

@stefanl as I see you use a UPS, may I give you some advise?

you are using ufs as a filesystem, I would move over to ZFS, as It will give you much more security. ZFS requires a system with ECC memory, otherwise you're still not 100% safe and the APU2C4 has ECC memory. So this, using ZFS, will guard you from filesystems damage etc. when a power surge comes along.

Cheers Qinn

Qinn

...and I would install Service_Watchdog, that way all you services are always up and running.

kevindd992002

For the apu2c4, would IPSec be faster than OpenVPN for site-to-site VPN? I have OpenVPN configured now but I'm not sure if it's wise to switch to IPSec as I read that it's almost always faster than OpenVPN.

cysiacom

@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

kevindd992002

@cysiacom said in PC Engines apu2 experiences:

@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

I don't understand though, why is there a "provider" involved between the IPSec tunnel that poses these crypto settings requirement? I thought it's just like OpenVPN where one end would be the VPN server and the other end the client?

stefanl

@Qinn Thanks for the tips. I don't think I'll reinstall pfSense to a single disk ZFS. My NAS haves it, but it haves 12x 4TB disks.

Regarding watchdog, it's enabled and working, but somehow it isn't showing on the dashboard nor in status --> services.

cysiacom

@kevindd992002 I'm afraid I didn't explain myself clearly.
Sorry for that.

One customer needed to access to their provider servers.
The access is granted via Site2Site IPsec VPN with very low crypto requirements and also quite unusual IPs (they use "fake" public IPs inside the tunnel).
We had some trouble getting Phase2 working for that customer and provider so we planned some lab tests.
There's no need for provider at all in general but just for this case, our customer, in particular.

When doing local tests we did realize the speed change on IPSec tunnels compared to OpenVPN so we did some other test for our own purposes.
Again.

There's no need for external or any provider. It was just only on our specific customer needs.

kevindd992002

@cysiacom said in PC Engines apu2 experiences:

@kevindd992002 I'm afraid I didn't explain myself clearly.
Sorry for that.

One customer needed to access to their provider servers.
The access is granted via Site2Site IPsec VPN with very low crypto requirements and also quite unusual IPs (they use "fake" public IPs inside the tunnel).
We had some trouble getting Phase2 working for that customer and provider so we planned some lab tests.
There's no need for provider at all in general but just for this case, our customer, in particular.

When doing local tests we did realize the speed change on IPSec tunnels compared to OpenVPN so we did some other test for our own purposes.
Again.

There's no need for external or any provider. It was just only on our specific customer needs.

No worries, I see what you mean now. Would you happen to have any best practice guide in setting up an IPSec tunnel in pfsense, at least for a home setup with "relaxed but secure" crypto requirements?

As for Remote Access VPN, is OpenVPN still the way to go? There's nothing stopping me of creating an IPSec s2s link and an OpenVPN remote access VPN gateway, right?

Qinn

@stefanl said in PC Engines apu2 experiences:

@Qinn Thanks for the tips. I don't think I'll reinstall pfSense to a single disk ZFS. My NAS haves it, but it haves 12x 4TB disks.

Regarding watchdog, it's enabled and working, but somehow it isn't showing on the dashboard nor in status --> services.

I thought and have the same, this service isn't present in the services status, but if you stop 1 service, you will see it will come up again (if you have an mail address setup for notices, you should also receive a mail. Maybe read https://forum.netgate.com/topic/59761/new-package-service-watchdog/9

fireodo

Bios Update from 4.11.0.4 to 4.11.0.6 on APU2C0, until now without any issues :-)

Regards,
fireodo

dugeem

Yes the PCIe power management feature is now disabled by default (ie maximum performance) and ACPI change reverted (ie sysctl dev.cpu works).

Detailed APU* Coreboot v4.11.0.6 release notes:

Rebased with official coreboot repository commit d6f7ec5.
Updated sortbootorder to v4.6.18 bringing the PCI Express power management features runtime option. For details refer to sortbootorder documentation. When PCI Express power management features features are enabled, the network controllers (NICs and WIFi cards) may have reduced performance at the cost of reduced power consumption. By default this option will be disabled to not impact the network performance.
Reverted changes to ACPI CPU definitions causing BSD systems to not probe CPU frequency driver. The ACPI compliance of current BSD systems is not up to date, the situation should improve when the distribution will start to use FreeBSD 12.x, which works well with most recent rules of defining processors in ACPI.
Reverted changes with PCIe reset logic causing mPCIe2 slot connected modules to not appear in OS. The change did more harm than good. We are working to improve the PCIe modules detection in firmware, which is dependent on the AGESA.
Added IOMMU IVRS generation expanded with IVHD type 11h for newer Xen. This change should allow newer Xen images to utilize more IOMMU features.
Fixed memtest hang on apu1.
Fixed TPM2 detection on FreeBSD 12.1. Since FreeBSD 12.1 the TPM2 support is available along with FreeBSD ports offering TPM2 tools. We will provide documentation how to install and utilize those tools on FreeBSD systems soon.
Fixed a problem where SD 3.0 mode could not be disabled.

Applied to apu2c4 test system. All good so far.

kevindd992002

@dugeem said in PC Engines apu2 experiences:

Yes the PCIe power management feature is now disabled by default (ie maximum performance) and ACPI change reverted (ie sysctl dev.cpu works).

Detailed APU* Coreboot v4.11.0.6 release notes:

Rebased with official coreboot repository commit d6f7ec5.

Updated sortbootorder to v4.6.18 bringing the PCI Express power management features runtime option. For details refer to sortbootorder documentation. When PCI Express power management features features are enabled, the network controllers (NICs and WIFi cards) may have reduced performance at the cost of reduced power consumption. By default this option will be disabled to not impact the network performance.

Reverted changes to ACPI CPU definitions causing BSD systems to not probe CPU frequency driver. The ACPI compliance of current BSD systems is not up to date, the situation should improve when the distribution will start to use FreeBSD 12.x, which works well with most recent rules of defining processors in ACPI.

Reverted changes with PCIe reset logic causing mPCIe2 slot connected modules to not appear in OS. The change did more harm than good. We are working to improve the PCIe modules detection in firmware, which is dependent on the AGESA.

Added IOMMU IVRS generation expanded with IVHD type 11h for newer Xen. This change should allow newer Xen images to utilize more IOMMU features.

Fixed memtest hang on apu1.

Fixed TPM2 detection on FreeBSD 12.1. Since FreeBSD 12.1 the TPM2 support is available along with FreeBSD ports offering TPM2 tools. We will provide documentation how to install and utilize those tools on FreeBSD systems soon.

Fixed a problem where SD 3.0 mode could not be disabled.

Applied to apu2c4 test system. All good so far.

Do you happen to know the what the recommendations are for the APU2C4 v4.11.0.6 BIOS settings to achieve max performance?

dugeem

@kevindd992002 said in PC Engines apu2 experiences:

Do you happen to know the what the recommendations are for the APU2C4 v4.11.0.6 BIOS settings to achieve max performance?

Defaults are fine. Only two that matter are:

Core Performance Boost (CPB) enabled (default)
PCIe Power Management disabled (now default)

Cheers