PC Engines apu2 experiences

Qinn · Apr 25, 2020, 1:08 PM

@kevindd992002 said in PC Engines apu2 experiences:

@Qinn said in PC Engines apu2 experiences:

@dugeem Kudos for the tweaks, could you explain what each of these tweak will do/accomplish on a APU2C4?

Btw my speeds here are not high, as I am mandatory of using aDSL which has a max of 10mb down and as it is combined with 4G a total of 60 down and a total of 15 up

Cheers Qinn

I thought he already did explain what the tweaks do on an APU2C4? It's in his post above.

I know what number on does, but the other 2 I don't and I don't seem to read what these to do

2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

3.Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local

dugeem · Apr 26, 2020, 7:30 AM

@fireodo

Weird. Has this issue been around for a while or has it appeared with 2.4.5?

Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)

fireodo · Apr 26, 2020, 7:42 AM

@dugeem said in PC Engines apu2 experiences:

@fireodo
Weird. Has this issue been around for a while or has it appeared with 2.4.5?

No. I had this issue only after adding hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
AND then disable TSO & LRO in the advanced settings.

Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)

The only thing that I found was: "ppp: can't lock /var/run/pppoe_wan.pid after 30 attempts"
As I said somewhere up in the thread without those "tunings" everything work rocksolid.

Have a nice Sunday,
fireodo

dugeem · Apr 26, 2020, 7:43 AM

@kevindd992002 said in PC Engines apu2 experiences:

By the way, why can't ICMP redirect be enabled at the same time as tryforward path?

This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.

Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?

Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5

dugeem · Apr 26, 2020, 7:59 AM

@Qinn said in PC Engines apu2 experiences:

2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

As hinted above FreeBSD has for years incorporated a fast routing path which speeds packet forwarding. Prior to pfSense 2.3 it was the ip_fastforward path - which then was changed to ip_tryforward with pfSense 2.3 (FreeBSD 10.3). Now that a bug in ICMP Redirect handling in the fast path has been fixed it is necessary to change the specified ICMP Redirects sysctls to enable the fast routing path.

3.Add hw.igb.rx_process_limit =-1 to /boot/loader.conf.local

Sysctl hw.igb.rx_process_limit sets the limit number of received packets which can consecutively handled by igb driver. Default is 100. On multicore systems this can usually be set to unlimited (value -1).

Personally I've had no problems but as per recent posts from @fireodo this has caused issues with PPPoE.

kevindd992002 · Apr 26, 2020, 4:14 PM

@dugeem said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

By the way, why can't ICMP redirect be enabled at the same time as tryforward path?

This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.

Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?

Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5

Ok, so if I understand that right, pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled, correct? You need to enable them explicitly in the system tunables if you want to use them.

dugeem · Apr 27, 2020, 9:11 AM

@kevindd992002 said in PC Engines apu2 experiences:

pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled

The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).

Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343

Qinn · Apr 27, 2020, 8:50 AM

@dugeem Kudos so tweak 2 only favors, if you use ipv6?

kevindd992002 · Apr 27, 2020, 9:11 AM

@dugeem said in PC Engines apu2 experiences:

@kevindd992002 said in PC Engines apu2 experiences:

pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled

The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).

Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343

Ok and for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct?

dugeem · Apr 27, 2020, 11:50 AM

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

dugeem · Apr 27, 2020, 11:51 AM

@kevindd992002 said in PC Engines apu2 experiences:

for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct

Yes. ICMP Redirects are generally only used where there are two or more routers accessible on an IP network.

Qinn · Apr 27, 2020, 12:43 PM

@dugeem said in PC Engines apu2 experiences:

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

btw do these tweaks need a reboot?

kevindd992002 · Apr 27, 2020, 1:10 PM

@Qinn said in PC Engines apu2 experiences:

@dugeem said in PC Engines apu2 experiences:

@Qinn

If you only use IPv4 then you just need to set net.inet.ip.redirect=0

If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

btw do these tweaks need a reboot?

You need to create the file if it isn't there (by default it isn't).

Qinn · Apr 27, 2020, 1:10 PM

@kevindd992002 Done it!

Cheers Qinn

stefanl · Apr 27, 2020, 3:10 PM

I've changed my settings as @dugeem mentioned and I see a somewhat lower CPU load and my network stays usable even when my Tor relay is doing 300mbit in/out.

Next step is to see if I can hookup my fiber connection (PPPoE) directly to pfSense and maintain those speeds.

Qinn · Apr 27, 2020, 3:52 PM

@stefanl as I see you use a UPS, may I give you some advise?

you are using ufs as a filesystem, I would move over to ZFS, as It will give you much more security. ZFS requires a system with ECC memory, otherwise you're still not 100% safe and the APU2C4 has ECC memory. So this, using ZFS, will guard you from filesystems damage etc. when a power surge comes along.

Cheers Qinn

Qinn · Apr 28, 2020, 6:30 AM

...and I would install Service_Watchdog, that way all you services are always up and running.

kevindd992002 · Apr 28, 2020, 7:08 AM

For the apu2c4, would IPSec be faster than OpenVPN for site-to-site VPN? I have OpenVPN configured now but I'm not sure if it's wise to switch to IPSec as I read that it's almost always faster than OpenVPN.

cysiacom · Apr 28, 2020, 8:59 AM

@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

kevindd992002 · Apr 28, 2020, 2:18 PM

@cysiacom said in PC Engines apu2 experiences:

@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

I don't understand though, why is there a "provider" involved between the IPSec tunnel that poses these crypto settings requirement? I thought it's just like OpenVPN where one end would be the VPN server and the other end the client?