PC Engines apu2 experiences
-
Weird. Has this issue been around for a while or has it appeared with 2.4.5?
Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)
-
@dugeem said in PC Engines apu2 experiences:
@fireodo
Weird. Has this issue been around for a while or has it appeared with 2.4.5?No. I had this issue only after adding hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
AND then disable TSO & LRO in the advanced settings.Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)
The only thing that I found was: "ppp: can't lock /var/run/pppoe_wan.pid after 30 attempts"
As I said somewhere up in the thread without those "tunings" everything work rocksolid.Have a nice Sunday,
fireodo -
@kevindd992002 said in PC Engines apu2 experiences:
By the way, why can't ICMP redirect be enabled at the same time as tryforward path?
This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.
Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?
Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5
-
@Qinn said in PC Engines apu2 experiences:
2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
As hinted above FreeBSD has for years incorporated a fast routing path which speeds packet forwarding. Prior to pfSense 2.3 it was the ip_fastforward path - which then was changed to ip_tryforward with pfSense 2.3 (FreeBSD 10.3). Now that a bug in ICMP Redirect handling in the fast path has been fixed it is necessary to change the specified ICMP Redirects sysctls to enable the fast routing path.
3.Add hw.igb.rx_process_limit =-1 to /boot/loader.conf.local
Sysctl hw.igb.rx_process_limit sets the limit number of received packets which can consecutively handled by igb driver. Default is 100. On multicore systems this can usually be set to unlimited (value -1).
Personally I've had no problems but as per recent posts from @fireodo this has caused issues with PPPoE.
-
@dugeem said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
By the way, why can't ICMP redirect be enabled at the same time as tryforward path?
This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.
Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?
Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5
Ok, so if I understand that right, pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled, correct? You need to enable them explicitly in the system tunables if you want to use them.
-
@kevindd992002 said in PC Engines apu2 experiences:
pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled
The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).
Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343
-
@dugeem Kudos so tweak 2 only favors, if you use ipv6?
-
@dugeem said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled
The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).
Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343
Ok and for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct?
-
If you only use IPv4 then you just need to set net.inet.ip.redirect=0
If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0
-
@kevindd992002 said in PC Engines apu2 experiences:
for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct
Yes. ICMP Redirects are generally only used where there are two or more routers accessible on an IP network.
-
@dugeem said in PC Engines apu2 experiences:
If you only use IPv4 then you just need to set net.inet.ip.redirect=0
If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0
Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?
btw do these tweaks need a reboot?
-
@Qinn said in PC Engines apu2 experiences:
@dugeem said in PC Engines apu2 experiences:
If you only use IPv4 then you just need to set net.inet.ip.redirect=0
If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0
Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?
btw do these tweaks need a reboot?
You need to create the file if it isn't there (by default it isn't).
-
@kevindd992002 Done it!
Cheers Qinn
-
I've changed my settings as @dugeem mentioned and I see a somewhat lower CPU load and my network stays usable even when my Tor relay is doing 300mbit in/out.
Next step is to see if I can hookup my fiber connection (PPPoE) directly to pfSense and maintain those speeds.
-
@stefanl as I see you use a UPS, may I give you some advise?
you are using ufs as a filesystem, I would move over to ZFS, as It will give you much more security. ZFS requires a system with ECC memory, otherwise you're still not 100% safe and the APU2C4 has ECC memory. So this, using ZFS, will guard you from filesystems damage etc. when a power surge comes along.
Cheers Qinn
-
...and I would install Service_Watchdog, that way all you services are always up and running.
-
For the apu2c4, would IPSec be faster than OpenVPN for site-to-site VPN? I have OpenVPN configured now but I'm not sure if it's wise to switch to IPSec as I read that it's almost always faster than OpenVPN.
-
@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client
iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears. -
@cysiacom said in PC Engines apu2 experiences:
@kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client
iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .OpenVPN: TLS Auth - 128-GCM- DH2
Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.I don't understand though, why is there a "provider" involved between the IPSec tunnel that poses these crypto settings requirement? I thought it's just like OpenVPN where one end would be the VPN server and the other end the client?
-
@Qinn Thanks for the tips. I don't think I'll reinstall pfSense to a single disk ZFS. My NAS haves it, but it haves 12x 4TB disks.
Regarding watchdog, it's enabled and working, but somehow it isn't showing on the dashboard nor in status --> services.
