PC Engines apu2 experiences
-
Bit of a long answer ...
pfSense 2.3 (released back in 2016) was based on FreeBSD 10.3 ... which introduced a new optimised IP routing path feature known as ip_tryforward() which replaced the older ip_fastforward(). (Refer to OS section in https://docs.netgate.com/pfsense/en/latest/releases/2-3-new-features-and-changes.html). The tryforward path has been in use ever since.
However back in August 2017 a bug noting ICMP Redirects not working was raised upstream in FreeBSD (refer https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221137). This bug was finally fixed in around August 2018 in FreeBSD head (now FreeBSD 12) and was also MFCed back to 11-STABLE. Note that the patch was not applied to FreeBSD 11.1 (pfSense 2.4-2.4.3) or 11.2 (pfSense 2.4.4).
pfSense 2.4.5 updated to FreeBSD 11.3-STABLE and thus now incorporates the ICMP Redirect fix. The pfSense default for ICMP Redirects is enabled - which therefore disables use of the tryforward path.
I've raised this in Redmine (refer https://redmine.pfsense.org/issues/10465). JimP has indicated they'll consider adding a pfSense GUI option to simplify disabling ICMP Redirects. No timeframe yet.
For now the workaround is simple enough - and for most networks there is no concern for disabling ICMP Redirects (only really required for networks where there are multiple routers directly accessible from end hosts - refer https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol#Redirect)
-
-
Unfortunately some of the advice in the link is incorrect.
Firstly TSO & LRO should always be disabled on routers. Netgate recommend this (hence pfSense defaults) as do others. BSDRP have even tested this and found routing performance drop negligible from enabling TSO & LRO (see link below).
In terms of loader.conf.local suggestions:
- hw.igb.rx_process_limit=-1 is a standard tweak for Intel igb NICs. Performance boost on APU2 though is only ~1%.
- hw.igb.tx_process_limit already defaults to -1 so no need to change this.
- hw.igb.num_queues=1 is not for APU2 as stated. Default 0 allows driver to allocate maximum queues across CPU cores (i210AT has 4 queues; i211AT has 2 queues)
- kern.ipc.nmbclusters=1000000 is unnecessary - default on APU2 with 4GB RAM is ~250000 (mbuf is 2kB - so this represents maximum 12% of RAM). Possibly for high bandwidth routers 500000 mbufs would be prudent. However use the command
netstat -mto verify mbuf use prior to changing. - net.pf.states_hashsize=2097152 is ridculous for an APU2. If you need to be tweaking this then you'll likely need better hardware.
- hw.igb.rxd=4096 & hw.igb.txd=4096. Increasing NIC descriptors on APU2 will actually decrease performance by 20%. And likely worsens buffer bloat. Default of 1024 is fine.
- net.inet.tcp.* sysctl tuning is for end clients (ie not routers).
Reference performance data for some of the above: https://bsdrp.net/documentation/technical_docs/performance#nic_drivers_tuning
Even longer version: https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf
The only caveat is that these BSDRP performance numbers were compiled in 2018 before the AMD CPB was enabled in APU2 BIOS - so performance should now exceed this.
My current APU2 performance tweak summary:
- Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)
- Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
- Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more
-
@dugeem said in PC Engines apu2 experiences:
My current APU2 performance tweak summary:
- Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)
- Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
- Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more
May I ask you what kind of Internet Access do you have? Is it PPPOE or something else?
Because here (I have Internet Access via PPPOE) if I activate hw.igb.rx_process_limit=-1 in /boot/loader.conf.local and then disable TSO & LRO in the advanced settings the PPPOE internet connection cannot be established. (PPPOE is bound on igb1).Thanks,
fireodo -
My primary internet is a 100Mb/s HFC service which terminates on Ethernet VLAN.
Your problem is interesting. Last year I had a secondary PPPoE service as well - and had no issues.
If you revert to pfSense defaults (TSO & LRO disabled - comment out anything in loader.conf.local etc) does the PPPoE connection work?
-
@dugeem said in PC Engines apu2 experiences:
@fireodo
If you revert to pfSense defaults (TSO & LRO disabled - comment out anything in loader.conf.local etc) does the PPPoE connection work?Yes!
-
@dugeem said in PC Engines apu2 experiences:
Unfortunately some of the advice in the link is incorrect.
Firstly TSO & LRO should always be disabled on routers. Netgate recommend this (hence pfSense defaults) as do others. BSDRP have even tested this and found routing performance drop negligible from enabling TSO & LRO (see link below).
In terms of loader.conf.local suggestions:
- hw.igb.rx_process_limit=-1 is a standard tweak for Intel igb NICs. Performance boost on APU2 though is only ~1%.
- hw.igb.tx_process_limit already defaults to -1 so no need to change this.
- hw.igb.num_queues=1 is not for APU2 as stated. Default 0 allows driver to allocate maximum queues across CPU cores (i210AT has 4 queues; i211AT has 2 queues)
- kern.ipc.nmbclusters=1000000 is unnecessary - default on APU2 with 4GB RAM is ~250000 (mbuf is 2kB - so this represents maximum 12% of RAM). Possibly for high bandwidth routers 500000 mbufs would be prudent. However use the command
netstat -mto verify mbuf use prior to changing. - net.pf.states_hashsize=2097152 is ridculous for an APU2. If you need to be tweaking this then you'll likely need better hardware.
- hw.igb.rxd=4096 & hw.igb.txd=4096. Increasing NIC descriptors on APU2 will actually decrease performance by 20%. And likely worsens buffer bloat. Default of 1024 is fine.
- net.inet.tcp.* sysctl tuning is for end clients (ie not routers).
Reference performance data for some of the above: https://bsdrp.net/documentation/technical_docs/performance#nic_drivers_tuning
Even longer version: https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf
The only caveat is that these BSDRP performance numbers were compiled in 2018 before the AMD CPB was enabled in APU2 BIOS - so performance should now exceed this.
My current APU2 performance tweak summary:
- Upgrade BIOS to enable CPB (mainline v4.9.0.2 or later, legacy v4.0.25 or later)
- Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
- Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
There may well be other tweaks but for our power efficient APU2 routers these tweaks should serve most well. And when my home internet evolves to 500Mb/s I'll worry some more
Thanks for the explanation again. I just applied these tweaks to my boxes. I just have a 300Mbps down/up Internet connection anyway but just want to optimize everything for my home networks.
-
@dugeem said in PC Engines apu2 experiences:
Bit of a long answer ...
pfSense 2.3 (released back in 2016) was based on FreeBSD 10.3 ... which introduced a new optimised IP routing path feature known as ip_tryforward() which replaced the older ip_fastforward(). (Refer to OS section in https://docs.netgate.com/pfsense/en/latest/releases/2-3-new-features-and-changes.html). The tryforward path has been in use ever since.
However back in August 2017 a bug noting ICMP Redirects not working was raised upstream in FreeBSD (refer https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221137). This bug was finally fixed in around August 2018 in FreeBSD head (now FreeBSD 12) and was also MFCed back to 11-STABLE. Note that the patch was not applied to FreeBSD 11.1 (pfSense 2.4-2.4.3) or 11.2 (pfSense 2.4.4).
pfSense 2.4.5 updated to FreeBSD 11.3-STABLE and thus now incorporates the ICMP Redirect fix. The pfSense default for ICMP Redirects is enabled - which therefore disables use of the tryforward path.
I've raised this in Redmine (refer https://redmine.pfsense.org/issues/10465). JimP has indicated they'll consider adding a pfSense GUI option to simplify disabling ICMP Redirects. No timeframe yet.
For now the workaround is simple enough - and for most networks there is no concern for disabling ICMP Redirects (only really required for networks where there are multiple routers directly accessible from end hosts - refer https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol#Redirect)
By the way, why can't ICMP redirect be enabled at the same time as tryforward path?
Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?
-
@dugeem Kudos for the tweaks, could you explain what each of these tweak will do/accomplish on a APU2C4?
Btw my speeds here are not high, as I am mandatory of using aDSL which has a max of 10mb down and as it is combined with 4G a total of 60 down and a total of 15 up
Cheers Qinn
-
@Qinn said in PC Engines apu2 experiences:
@dugeem Kudos for the tweaks, could you explain what each of these tweak will do/accomplish on a APU2C4?
Btw my speeds here are not high, as I am mandatory of using aDSL which has a max of 10mb down and as it is combined with 4G a total of 60 down and a total of 15 up
Cheers Qinn
I thought he already did explain what the tweaks do on an APU2C4? It's in his post above.
-
@kevindd992002 said in PC Engines apu2 experiences:
@Qinn said in PC Engines apu2 experiences:
@dugeem Kudos for the tweaks, could you explain what each of these tweak will do/accomplish on a APU2C4?
Btw my speeds here are not high, as I am mandatory of using aDSL which has a max of 10mb down and as it is combined with 4G a total of 60 down and a total of 15 up
Cheers Qinn
I thought he already did explain what the tweaks do on an APU2C4? It's in his post above.
I know what number on does, but the other 2 I don't and I don't seem to read what these to do
2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
3.Add hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
-
Weird. Has this issue been around for a while or has it appeared with 2.4.5?
Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)
-
@dugeem said in PC Engines apu2 experiences:
@fireodo
Weird. Has this issue been around for a while or has it appeared with 2.4.5?No. I had this issue only after adding hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
AND then disable TSO & LRO in the advanced settings.Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)
The only thing that I found was: "ppp: can't lock /var/run/pppoe_wan.pid after 30 attempts"
As I said somewhere up in the thread without those "tunings" everything work rocksolid.Have a nice Sunday,
fireodo -
@kevindd992002 said in PC Engines apu2 experiences:
By the way, why can't ICMP redirect be enabled at the same time as tryforward path?
This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.
Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?
Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5
-
@Qinn said in PC Engines apu2 experiences:
2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)
As hinted above FreeBSD has for years incorporated a fast routing path which speeds packet forwarding. Prior to pfSense 2.3 it was the ip_fastforward path - which then was changed to ip_tryforward with pfSense 2.3 (FreeBSD 10.3). Now that a bug in ICMP Redirect handling in the fast path has been fixed it is necessary to change the specified ICMP Redirects sysctls to enable the fast routing path.
3.Add hw.igb.rx_process_limit =-1 to /boot/loader.conf.local
Sysctl hw.igb.rx_process_limit sets the limit number of received packets which can consecutively handled by igb driver. Default is 100. On multicore systems this can usually be set to unlimited (value -1).
Personally I've had no problems but as per recent posts from @fireodo this has caused issues with PPPoE.
-
@dugeem said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
By the way, why can't ICMP redirect be enabled at the same time as tryforward path?
This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.
Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?
Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5
Ok, so if I understand that right, pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled, correct? You need to enable them explicitly in the system tunables if you want to use them.
-
@kevindd992002 said in PC Engines apu2 experiences:
pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled
The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).
Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343
-
@dugeem Kudos so tweak 2 only favors, if you use ipv6?
-
@dugeem said in PC Engines apu2 experiences:
@kevindd992002 said in PC Engines apu2 experiences:
pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled
The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).
Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343
Ok and for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct?
-
If you only use IPv4 then you just need to set net.inet.ip.redirect=0
If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0
