Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PC Engines apu2 experiences

    Scheduled Pinned Locked Moved Hardware
    711 Posts 73 Posters 777.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • fireodoF
      fireodo @dugeem
      last edited by

      @dugeem said in PC Engines apu2 experiences:

      @fireodo
      Weird. Has this issue been around for a while or has it appeared with 2.4.5?

      No. I had this issue only after adding hw.igb.rx_process_limit=-1 to /boot/loader.conf.local
      AND then disable TSO & LRO in the advanced settings.

      Any hints from PPPoE logging? Also you could try reducing interface MTU (though normally PPPoE gets it right)

      The only thing that I found was: "ppp: can't lock /var/run/pppoe_wan.pid after 30 attempts"
      As I said somewhere up in the thread without those "tunings" everything work rocksolid.

      Have a nice Sunday,
      fireodo

      Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
      SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
      pfsense 2.8.0 CE
      Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

      1 Reply Last reply Reply Quote 0
      • D
        dugeem @kevindd992002
        last edited by

        @kevindd992002 said in PC Engines apu2 experiences:

        By the way, why can't ICMP redirect be enabled at the same time as tryforward path?

        This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.

        Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?

        Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5

        K 1 Reply Last reply Reply Quote 0
        • D
          dugeem @Qinn
          last edited by

          @Qinn said in PC Engines apu2 experiences:

          2.Disable ICMP Redirects to enable tryforward routing path (under System / Advanced / System Tunables set net.inet.ip.redirect & net.inet6.ip6.redirect to 0)

          As hinted above FreeBSD has for years incorporated a fast routing path which speeds packet forwarding. Prior to pfSense 2.3 it was the ip_fastforward path - which then was changed to ip_tryforward with pfSense 2.3 (FreeBSD 10.3). Now that a bug in ICMP Redirect handling in the fast path has been fixed it is necessary to change the specified ICMP Redirects sysctls to enable the fast routing path.

          3.Add hw.igb.rx_process_limit =-1 to /boot/loader.conf.local

          Sysctl hw.igb.rx_process_limit sets the limit number of received packets which can consecutively handled by igb driver. Default is 100. On multicore systems this can usually be set to unlimited (value -1).

          Personally I've had no problems but as per recent posts from @fireodo this has caused issues with PPPoE.

          QinnQ 1 Reply Last reply Reply Quote 1
          • K
            kevindd992002 @dugeem
            last edited by

            @dugeem said in PC Engines apu2 experiences:

            @kevindd992002 said in PC Engines apu2 experiences:

            By the way, why can't ICMP redirect be enabled at the same time as tryforward path?

            This is a FreeBSD kernel restriction. I think it comes from concern that if you have to start generating ICMP packets from within the fast routing path then you can potentially overwhelm gateways with ICMP redirects.

            Also, if tryforward was used ever since does that mean ICMP redirect was disabled by default back then?

            Prior to FreeBSD 10.3 ip_tryforward() there was ip_fastforward() routing path - which had to be explicitly enabled by setting sysctl net.inet.ip.fastforwarding=1. pfSense has always had ICMP Redirects enabled - although they were not working from 2.3 to 2.4.4 due to upstream bug with FreeBSD. This is now fixed in pfSense 2.4.5

            Ok, so if I understand that right, pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled, correct? You need to enable them explicitly in the system tunables if you want to use them.

            D 1 Reply Last reply Reply Quote 0
            • D
              dugeem @kevindd992002
              last edited by

              @kevindd992002 said in PC Engines apu2 experiences:

              pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled

              The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).

              Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343

              K 1 Reply Last reply Reply Quote 0
              • QinnQ
                Qinn @dugeem
                last edited by Qinn

                @dugeem Kudos so tweak 2 only favors, if you use ipv6?

                Hardeware: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz 102 GB mSATA SSD (ZFS)
                Firmware: Latest-stable-pfSense CE (amd64)
                Packages: pfBlockerNG devel-beta (beta tester) - Avahi - Notes - Ntopng - PIMD/udpbroadcastrelay - Service Watchdog - System Patches

                D 1 Reply Last reply Reply Quote 0
                • K
                  kevindd992002 @dugeem
                  last edited by

                  @dugeem said in PC Engines apu2 experiences:

                  @kevindd992002 said in PC Engines apu2 experiences:

                  pfsense never used ip_tryforward (pfsense 2.3 and above) or ip_fastforward (pfsense versions prior to 2.3) by default because it had ICMP Redirects enabled

                  The upstream bug was actually that the FreeBSD kernel didn't check the ICMP Redirect sysctls - which meant that ip_tryforward was always used and that ICMP Redirects did not work on FreeBSD 10.3 (pfsense 2.3) thru FreeBSD 11.2 (pfSense 2.4.4). So the pfSense defaults for these sysctls had no bearing on this issue until upstream fix was implemented and pfSense switched to that release (FreeBSD 11-STABLE).

                  Here is the actual fix applied to FreeBSD kernel: https://svnweb.freebsd.org/base/stable/11/sys/netinet/ip_input.c?r1=332513&r2=338343&pathrev=338343

                  Ok and for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct?

                  D 1 Reply Last reply Reply Quote 0
                  • D
                    dugeem @Qinn
                    last edited by

                    @Qinn

                    If you only use IPv4 then you just need to set net.inet.ip.redirect=0

                    If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

                    QinnQ 1 Reply Last reply Reply Quote 1
                    • D
                      dugeem @kevindd992002
                      last edited by

                      @kevindd992002 said in PC Engines apu2 experiences:

                      for home networks where only one router/firewall (pfsense) is used anyway, ICMP Redirects aren't really being used, correct

                      Yes. ICMP Redirects are generally only used where there are two or more routers accessible on an IP network.

                      1 Reply Last reply Reply Quote 0
                      • QinnQ
                        Qinn @dugeem
                        last edited by Qinn

                        @dugeem said in PC Engines apu2 experiences:

                        @Qinn

                        If you only use IPv4 then you just need to set net.inet.ip.redirect=0

                        If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

                        Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

                        btw do these tweaks need a reboot?

                        Hardeware: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz 102 GB mSATA SSD (ZFS)
                        Firmware: Latest-stable-pfSense CE (amd64)
                        Packages: pfBlockerNG devel-beta (beta tester) - Avahi - Notes - Ntopng - PIMD/udpbroadcastrelay - Service Watchdog - System Patches

                        K 1 Reply Last reply Reply Quote 0
                        • K
                          kevindd992002 @Qinn
                          last edited by

                          @Qinn said in PC Engines apu2 experiences:

                          @dugeem said in PC Engines apu2 experiences:

                          @Qinn

                          If you only use IPv4 then you just need to set net.inet.ip.redirect=0

                          If you use both IPv4 & IPv6 then you just need to set net.inet.ip.redirect=0 & net.inet6.ip6.redirect=0

                          Done it. I looked for boot.conf.local in /boot ,but only found boot.conf? Do I overlook something?

                          btw do these tweaks need a reboot?

                          You need to create the file if it isn't there (by default it isn't).

                          QinnQ 1 Reply Last reply Reply Quote 1
                          • QinnQ
                            Qinn @kevindd992002
                            last edited by

                            @kevindd992002 Done it!

                            Cheers Qinn

                            Hardeware: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz 102 GB mSATA SSD (ZFS)
                            Firmware: Latest-stable-pfSense CE (amd64)
                            Packages: pfBlockerNG devel-beta (beta tester) - Avahi - Notes - Ntopng - PIMD/udpbroadcastrelay - Service Watchdog - System Patches

                            1 Reply Last reply Reply Quote 0
                            • S
                              stefanl
                              last edited by

                              I've changed my settings as @dugeem mentioned and I see a somewhat lower CPU load and my network stays usable even when my Tor relay is doing 300mbit in/out.

                              37cc4650-19c8-4582-a782-d6951d67a3bd-image.png

                              Next step is to see if I can hookup my fiber connection (PPPoE) directly to pfSense and maintain those speeds.

                              QinnQ 1 Reply Last reply Reply Quote 2
                              • QinnQ
                                Qinn @stefanl
                                last edited by Qinn

                                @stefanl as I see you use a UPS, may I give you some advise?

                                you are using ufs as a filesystem, I would move over to ZFS, as It will give you much more security. ZFS requires a system with ECC memory, otherwise you're still not 100% safe and the APU2C4 has ECC memory. So this, using ZFS, will guard you from filesystems damage etc. when a power surge comes along.

                                Cheers Qinn

                                Hardeware: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz 102 GB mSATA SSD (ZFS)
                                Firmware: Latest-stable-pfSense CE (amd64)
                                Packages: pfBlockerNG devel-beta (beta tester) - Avahi - Notes - Ntopng - PIMD/udpbroadcastrelay - Service Watchdog - System Patches

                                QinnQ 1 Reply Last reply Reply Quote 1
                                • QinnQ
                                  Qinn @Qinn
                                  last edited by

                                  ...and I would install Service_Watchdog, that way all you services are always up and running.

                                  Hardeware: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz 102 GB mSATA SSD (ZFS)
                                  Firmware: Latest-stable-pfSense CE (amd64)
                                  Packages: pfBlockerNG devel-beta (beta tester) - Avahi - Notes - Ntopng - PIMD/udpbroadcastrelay - Service Watchdog - System Patches

                                  S 1 Reply Last reply Reply Quote 0
                                  • K
                                    kevindd992002
                                    last edited by

                                    For the apu2c4, would IPSec be faster than OpenVPN for site-to-site VPN? I have OpenVPN configured now but I'm not sure if it's wise to switch to IPSec as I read that it's almost always faster than OpenVPN.

                                    C 1 Reply Last reply Reply Quote 0
                                    • C
                                      cysiacom @kevindd992002
                                      last edited by

                                      @kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
                                      On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

                                      PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

                                      iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
                                      After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
                                      Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

                                      OpenVPN: TLS Auth - 128-GCM- DH2
                                      Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

                                      Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
                                      I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
                                      Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

                                      K 1 Reply Last reply Reply Quote 0
                                      • K
                                        kevindd992002 @cysiacom
                                        last edited by

                                        @cysiacom said in PC Engines apu2 experiences:

                                        @kevindd992002 On my recent experience it is 2-3 times faster, with warnings on these numbers.
                                        On lab with this settings, using just a gigabit link between both APUs simulating a WAN link:

                                        PC1iPerf3 Server---LAN --- APU3C4 --- WAN (Gigabit LAN) --- APU4D4 -- LAN---PC2iPerf3 Client

                                        iPerf3 test were made just plain, no specific parameters apart from -P10 to simulate 10 concurrent data streams.
                                        After all tuning, the faster I get is 89-92 Mb/s on OpenVPN and 240-250 Mb/s on IPsec on the same link.
                                        Be aware that my IPSec config was made to accomplish a link with a provider that request the lowest crypto-settings available, so OpenVPN and IPSec are not 1:1 comparable on crypto-computing requirements .

                                        OpenVPN: TLS Auth - 128-GCM- DH2
                                        Ipsec; Phase1: PSK-3DES-SHA1-DH2 and Phase2: 3DES-SHA1

                                        Maybe if requirements are higher on IPSec the numbers will get closer. I don't know.
                                        I had been deploying OpenVPN Site2Site links without giving IPSec a chance. Don't ask me why. Stubborn on OpenVPN maybe.
                                        Now we're retesting our configurations and moving them to IPSec if all test are successful and no other requirements appears.

                                        I don't understand though, why is there a "provider" involved between the IPSec tunnel that poses these crypto settings requirement? I thought it's just like OpenVPN where one end would be the VPN server and the other end the client?

                                        C 1 Reply Last reply Reply Quote 0
                                        • S
                                          stefanl @Qinn
                                          last edited by

                                          @Qinn Thanks for the tips. I don't think I'll reinstall pfSense to a single disk ZFS. My NAS haves it, but it haves 12x 4TB disks.

                                          Regarding watchdog, it's enabled and working, but somehow it isn't showing on the dashboard nor in status --> services.

                                          QinnQ 1 Reply Last reply Reply Quote 0
                                          • C
                                            cysiacom @kevindd992002
                                            last edited by

                                            @kevindd992002 I'm afraid I didn't explain myself clearly.
                                            Sorry for that.

                                            One customer needed to access to their provider servers.
                                            The access is granted via Site2Site IPsec VPN with very low crypto requirements and also quite unusual IPs (they use "fake" public IPs inside the tunnel).
                                            We had some trouble getting Phase2 working for that customer and provider so we planned some lab tests.
                                            There's no need for provider at all in general but just for this case, our customer, in particular.

                                            When doing local tests we did realize the speed change on IPSec tunnels compared to OpenVPN so we did some other test for our own purposes.
                                            Again.

                                            There's no need for external or any provider. It was just only on our specific customer needs.

                                            K 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.