Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    21.02-p1 really fix the issue on SG-3100?

    Official Netgate® Hardware
    9
    37
    3.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      solarizde
      last edited by solarizde

      Hey,

      I read through the whole Article what causes the Problem and applied the 21.02-p1 update yesterday to a SG-3100. Before the Patch I applied the mentioned fix to limit the System to 1 CPU in /boot/loader.conf.

      After the Patch I removed the hw.ncpu=1 from the loader.conf and rebooted. It doesn't took long and the system was in the "bug state" again. Not Forwarding any traffic, pf stuck again.

      So what actually changed that p1?

      Was it just a quick workaround by adding hw.ncpu=1 to the loader.conf? This would explain why it stuck again after I remove it.

      For me currently I still cannot run the SG-3100 without the hw.ncpu=1 limit.

      Thanks for any Ideas / Insights.

      R S 2 Replies Last reply Reply Quote 0
      • R
        rsherwood_va @solarizde
        last edited by

        @solarizde I have similar problem, though by now I have the workaround down cold. after an upgrade or reboot, no traffic is passed through my 3100 and the webGUI is unavailable. However, I can ssh into the console, restart webConfigurater (option 11), then access the site, rerun the initial setup wizard ("System" -> "Setup Wizard") choosing all of the default options and entering my existing admin password as the "new" password, then on reload the whole thing starts working beautifully. Not sure why this works but it's extremely reliable for me. Thinking about selling my 3100 on craigslist and getting a unifi gw. :(

        1 Reply Last reply Reply Quote 2
        • bmeeksB
          bmeeks
          last edited by

          Would both of you mind doing me a favor? I'm chasing a bug in the Snort package that is impacting SG-3100 devices only.

          Look in your system log (under STATUS > SYSTEM LOGS) and see if you find any messages logged about a Signal 11 crash from php or php-fpm.

          Also, would you mind listing any installed packages you are running?

          Thanks,
          Bill

          R lohphatL 2 Replies Last reply Reply Quote 0
          • S
            SteveITS Rebel Alliance @solarizde
            last edited by

            stephenw10 said in pfSense Plus and SG-3100:

            No, it's a real fix. See: https://reviews.freebsd.org/D28821

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            1 Reply Last reply Reply Quote 0
            • R
              rsherwood_va @bmeeks
              last edited by

              @bmeeks

              Here is a screenshot from my dashboard of installed packages. During the upgrade, I removed pfBlockerNG, which is the only one I had installed, and which was disabled in any case.

              alt text

              I do see signal 11 errors in the log:

              Feb 17 20:55:30 kernel pid 390 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 17 21:10:38 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 17 21:46:12 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 17 22:50:18 kernel pid 27679 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 17 23:08:17 kernel pid 30028 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 23 10:56:00 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 23 11:11:08 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 23 11:22:52 kernel pid 400 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 26 02:11:29 kernel pid 54431 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 26 06:18:04 kernel pid 589 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 26 06:18:20 kernel pid 32405 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 26 06:27:51 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
              Feb 26 06:33:24 kernel pid 60772 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)

              bmeeksB 1 Reply Last reply Reply Quote 0
              • lohphatL
                lohphat @bmeeks
                last edited by lohphat

                @bmeeks Do you think this is a php package bug or another kernel lock bug? I don't use snort or suricata but do use pfblockerNG and now that I've seen this issue, I think holding off on 21.02_1 too until it's isolated.

                SG-3100 24.03-RELEASE (ARM) | Avahi (2.2_5) | ntopng (5.6.0) | openvpn-client-export (1.9.3) | pfBlockerNG-devel (3.2.0_20) | System_Patches (2.2.11_15)

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @rsherwood_va
                  last edited by

                  @rsherwood_va said in 21.02-p1 really fix the issue on SG-3100?:

                  @bmeeks

                  Here is a screenshot from my dashboard of installed packages. During the upgrade, I removed pfBlockerNG, which is the only one I had installed, and which was disabled in any case.

                  alt text

                  I do see signal 11 errors in the log:

                  Feb 17 20:55:30 kernel pid 390 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 17 21:10:38 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 17 21:46:12 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 17 22:50:18 kernel pid 27679 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 17 23:08:17 kernel pid 30028 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 23 10:56:00 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 23 11:11:08 kernel pid 401 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 23 11:22:52 kernel pid 400 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 26 02:11:29 kernel pid 54431 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 26 06:18:04 kernel pid 589 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 26 06:18:20 kernel pid 32405 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 26 06:27:51 kernel pid 399 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
                  Feb 26 06:33:24 kernel pid 60772 (php-fpm), jid 0, uid 0: exited on signal 11 (core dumped)

                  Thank you for the info. That confirms my suspicion.

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks @lohphat
                    last edited by

                    @lohphat said in 21.02-p1 really fix the issue on SG-3100?:

                    @bmeeks Do you think this is a php package bug or another kernel lock bug? I don't use snort or suricata but do use pfblockerNG and now that I've seen this issue, I think holding of on 21.02_1 too until it's isolated.

                    I'm leaning towards it being a PHP bug on 32-bit ARM hardware. Especially since it seems several packages are impacted in similar ways. Whatever this problem is, it's not caused by the latest 21.02_1 update (nor is that update likely to fix it). It looks like something that came in with FreeBSD-12.2/STABLE.

                    M lohphatL 2 Replies Last reply Reply Quote 1
                    • M
                      mcury @bmeeks
                      last edited by

                      I'm also affected by it..

                      Feb 25 19:52:38 kernel pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

                      Removing pfblockerNG-devel completely, solved the problem for me.
                      I don't use Snort/Suricata

                      A 1 Reply Last reply Reply Quote 0
                      • A
                        alhh @mcury
                        last edited by

                        @mcury

                        Almost the same here, I was using pfBlockerNG (non-dev).
                        Tested both: cpu restriction on/off.

                        Disabling pfBlockerNG (it's still installed) solved the problem.
                        CPU restriction is NOT necessary.

                        1 Reply Last reply Reply Quote 0
                        • S
                          solarizde
                          last edited by solarizde

                          @bmeeks said in 21.02-p1 really fix the issue on SG-3100?:

                          messages logged about a Signal 11 crash

                          Here we go, not so many but still there.
                          SG-3100, 21.02-RELEASE-p1

                          Packages:
                          Avahi, Cron, iperf, openvpn-client-export, pfBlockerNG-dev, Service_Watchdog

                          Feb 26 04:31:19 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:35:22 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:44:38 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

                          I currently try over night with 21.02-RELEASE-p1 fixed to 1 CPU by loader.conf and pfBlocker enabled. If I enable pfBlocker without the CPU Limit in loader.conf pf stop forwarding traffic within half an hour.

                          bmeeksB M 2 Replies Last reply Reply Quote 0
                          • lohphatL
                            lohphat @bmeeks
                            last edited by

                            @bmeeks Could you please post the FreeBSD PHP bug link and the pfSense tracking bug here for reference so that we can follow?

                            I think I'm going to hold off until this bug is fixed before upgrading to 21.02_x unless the PHP package can be fixed as part of a package update.

                            SG-3100 24.03-RELEASE (ARM) | Avahi (2.2_5) | ntopng (5.6.0) | openvpn-client-export (1.9.3) | pfBlockerNG-devel (3.2.0_20) | System_Patches (2.2.11_15)

                            S 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @solarizde
                              last edited by bmeeks

                              @solarizde said in 21.02-p1 really fix the issue on SG-3100?:

                              @bmeeks said in 21.02-p1 really fix the issue on SG-3100?:

                              messages logged about a Signal 11 crash

                              Here we go, not so many but still there.
                              SG-3100, 21.02-RELEASE-p1

                              Packages:
                              Avahi, Cron, iperf, openvpn-client-export, pfBlockerNG-dev, Service_Watchdog

                              Feb 26 04:31:19 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:35:22 pfSense kernel: pid 375 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)
Feb 26 07:44:38 pfSense kernel: pid 374 (php-cgi), jid 0, uid 0: exited on signal 11 (core dumped)

                              Thanks! These reports will, I hope, help make the case the problem is really in the PHP engine and not the packages themselves. Snort, Suricata, Unbound and pfBlockerNG-dev are all triggering Signal 11 crashes. And Snort, Suricata and pfBlockerNG-dev are all doing so in the PHP engine. You really should never be able to crash PHP itself.

                              1 Reply Last reply Reply Quote 0
                              • M
                                mcury @solarizde
                                last edited by

                                @solarizde Remove pfblockerNG, not sure if only disabling it would solve this issue.
                                I opened a TAC ticket with Netgate #INC-76936, and they said: "dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:

                                https://redmine.pfsense.org/issues/5413 "

                                So, they are working on it..
                                They did a great job with 21.02p1 and I know that they will do it again..

                                S 1 Reply Last reply Reply Quote 1
                                • S
                                  solarizde @mcury
                                  last edited by solarizde

                                  @mcury said in 21.02-p1 really fix the issue on SG-3100?:

                                  So, they are working on it..

                                  That redmine Issue is related to a DNS service interrupion which is bad too, but not as bad as the Sig11 on pf.

                                  This is the better place:
                                  redmine.pfsense.org/issues/11444

                                  @mcury said in 21.02-p1 really fix the issue on SG-3100?:

                                  Remove pfblockerNG

                                  Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.

                                  M 2 Replies Last reply Reply Quote 1
                                  • M
                                    mcury @solarizde
                                    last edited by

                                    @solarizde Thanks, I thought that one was already closed, they reopened.
                                    Nice to see that they are taking this seriously :)

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      mcury @solarizde
                                      last edited by

                                      @solarizde said in 21.02-p1 really fix the issue on SG-3100?:

                                      Sure this will "fix" the Crash but I want to figure out in which case it really happens. If it happen with 1 CPU Disabled too it is not so much reltated to the memory baricade bug.

                                      I'm kind of a noob regarding this technical stuff, cores and such.. but what I understood is that one CPU is trying to read in memory while the other is still writing to it, basically some kind of sync issue between cores, but again, I'm noob and maybe got it all wrong :)

                                      lohphatL 1 Reply Last reply Reply Quote 0
                                      • S
                                        SteveITS Rebel Alliance @lohphat
                                        last edited by

                                        @lohphat said in 21.02-p1 really fix the issue on SG-3100?:

                                        PHP bug link

                                        Here is another about the PHP crash/signal 11 on SG-3100.

                                        21.02-p1 fixed a different locking issue in the kernel on SG-3100s.

                                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                        Upvote 👍 helpful posts!

                                        lohphatL 1 Reply Last reply Reply Quote 1
                                        • lohphatL
                                          lohphat @SteveITS
                                          last edited by lohphat

                                          @teamits Perfect. Thank you.

                                          I'm a bit more concerned about some of the other open issues cited here in earlier posts. One bug has been open for 5 years; I hope it's not a dependency.

                                          I was joking the other day with my ex-NSCP coworkers that there's an open Thunderbird (Mozilla) issue in Bugzilla that's been open 20 YEARS and still isn't fixed. https://bugzilla.mozilla.org/show_bug.cgi?id=92165

                                          "[Free software] is only 'free' if your time has no value." - jwz (he was talking about linux)

                                          SG-3100 24.03-RELEASE (ARM) | Avahi (2.2_5) | ntopng (5.6.0) | openvpn-client-export (1.9.3) | pfBlockerNG-devel (3.2.0_20) | System_Patches (2.2.11_15)

                                          1 Reply Last reply Reply Quote 0
                                          • lohphatL
                                            lohphat @mcury
                                            last edited by lohphat

                                            @mcury Basically the 3100 fix was to address a missing "memory barrier" instruction on the arm7 platform.

                                            Since modern CPUs can execute instructions out of order to speed execution, there are times where a process needs to guarantee that all previous instructions are complete (and not being executed still in parallel or out-of-order). This is usually to prevent a race/deadlock condition.

                                            More info here: https://en.wikipedia.org/wiki/Memory_barrier

                                            SG-3100 24.03-RELEASE (ARM) | Avahi (2.2_5) | ntopng (5.6.0) | openvpn-client-export (1.9.3) | pfBlockerNG-devel (3.2.0_20) | System_Patches (2.2.11_15)

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post