Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Upgrade to 21.02-RELEASE borked on SG-3100

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    81 Posts 28 Posters 25.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Y
      yaminb
      last edited by yaminb

      Another update.

      With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
      It feels like ncpu=1 has helped, but I don't think it's the issue

      I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @yaminb
        last edited by bmeeks

        @yaminb said in Upgrade to 21.02-RELEASE borked on SG-3100:

        Another update.

        With the hw.ncpu=1 fix, it seemed to run fine, but now has locked up twice.
        It feels like ncpu=1 has helped, but I don't think it's the issue

        I've put in a scheduled cron reboot every night to see if that keeps it up during working hours.

        The switch to using a single CPU is a workaround that minimizes the chance of hitting the bug, but it does not eliminate the chance.

        The actual problem has been identified and a fix is being tested. Here is a link to the discussion by the FreeBSD kernel programming nerds* of the problem and the fix: https://reviews.freebsd.org/D28821. I believe the pfSense team is now vigorously testing images with this fix applied to be sure the fix is really "the fix". And from the activity on the Redmine bug site for pfSense, it looks like a few other bugs are being addressed as well.

        Note -- I don't mean "nerds" in an insulting sense 😀. But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

        lohphatL 1 Reply Last reply Reply Quote 3
        • R
          RobertBrooks
          last edited by

          How likely is this issue specific to the SG-3100?

          My faithful SG-2400 has required a few reboots since the upgrade,
          after never needing to be restarted in its five years of service.

                                   -- Rob
          
          bmeeksB 1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @RobertBrooks
            last edited by

            @robertbrooks said in Upgrade to 21.02-RELEASE borked on SG-3100:

            How likely is this issue specific to the SG-3100?

            My faithful SG-2400 has required a few reboots since the upgrade,
            after never needing to be restarted in its five years of service.

                                     -- Rob
            

            An SG-2400? Do you maybe mean the SG-2100? If 2100, that is also an ARM CPU, but it is a 64-bit version.

            R 1 Reply Last reply Reply Quote 0
            • R
              RobertBrooks @bmeeks
              last edited by

              @bmeeks

              I misspoke; I meant an SG-2440, which I recognize has an Atom CPU; I didn't know if this problem is specific to the
              ARM.

                                                      -- Rob
              
              1 Reply Last reply Reply Quote 0
              • lohphatL
                lohphat @bmeeks
                last edited by lohphat

                @bmeeks said in Upgrade to 21.02-RELEASE borked on SG-3100:

                Note -- I don't mean "nerds" in an insulting sense 😀. But when you live in the world of kernel spin locks and mutexes, and actually understand all that stuff, you are obligated to proudly wear the title of "kernel programming nerd".

                I've worked with kernel and compiler hackers and they are indeed a breed apart. You have to be able to abstract activity across interrupts, CPU rings, and microcode idiosyncrasies. It's the difference between playing checkers and 4D chess.

                SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_1)

                1 Reply Last reply Reply Quote 0
                • lohphatL
                  lohphat
                  last edited by lohphat

                  Note there is now a 21.02_1 release for SG-3100 to fix the FreeBSD kernel bug causing the instability.

                  https://forum.netgate.com/topic/161421/pfsense-plus-21-02-p1-now-available/11

                  SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_1)

                  D 1 Reply Last reply Reply Quote 0
                  • D
                    DeltaOne @lohphat
                    last edited by

                    @lohphat

                    We applied the 21.02_1 update about 12. hours ago. So far...all is good, our 3100 is running normally and back on two CPUs.

                    Many thanks to the Netgate teams for this quick fix!

                    S 1 Reply Last reply Reply Quote 1
                    • J
                      JGdgZPQatDDjpA @JGdgZPQatDDjpA
                      last edited by

                      @jgdgzpqatddjpa

                      Just upgraded my 3100. No issues so far.

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        shadtheman @JGdgZPQatDDjpA
                        last edited by

                        Upgraded to 21.02.1 yesterday, ran 14 hours then hard lockup. Anyone else seen this?

                        L 1 Reply Last reply Reply Quote 0
                        • L
                          lnguyen @shadtheman
                          last edited by lnguyen

                          @shadtheman Yes. One of my remote locations had a lockup after 12+ hours today around 1:15PM PST. Unfortunately no smart hands to get any useful information from console.

                          1 Reply Last reply Reply Quote 0
                          • S
                            sabennett @DeltaOne
                            last edited by

                            @deltaone Same here. No issues, running smooth. I’ve also added a 2 WG clients.
                            I plan on adding pfBlocker-ng tomorrow.
                            And let that settle in for a couple days.

                            The Netgate support staff have been nothing less then awesome to me, and for that I Thank-you.

                            M 1 Reply Last reply Reply Quote 0
                            • M
                              mcury @sabennett
                              last edited by

                              @sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

                              I plan on adding pfBlocker-ng tomorrow.

                              Don't install pfBlockerNG just yet.

                              https://redmine.pfsense.org/issues/5413
                              https://redmine.pfsense.org/issues/11444

                              dead on arrival, nowhere to be found.

                              N lohphatL S 3 Replies Last reply Reply Quote 1
                              • N
                                nonvtec @mcury
                                last edited by

                                @mcury Thanks for all the intel, I'm holding out for a while longer.

                                1 Reply Last reply Reply Quote 0
                                • lohphatL
                                  lohphat @mcury
                                  last edited by lohphat

                                  @mcury said in Upgrade to 21.02-RELEASE borked on SG-3100:

                                  @sabennett said in Upgrade to 21.02-RELEASE borked on SG-3100:

                                  I plan on adding pfBlocker-ng tomorrow.

                                  Don't install pfBlockerNG just yet.

                                  https://redmine.pfsense.org/issues/5413
                                  https://redmine.pfsense.org/issues/11444

                                  11444 is fixed.
                                  5413 has to do with the unbound DNS resolver having issues and has been an open bug for 5+ years.

                                  The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

                                  SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_1)

                                  M 2 Replies Last reply Reply Quote 0
                                  • M
                                    mcury @lohphat
                                    last edited by

                                    @lohphat

                                    1121179c-3a62-4adf-8ee7-3c7e081cb95b-image.png

                                    They will fix 5413 first...

                                    dead on arrival, nowhere to be found.

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      sabennett @mcury
                                      last edited by

                                      @mcury Ok thanks for the heads up and the links.

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        mcury @lohphat
                                        last edited by mcury

                                        @lohphat said in Upgrade to 21.02-RELEASE borked on SG-3100:

                                        The new problem is php seems to be crashing -- that fix still hasn't been documented in a bug here or over at FreeBSD AFAIK. This new php bug is causing problems with snort, suricata, and pfBlockerNG and is not related to 5413 I believe.

                                        I opened a TAC for the pfblockerNG problem and that was their answer..

                                        "The package dev knows about this already, and there's some work to be done beforehand as well. Mainly, the following is somewhat of a per-requisite:
                                        https://redmine.pfsense.org/issues/5413"

                                        Ticket was marked as resolved yesterday..

                                        dead on arrival, nowhere to be found.

                                        lohphatL 1 Reply Last reply Reply Quote 0
                                        • lohphatL
                                          lohphat @mcury
                                          last edited by lohphat

                                          @mcury Your ticket may be resolved as is usual because the cause of your support ticket is known, but the defect itself -- tracked separately -- is still open. There's no indication they're close to fixing 5413 yet and I haven't seen the new bug tracking for the php crashes yet over on the FreeBSD tracking site.

                                          Each "scope" have their own tracking systems. 1) NetGate Support 2) pfSense bug tracking and 3) FreeBSD bug tracking.

                                          They are handled by different independent orgs -- it was pretty impressive for the FreeBSD 11444 bug to be addressed so quickly but it was due to its criticality.

                                          pfSense 5413 has been open 5 years and the status only shows a "target" for fixing but that's not a commitment of timing.

                                          So my expectation is there will be a short term work around instead of a permanent fix in the near-term.

                                          SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_1)

                                          M 1 Reply Last reply Reply Quote 2
                                          • M
                                            mcury @lohphat
                                            last edited by

                                            @lohphat Yes, ticket marked as resolved yesterday was to indicate that it's a recent feedback from Netgate and they are working on it.
                                            But thanks, personally I didn't know the bug tracking flow and how it works

                                            They are working on it, I'll be checking the redmine for news during this period

                                            dead on arrival, nowhere to be found.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.