Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    21.02 Sudden lockup

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    164 Posts 30 Posters 51.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rloeb
      last edited by

      Thank you!

      1 Reply Last reply Reply Quote 0
      • R
        rloeb @kphillips
        last edited by

        @kphillips FYI. Went to 1 CPU. Snort does not start. Snort does not appear in Services menu. Re-installing snort fails; Window just sits there. Would like to have that protection, but not sure what to do next.

        K 1 Reply Last reply Reply Quote 0
        • L
          lnguyen @kphillips
          last edited by

          @kphillips said in 21.02 Sudden lockup:

          Command Prompt and run "echo hw.ncpu=1 >> /boot/loader.conf" without quotes.

          I think what @jimp stated was:
          Create /boot/loader.conf.local if it doesn't exist, as loader.conf can be overwritten by pfSense.

          echo hw.ncpu=1 >> /boot/loader.conf.local
          

          I agree with this as it won't be overwritten and easily reverted once a patch is released by simply issuing:

          rm /boot/loader.conf.local
          
          R K 2 Replies Last reply Reply Quote 1
          • R
            rloeb @lnguyen
            last edited by

            @lnguyen Nice catch. The command did not create the file. My Linux is pretty feeble these days, so I'm unclear what to do next. I'll see if I can pull one of my techies off what they're doing and he can chase this. I'll go back to running the company, which is all I'm competent to do.

            1 Reply Last reply Reply Quote 0
            • K
              kphillips Administrator Netgate @lnguyen
              last edited by

              @lnguyen You are correct. I've updated my original post.

              1 Reply Last reply Reply Quote 1
              • K
                kphillips Administrator Netgate @rloeb
                last edited by

                @rloeb Snort is broken on the SG-3100 and pfSense Plus 21.02. This is due to a bug in the package, not pfSense Plus. Snort has some badly coded components that Intel CPUs usually just "auto fix", but on ARM that mechanism doesn't exist. As such, something appears to have broken during the move to 21.02. We have a bug report for that, but if you need snort you'll want to be on 2.4.5p1 until that is sorted. Otherwise, Suricata works fine on the SG-3100 AFAIK right now on 21.02.

                R styxlS 2 Replies Last reply Reply Quote 0
                • R
                  rloeb @kphillips
                  last edited by

                  @kphillips Thank you. Good to know. Going to try to tough it out. Alternative is to switch to a gateway router with no filtering, just to keep folks productive.

                  1 Reply Last reply Reply Quote 0
                  • styxlS
                    styxl @kphillips
                    last edited by

                    @kphillips did they block all packages from 2.4.5-p1? i rolled back and restored from 21.02 now i cannot re-install all the packages for 2.4.5-p1 what is going on?

                    S K 2 Replies Last reply Reply Quote 0
                    • S
                      SteveITS Galactic Empire @styxl
                      last edited by

                      @styxl Did you set System/Update/Update Settings to "previous stable version (2.4.5)"? I can see packages on a 3100 that wasn't upgraded.
                      You might try https://docs.netgate.com/pfsense/en/latest/troubleshooting/pkg-broken-database.html.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote 👍 helpful posts!

                      styxlS 1 Reply Last reply Reply Quote 1
                      • K
                        kphillips Administrator Netgate @styxl
                        last edited by

                        @styxl Make sure you select "Previous Stable Version (2.4.x)" under System --> Update. The repos for 21.02 are now offline to keep people from upgrading to it for the SG-3100 right now.

                        1 Reply Last reply Reply Quote 1
                        • styxlS
                          styxl @SteveITS
                          last edited by

                          @teamits i just did and it worked, thx

                          1 Reply Last reply Reply Quote 0
                          • A
                            alpharulez
                            last edited by

                            Does anyone know if Suricata on 21.02 is impacted the same as Snort? Thanks!

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              On the SG-3100 it would be, in blocking mode at least. Like Snort it has to reload the ruleset whenever a new IP is added to the block table.

                              Steve

                              A 1 Reply Last reply Reply Quote 1
                              • A
                                alpharulez @stephenw10
                                last edited by

                                @stephenw10 ok thanks for the response 👍
                                Will hold fire.

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  ....unless you're seeing this: https://redmine.pfsense.org/issues/11466
                                  That applies to Snort only.

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    router
                                    last edited by

                                    Hello, is there an update on this issue?

                                    I'm experiencing major packet loss and unable to download new packages.

                                    I've already added hw.ncpu=1 to /boot/loader.conf.local.
                                    This had no noticeable affect.

                                    Our systems are completely degraded by this issue. We cannot handle the risk and downtime required to reinstall. This is a major impact for us.

                                    Thanks...

                                    K 1 Reply Last reply Reply Quote 0
                                    • K
                                      kphillips Administrator Netgate @router
                                      last edited by

                                      @router Packet loss is not a symptom of this issue. The SG-3100 would completely freeze up and force a reboot. If you have packet loss, its not 21.02 most likely. Check your gateway monitoring.

                                      N R 2 Replies Last reply Reply Quote 0
                                      • B
                                        bldnightowl
                                        last edited by

                                        Well, add me to the list of people that downgraded back to 2.4.5p1. And that was quite a hassle/nightmare by itself. Even with pfBlockerNG removed (which was an unsustainable solution for any period of time, of course), the system was still freezing or behaving erratically at times. Getting the packages back to the way they were pre-21.02 did not automatically happen as it should have -- and I had to manually intervene several times. This failed upgrade cost me a couple of days at least of my time, and like others I am very unhappy about that.

                                        I am a software engineer too and understand how very hard it is to test field configurations for an extremely customizable product. So I'm not trying to make anyone at Netgate fill badly --- but this was pretty disastrous for many users, and a a detailed post mortem explaining what went wrong, why and how it will be avoided in the future would be hugely appreciated. For example, it appears your QA did not have pfBlockerNG(-devel) (which I would be willing to guess is in very widespread use) properly in its standard performance testsuite. I hope that's been rectified.

                                        Thanks for the hard work and responsiveness when things did blow up, particularly you moderators on the front lines absorbing all the screams from your users. And especially to those of you responding while impacted by the much worse disasters in Texas.

                                        K MaxK 0M 2 Replies Last reply Reply Quote 3
                                        • K
                                          kphillips Administrator Netgate
                                          last edited by

                                          In case anyone is wondering what the root cause of the SG-3100 locking up was, here is the FreeBSD compiler issue that has been fixed and will be used for the fixed release when it comes out. Dev team has been working hard over the weekend on this one.

                                          https://reviews.freebsd.org/D28821

                                          A 1 Reply Last reply Reply Quote 5
                                          • N
                                            nick108 @kphillips
                                            last edited by nick108

                                            @kphillips I have to add my name into the packet loss issue. I've had this SG-3100 since approx 2019 and across multiple ISPs I've only had one instance of packet loss and that was not pfSense related. After disabling pfBlockerNG-devel I have so far had 1 or 2 complete lockup and today had 2 instances of 90%+ packetloss over my IPV4 main gateway and the overlying IPv6 over v4 tunnel which exits the same gateway. No CPU spikes that I could see.

                                            K 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.