Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Snort won't start after upgrade to 21.02 on SG-3100

    IDS/IPS
    12
    43
    7.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeksB
      bmeeks @marcosm
      last edited by bmeeks

      @marcos-ng said in Snort won't start after upgrade to 21.02 on SG-3100:

      Thanks for the info. It's possible this is related to a current issue with the SG-3100. I've created a bug report here to keep track:
      https://redmine.pfsense.org/issues/11466

      No, not likely related. Signal 10 Bus Error aborts are caused by non-word aligned access to memory. A Signal 11 is a conventional memory segmentation fault where a program attempted to access a memory region it was not authorized to access.

      I am very familiar with the Snort issue on SG-3100 and SG-1000 hardware as I debugged and identified the cause shortly after the SG-3100 was introduced. As I stated, there previously was a partial workaround in place by compiling the Snort and Suricata packages with debugging enabled to prevent the llvm compiler from using the specific binary instructions that do not perform auto-fixup for non-aligned access. Either that workaround has ceased to function in FreeBSD-12.2, or the package compiler switch was toggled off. I suspect it may be the former, though.

      P.S. -- I am the volunteer developer and maintainer for the Snort and Suricata packages on pfSense, so I know them both intimately.

      1 Reply Last reply Reply Quote 4
      • M
        marcosm Netgate
        last edited by

        Thank you for the input and all of the hard work! :)

        It sounds like sig 10 is its own regression then, and sig 11 being the one possibly related to the issue linked on the redmine entry.

        1 Reply Last reply Reply Quote 0
        • S
          SteveITS Rebel Alliance
          last edited by

          Just to be clear this issue affects Suricata also?

          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
          Upvote 👍 helpful posts!

          bmeeksB 1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @SteveITS
            last edited by bmeeks

            @teamits said in Snort won't start after upgrade to 21.02 on SG-3100:

            Just to be clear this issue affects Suricata also?

            Yes, it likely can. No direct reports yet, but I would not be surprised. Especially in the case of the PHP code that is causing the Signal 11 segfault in PHP. That code is identical in the two packages. And it's not really the code itself that is the problem. It's the PHP engine underneath that is crashing. The same GUI code is used in all of the pfSense images (CE and pfSense+, and for all hardware platforms such as aarch64, x86-64 and arm. So if the upper-level GUI code was faulty, you would expect it to fail on all the platforms. That's not the case. It is only failing on the ARM 32-bit platform.

            S 1 Reply Last reply Reply Quote 0
            • S
              SteveITS Rebel Alliance @bmeeks
              last edited by

              Yeah, I saw your posts in other threads, after mine. Sure sounds like PHP. Unfortunately that presumably means it's on someone upstream to fix it which means a 21.02p2. 😐 Good sleuthing.

              Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
              When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
              Upvote 👍 helpful posts!

              1 Reply Last reply Reply Quote 1
              • R
                rsm4
                last edited by

                Just for some analytics on Snort and PHp crashes, I ran a Splunk query against my syslogs from the router going back 30 days to see what was logged:

                2021-02-18T14:02:43.000-0500,Feb 18 14:02:43  kernel: pid 46827 (snort) uid 0: exited on signal 10
2021-02-18T14:07:54.000-0500,Feb 18 14:07:54  kernel: pid 56899 (snort) uid 0: exited on signal 10
2021-02-18T14:10:04.000-0500,Feb 18 14:10:04  kernel: pid 21583 (snort) uid 0: exited on signal 10
2021-02-18T14:13:13.000-0500,Feb 18 14:13:13  kernel: pid 78778 (php-cgi) uid 0: exited on signal 11 (core dumped)
2021-02-18T14:18:38.000-0500,Feb 18 14:18:38  kernel: pid 1154 (snort) uid 0: exited on signal 10
2021-02-18T14:23:24.000-0500,Feb 18 14:23:24  kernel: pid 75058 (snort) uid 0: exited on signal 10
2021-02-18T14:26:01.000-0500,Feb 18 14:26:01  kernel: pid 26020 (snort) uid 0: exited on signal 10
2021-02-18T14:26:30.000-0500,Feb 18 14:26:30  kernel: pid 97052 (snort) uid 0: exited on signal 10
2021-02-18T14:29:12.000-0500,Feb 18 14:29:12  kernel: pid 84487 (snort) uid 0: exited on signal 10
2021-02-18T14:53:13.000-0500,Feb 18 14:53:13  kernel: pid 63165 (snort) uid 0: exited on signal 10
2021-02-18T14:55:54.000-0500,Feb 18 14:55:54  kernel: pid 64348 (snort) uid 0: exited on signal 10
2021-02-18T15:04:04.000-0500,Feb 18 15:04:04  kernel: pid 17533 (php-fpm) uid 0: exited on signal 11 (core dumped)
2021-02-18T15:05:49.000-0500,Feb 18 15:05:49  kernel: pid 9318 (snort) uid 0: exited on signal 10
2021-02-18T15:11:43.000-0500,Feb 18 15:11:43  kernel: pid 65338 (snort) uid 0: exited on signal 10
2021-02-18T15:22:04.000-0500,Feb 18 15:22:04  kernel: pid 24027 (snort) uid 0: exited on signal 10
2021-02-18T19:21:13.000-0500,Feb 18 19:21:13  kernel: pid 5625 (snort) uid 0: exited on signal 10
2021-02-20T10:22:02.000-0500,Feb 20 10:22:02  kernel: pid 42369 (php-cgi) uid 0: exited on signal 11 (core dumped)
2021-02-20T10:24:28.000-0500,Feb 20 10:24:28  kernel: pid 74738 (snort) uid 0: exited on signal 10
2021-02-21T16:06:59.000-0500,Feb 21 16:06:59  kernel: pid 30776 (snort) uid 0: exited on signal 10
2021-02-21T16:38:27.000-0500,Feb 21 16:38:27  kernel: pid 75666 (snort) uid 0: exited on signal 10
2021-02-21T19:28:30.000-0500,Feb 21 19:28:30  kernel: pid 67353 (snort) uid 0: exited on signal 10
2021-02-21T19:43:31.000-0500,Feb 21 19:43:31  kernel: pid 86017 (snort) uid 0: exited on signal 10
2021-02-21T19:48:33.000-0500,Feb 21 19:48:33  kernel: pid 81269 (snort) uid 0: exited on signal 10
2021-02-24T01:36:24.000-0500,Feb 24 01:36:24  kernel: pid 81513 (snort) uid 0: exited on signal 11
2021-02-24T01:36:26.000-0500,Feb 24 01:36:26  kernel: pid 62078 (snort) uid 0: exited on signal 11
2021-02-25T22:28:10.000-0500,Feb 25 22:28:10  kernel: pid 78826 (php-fpm) uid 0: exited on signal 11 (core dumped)
2021-02-25T22:29:59.000-0500,Feb 25 22:29:59  kernel: pid 73568 (php-fpm) uid 0: exited on signal 11 (core dumped)
2021-02-25T22:34:50.000-0500,Feb 25 22:34:50  kernel: pid 77758 (php-fpm) uid 0: exited on signal 11 (core dumped)
2021-02-25T22:35:09.000-0500,Feb 25 22:35:09  kernel: pid 28596 (snort) uid 0: exited on signal 11
2021-02-25T22:35:11.000-0500,Feb 25 22:35:11  kernel: pid 28801 (snort) uid 0: exited on signal 11
2021-02-25T23:15:01.000-0500,Feb 25 23:15:01  kernel: pid 60474 (snort) uid 0: exited on signal 10
2021-02-26T07:15:01.000-0500,Feb 26 07:15:01  kernel: pid 32892 (snort) uid 0: exited on signal 10
2021-02-27T09:24:14.000-0500,Feb 27 09:24:14  kernel: pid 45195 (snort) uid 0: exited on signal 10
2021-02-28T10:15:01.000-0500,Feb 28 10:15:01  kernel: pid 53195 (snort) uid 0: exited on signal 10
2021-02-28T21:12:07.000-0500,Feb 28 21:12:07  kernel: pid 21339 (snort) uid 0: exited on signal 10

                Feb 18 was the day I upgraded my SG-3100, so, as expected, no crashes were occurring prior to that. (I can go back further--there were no crashes in the last 180 days)

                Also, it's not clear to me that the PHP and Snort crashes are related. I have a few PHP crashes and some seem coincident with Snort, but Snort definitely crashes without PHP crashing.

                My SG-3100 is removed from service so now I can start Snort and it won't crash until later. Incidentally, the pfBlockerNG update will almost always cause it to crash. This could be because Snort has traffic to scan at this point. (Traffic to the WebConfigurator doesn't count? Perhaps because it is SSL?)

                1 Reply Last reply Reply Quote 0
                • styxlS
                  styxl
                  last edited by

                  is there a fix for this in the hopper? and when should we expect it? i had to rollback to 2.4.5p1 thus i am still waiting for the fix

                  1 Reply Last reply Reply Quote 1
                  • R
                    rek0n
                    last edited by

                    Im having the same problem aswell, SG-3100 on 21.02-p1. Snort is not visible anymore. I managed to uninstall it the package but when trying to reinstall it it just hangs on "Please wait while the update system initializes". Is it possible to revert to 2.4.5? And if so, how can I do this? It seems there is no official rollback feature that I can use to perform a rollback.

                    S 1 Reply Last reply Reply Quote 1
                    • S
                      SteveITS Rebel Alliance @rek0n
                      last edited by

                      @styxl said in Snort won't start after upgrade to 21.02 on SG-3100:

                      is there a fix for this in the hopper

                      I'm not a dev but if the issue is with PHP then it could go all the way back to Zend to find and fix it. And then it would presumably be in a p2 patch. I'm not optimistic it will be "soon."

                      @rek0n I haven't done this myself but this seems like the right path:

                      • get a copy of the 2.4.5 installer from Netgate (go.netgate.com)
                      • install 2.4.5
                      • set to Previous Stable Version in System/Update (to install 2.4.5 packages not 2.5)
                      • install desired packages
                      • restore configuration
                      • double check Previous Stable Version is still set

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote 👍 helpful posts!

                      R H 2 Replies Last reply Reply Quote 0
                      • R
                        rek0n @SteveITS
                        last edited by

                        @teamits It seems that since I do not have an active subscription, so I am unable to create a ticket. I tried searching ftp repositories, and discovered that there are community releases, but these seem all to be amd64. Since I have the SG-3100 that is on arm64 Im wondering where I could obtain the SG-3100 arm64 2.4.5 image..

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          marcosm Netgate @rek0n
                          last edited by

                          There have been no news as of yet. To roll back, you may create an account and request the previous stable version. A support subscription is not required.

                          1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks
                            last edited by bmeeks

                            You can follow the status of the bug here: https://redmine.pfsense.org/issues/11466.

                            As you can see in the notes, I worked on finding the bug as far as I could go. The issue is within PHP itself, and appears limited to the 32-bit ARM processor in the SG-3100. I say this because the identically same PHP code runs without issue on 64-bit ARM hardware such as the SG-1100 and also on all Intel hardware. PHP crashing is why Snort won't start.

                            1 Reply Last reply Reply Quote 1
                            • styxlS
                              styxl
                              last edited by

                              It is sad we are being forced to choose between security/stability and access. The SG-3100 is a good viable product and a lot of Small to Medium Enterprises use it. I am still surprised that Netgate is yet to patch this knowing a lot of their SG-3100 installs are using SNORT as an IPS/IDS. Personally i chose to rollback and wait until a patch is published but one of my peers decided to ditch the SG-3100 and buy an SG-5100.... unfortunately i dont have that kind of money.

                              S 1 Reply Last reply Reply Quote 2
                              • S
                                SteveITS Rebel Alliance @styxl
                                last edited by

                                @styxl said in Snort won't start after upgrade to 21.02 on SG-3100:

                                Netgate is yet to patch this

                                But that's the thing, if it's a PHP problem Netgate may have to wait for Zend to fix it? Zend tends to update once a month and the March 4 update just came out. Plus if it's a compilation bug and not a code bug I would think that makes it harder. And if a new PHP is included in a 21.2-p2 then Netgate would presumably need to test all of pfSense before release. (I don't have any inside knowledge of this, I'm just connecting dots.)

                                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                Upvote 👍 helpful posts!

                                1 Reply Last reply Reply Quote 0
                                • B
                                  Biggy823
                                  last edited by

                                  I too have run into this issue. I spent too much time trying to get this to work before coming across this post. Netgate should flag the upgrade and caution Snort users that it will break their setup. Does anyone have any idea when this will be fixed? I don't want have to pull out the SG-3100 out of my environment. I think it's a great product.

                                  Thanks

                                  1 Reply Last reply Reply Quote 1
                                  • B
                                    bartekmao
                                    last edited by

                                    Having the same issue, as I understood from the bug report it seems to be more of an PHP issue now. Too bad Netgate did not warn us before upgrading.

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      For the PHP crashes, try the patch to disable PHP PCRE JIT on #11466 Note 32.

                                      You can install the System Patches package and then create an entry for the patch URL https://redmine.pfsense.org/attachments/download/3707/patch-disable-pcrejit-arm.diff to apply the fix.

                                      Then run console menu options 16 and 11 to restart PHP and the GUI, or reboot.

                                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      S 1 Reply Last reply Reply Quote 1
                                      • S
                                        s0m3f00l @jimp
                                        last edited by

                                        @jimp I have applied the patch and will begin testing. I use Suricata, OPENVPN, and PfBlockerNG so hopefully this fixes things and we can mark it solved.

                                        1 Reply Last reply Reply Quote 0
                                        • R
                                          rsm4
                                          last edited by

                                          This post is deleted!
                                          1 Reply Last reply Reply Quote 0
                                          • R
                                            rsm4
                                            last edited by

                                            This sounds like it fixes the PHP issue, but what about the Snort issue? These are two separate problems on the SG-3100. PHP is crashing, but also, on my SG-3100, Snort starts and then Snort core dumps when it starts looking at traffic.

                                            I applied the diff above and restarted PHP and the GUI. I no longer see the PHP errors (in the few hours since I have applied it), but I still see Snort crashing:

                                            "Jun 12 10:21:42 sg3100 kernel: pid 92832 (snort), jid 0, uid 0: exited on signal 10"

                                            My SG-2220 (Intel-based) of course is working fine with Snort.

                                            bmeeksB 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post