Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

[bmeeks]

@johnpitton said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

I have a few Netgate 7100 units all running 23.09 with Snort VRT rules in legacy Blocking mode and Kill State enabled with 7 days blocking set.
They've been at this 23.09 ver for almost a week now and stable.
All of them have active blocks listed.
I checked and verified SNORT rulesets have been updating successfully.
I don't see in any of the logs any signal 11 core dumps logged.

What version of the Snort package is installed on your box? Look under SYSTEM > PACKAGE MANAGER and post back the version shown there. There has been an issue with the package builder for 23.09, so my understanding is that some of the most recent package updates have not been deployed in that branch.

[johnpitton]

@bmeeks

[bmeeks]

@johnpitton said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks

Yes, that is the older version of the package. Current is now 4.1.6_13 for the GUI and 2.9.20_5 for the associated binary. The newer package is held up currently for 23.09 because of a different problem with the package builder backend. In your case, I would advise sticking with the version you have until we find and squash the bug introduced in the newest package.

Thank you for this data point. It is helpful in my troubleshooting to isolate the new bug. The code change which appears to have introduced the bug happened with the 2.9.20_4 version of the Snort binary.

[bmeeks]

I think I may have found the Signal 11 core dump bug. If I am correct, it's actually in the FreeBSD libpfctl library and not directly in the Snort code. I'm waiting on the Netgate kernel developer I'm working with to either confirm my finding or show me where I went off track analyzing the library code .

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

If I am correct, it's actually in the FreeBSD libpfctl library and not directly in the Snort code.

Just out of curiosity: that could be corrected by rebuilding and replacing that library or there would be necessary a new pfsense revision?

[yorke]

pfsense CE 2.7.0
Nov 17 06:59:00 kernel pid 25245 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

[bmeeks]

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Just out of curiosity: that could be corrected by rebuilding and replacing that library or there would be necessary a new pfsense revision?

No, I don't think that could work.

However, there is a way to workaround the problem by calling the older legacy functions for killing firewall states in my custom blocking module when the FreeBSD version running on the box is less than a particular value. That would workaround the faulty library until it could be updated with a kernel update.

Updated Info:
A new version of the libpfctl package will be built and added to pfSense 2.7.1 CE and 23.09 Plus builds. I'm not sure of all the logistics involved here, so won't speculate much more.

The root cause was an effort to migrate pfctl functionality over to a wrapper library instead of using direct kernel ioctl() calls. One of the things the new wrapper library (libpfctl) does is use the new nvlist functionality in the kernel. Here is the official man page for nvlist: https://man.freebsd.org/cgi/man.cgi?query=nvlist&apropos=0&sektion=0&manpath=FreeBSD+15.0-CURRENT&arch=default&format=html. As more pfctl functionality is shifted to the libpfctl library, a few of the legacy ioctl() calls are being removed. The upcoming new versions of pfSense (sorry, but I don't know which ones) will not be able to use the legacy ioctl() calls at all for some pfctl operations. That means my custom blocking plugin had to be updated to cope with that. Without realizing there was an issue in libpfctl, the latest Snort and Suricata packages were bundled with the newly modified custom plugin code because libpfctl has been available since at least FreeBSD 12. Since both CE and Plus are on a version of FreeBSD 14, it seemed like a no-brainer.

But after digging around there appears to be a couple of lingering bugs in libpfctl depending on which specific FreeBSD kernel is in use. For example, the libpfctl version bundled with pfSense CE 2.7.0 is slightly different from the version bundled with 23.09 Plus. The CE 2.7.0 version appears to have two bugs (one of which was fixed in the later libpfctl library version bundled with 23.09 Plus). That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well. The bug triggers somewhat randomly because a unique set of circumstances have to occur to trigger it.

The fix will almost certainly involve an update to my custom blocking plugin code used in Snort and Suricata. Then the libpfctl library will likely be updated in the next pfSense version release.

More Updated Info:
I have some further refinements/updates for the custom blocking modules in both Snort and Suricata. I will be posting new versions of both packages a bit later- after the updates for libpfctl have landed.

[bmeeks]

@yorke said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

pfsense CE 2.7.0
Nov 17 06:59:00 kernel pid 25245 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

Are you using Legacy Blocking Mode? And if so, is the Kill States option checked (enabled) on the INTERFACE SETTINGS tab?

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

[bmeeks]

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

I'm not sure of all the logistics of getting that updated library onto individual machines, though. For CE 2.7.1-RC users it could happen with a new incremental build of that version. I don't know how it will work for 23.09 Plus users or CE 2.7.0 users.

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

Ah, OK then I get back to the NO Kill States ... and wait :-)

[SteveITS]

@bmeeks 2.7.1 released yesterday actually…didn’t notice any posts here but it’s on Netgate’s blog.

Perhaps this and the URL Alias bug will be sufficient to quickly produce a point release.

[bmeeks]

@SteveITS said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

2.7.1 released yesterday actually…didn’t notice any posts here but it’s on Netgate’s blog.

I've been buried head down chasing the IDS/IPS package bugs and have not checked the release status.

I do know the kernel developer and author of all the recent libpfctl library changes is working to post the fixes and get them into at least 2.7.1 and 23.09. That is all happening this morning.

[bmeeks]

I've updated the original post at the top of this thread with new information.

I'll briefly repeat that here --

A fix has been identified and implemented for the bug exposed by the custom blocking module changes in the most recent Snort and Suricata package updates. The fix requires the publishing of a new libpfctl library package and then rebuilding impacted packages where libpfctl is a build dependency. That will happen soon in the pfSense CE 2.7.1 and pfSense Plus 23.09 branches.

Not sure at this point if the fix can be fully migrated back to CE 2.7.0 because of the kernel age difference between 2.7.0 CE and the just releasted 2.7.1 CE. Some kernel functions used by the updated libpfctl library code are not present in the 2.7.0 CE kernel. It may turn out that the only fix for 2.7.0 CE users is to update to 2.7.1 CE after the fixed packages are present there. But this question is still being looked at by the Netgate developer team.

[fireodo]

@bmeeks OFF Topic:

Thanks for your effort and work chasing that annoying bug! (has to be said)

[NogBadTheBad]

@bmeeks Is this the new version that's just come out today, if it is its still dumping core:-

Nov 17 15:58:20 kernel pid 93766 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 17 15:58:19 suricata 92214 [634254] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 17 15:58:19 php 82334 [Suricata] Suricata START for LAN(igb0)...

[SteveITS]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

fix requires the publishing of a new libpfctl library package

I might be low on coffee but does this mean it would be distributed as part of the Suricata/Snort packages, and not a pfSense version update?

Thanks,

[bmeeks]

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks Is this the new version that's just come out today, if it is its still dumping core:-

Nov 17 15:58:20 kernel pid 93766 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 17 15:58:19 suricata 92214 [634254] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 17 15:58:19 php 82334 [Suricata] Suricata START for LAN(igb0)...

No, there is no new version yet. You are likely seeing the old code if you are on 23.09 Plus. There was a problem with the package builder for 23.09, so what was the "new" Suricata package for 2.7.0 and 2.7.1 CE users did not get into 23.09 at the same time. 23.09 continued with the old 7.0.0 package until probably this morning. They were still working on the 23.09 package builder server yesterday, but expected it to be working last night.

So, long story is you do not have the fixed package. In fact, it may not get posted until either later tonight, over the weekend, or potentially it might be Monday. Not sure right now.

[bmeeks]

@SteveITS said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

fix requires the publishing of a new libpfctl library package

I might be low on coffee but does this mean it would be distributed as part of the Suricata/Snort packages, and not a pfSense version update?

Thanks,

Yes, indirectly. Because libpfctl is now a build dependency for the Snort and Suricata binaries, when Snort or Suricata is rebuilt the updated libpfctl will be used.

At some point over the next four days (since this is a Friday), new versions of the Snort and Suricata packages will show up for 2.7.1 CE and 23.09 Plus users. Because we are spanning a weekend, things may pause over Saturday and Sunday and pick up on Monday. That will be Netgate's call. I am working now to get my changes in.

[computerhousecalls]

2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT

After update last week, snort status on interfaces keeps crashing so far every four to five mins. Snort version 4.1.6-13 is using legacy blocking with kill states enabled. At this point I decided to switch to Suricata 7.0.2 and still the same thing. Suricata would also crash about every five mins. So then I switched back to snort and disabled killstates. Then the service was not error 11 status. So as to the Original Note it is affecting both Snort and Suricata packages.