Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

[slu]

@bmeeks

don't see such an issue on my two 2.7.0 CE boxes:
2.7.0 CE
Snort 4.1.6_13
Legacy Mode
Kill States enabled

[Gblenn]

I have Suricata running in Legacy mode at two different sites, (2.7.0 and 23.09). Both with Kill States enabled and no problems whatsoever...

[Bob.Dig]

@slu Do you have actually blocks?

[Gblenn]

@Bob-Dig Yes I do, and I have the Remove Blocked Hosts Interval set to 1h but could change it to something higher if that would have an effect?
Up until a few days ago I have had the 23.09 site set up with Block on DROP Only NOT being checked. Meaning I had every single Alert resulting in a Block and basically having built up a Passlist that works for me. As a result I would usually see quite a long list in the Blocks tab. Most of them ET INFO actually, but still blocks.
The 2.7.0 site is also set up this way, but it's for our vacation home so not much going on there at the moment.

[slu]

@Bob-Dig
good point, no, not at the moment.

[Gblenn]

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@Bob-Dig
good point, no, not at the moment.

Ah, that was a question to you obviously...

[slu]

@Bob-Dig said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@slu Do you have actually blocks?

You are right, searching in the logs and found it:

Nov 15 06:47:55 	kernel 		pid 44159 (snort), jid 0, uid 0: exited on signal 11 (core dumped)

[Bob.Dig]

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

You are right, searching in the logs and found it:

I didn't searched in the logs. I think, it will disable IPS on that interface and is easy to spot in the GUI? At least I hope.

[johnpitton]

I have a few Netgate 7100 units all running 23.09 with Snort VRT rules in legacy Blocking mode and Kill State enabled with 7 days blocking set.
They've been at this 23.09 ver for almost a week now and stable.
All of them have active blocks listed.
I checked and verified SNORT rulesets have been updating successfully.
I don't see in any of the logs any signal 11 core dumps logged.

[bmeeks]

@johnpitton said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

I have a few Netgate 7100 units all running 23.09 with Snort VRT rules in legacy Blocking mode and Kill State enabled with 7 days blocking set.
They've been at this 23.09 ver for almost a week now and stable.
All of them have active blocks listed.
I checked and verified SNORT rulesets have been updating successfully.
I don't see in any of the logs any signal 11 core dumps logged.

What version of the Snort package is installed on your box? Look under SYSTEM > PACKAGE MANAGER and post back the version shown there. There has been an issue with the package builder for 23.09, so my understanding is that some of the most recent package updates have not been deployed in that branch.

[johnpitton]

@bmeeks

[bmeeks]

@johnpitton said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks

Yes, that is the older version of the package. Current is now 4.1.6_13 for the GUI and 2.9.20_5 for the associated binary. The newer package is held up currently for 23.09 because of a different problem with the package builder backend. In your case, I would advise sticking with the version you have until we find and squash the bug introduced in the newest package.

Thank you for this data point. It is helpful in my troubleshooting to isolate the new bug. The code change which appears to have introduced the bug happened with the 2.9.20_4 version of the Snort binary.

[bmeeks]

I think I may have found the Signal 11 core dump bug. If I am correct, it's actually in the FreeBSD libpfctl library and not directly in the Snort code. I'm waiting on the Netgate kernel developer I'm working with to either confirm my finding or show me where I went off track analyzing the library code .

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

If I am correct, it's actually in the FreeBSD libpfctl library and not directly in the Snort code.

Just out of curiosity: that could be corrected by rebuilding and replacing that library or there would be necessary a new pfsense revision?

[yorke]

pfsense CE 2.7.0
Nov 17 06:59:00 kernel pid 25245 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

[bmeeks]

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Just out of curiosity: that could be corrected by rebuilding and replacing that library or there would be necessary a new pfsense revision?

No, I don't think that could work.

However, there is a way to workaround the problem by calling the older legacy functions for killing firewall states in my custom blocking module when the FreeBSD version running on the box is less than a particular value. That would workaround the faulty library until it could be updated with a kernel update.

Updated Info:
A new version of the libpfctl package will be built and added to pfSense 2.7.1 CE and 23.09 Plus builds. I'm not sure of all the logistics involved here, so won't speculate much more.

The root cause was an effort to migrate pfctl functionality over to a wrapper library instead of using direct kernel ioctl() calls. One of the things the new wrapper library (libpfctl) does is use the new nvlist functionality in the kernel. Here is the official man page for nvlist: https://man.freebsd.org/cgi/man.cgi?query=nvlist&apropos=0&sektion=0&manpath=FreeBSD+15.0-CURRENT&arch=default&format=html. As more pfctl functionality is shifted to the libpfctl library, a few of the legacy ioctl() calls are being removed. The upcoming new versions of pfSense (sorry, but I don't know which ones) will not be able to use the legacy ioctl() calls at all for some pfctl operations. That means my custom blocking plugin had to be updated to cope with that. Without realizing there was an issue in libpfctl, the latest Snort and Suricata packages were bundled with the newly modified custom plugin code because libpfctl has been available since at least FreeBSD 12. Since both CE and Plus are on a version of FreeBSD 14, it seemed like a no-brainer.

But after digging around there appears to be a couple of lingering bugs in libpfctl depending on which specific FreeBSD kernel is in use. For example, the libpfctl version bundled with pfSense CE 2.7.0 is slightly different from the version bundled with 23.09 Plus. The CE 2.7.0 version appears to have two bugs (one of which was fixed in the later libpfctl library version bundled with 23.09 Plus). That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well. The bug triggers somewhat randomly because a unique set of circumstances have to occur to trigger it.

The fix will almost certainly involve an update to my custom blocking plugin code used in Snort and Suricata. Then the libpfctl library will likely be updated in the next pfSense version release.

More Updated Info:
I have some further refinements/updates for the custom blocking modules in both Snort and Suricata. I will be posting new versions of both packages a bit later- after the updates for libpfctl have landed.

[bmeeks]

@yorke said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

pfsense CE 2.7.0
Nov 17 06:59:00 kernel pid 25245 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

Are you using Legacy Blocking Mode? And if so, is the Kill States option checked (enabled) on the INTERFACE SETTINGS tab?

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

[bmeeks]

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

I'm not sure of all the logistics of getting that updated library onto individual machines, though. For CE 2.7.1-RC users it could happen with a new incremental build of that version. I don't know how it will work for 23.09 Plus users or CE 2.7.0 users.

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

Ah, OK then I get back to the NO Kill States ... and wait :-)