Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

bmeeks

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Just out of curiosity: that could be corrected by rebuilding and replacing that library or there would be necessary a new pfsense revision?

~~No, I don't think that could work.~~

However, there is a way to workaround the problem by calling the older legacy functions for killing firewall states in my custom blocking module when the FreeBSD version running on the box is less than a particular value. That would workaround the faulty library until it could be updated with a kernel update.

Updated Info:
A new version of the libpfctl package will be built and added to pfSense 2.7.1 CE and 23.09 Plus builds. I'm not sure of all the logistics involved here, so won't speculate much more.

The root cause was an effort to migrate pfctl functionality over to a wrapper library instead of using direct kernel ioctl() calls. One of the things the new wrapper library (libpfctl) does is use the new nvlist functionality in the kernel. Here is the official man page for nvlist: https://man.freebsd.org/cgi/man.cgi?query=nvlist&apropos=0&sektion=0&manpath=FreeBSD+15.0-CURRENT&arch=default&format=html. As more pfctl functionality is shifted to the libpfctl library, a few of the legacy ioctl() calls are being removed. The upcoming new versions of pfSense (sorry, but I don't know which ones) will not be able to use the legacy ioctl() calls at all for some pfctl operations. That means my custom blocking plugin had to be updated to cope with that. Without realizing there was an issue in libpfctl, the latest Snort and Suricata packages were bundled with the newly modified custom plugin code because libpfctl has been available since at least FreeBSD 12. Since both CE and Plus are on a version of FreeBSD 14, it seemed like a no-brainer.

But after digging around there appears to be a couple of lingering bugs in libpfctl depending on which specific FreeBSD kernel is in use. For example, the libpfctl version bundled with pfSense CE 2.7.0 is slightly different from the version bundled with 23.09 Plus. The CE 2.7.0 version appears to have two bugs (one of which was fixed in the later libpfctl library version bundled with 23.09 Plus). That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well. The bug triggers somewhat randomly because a unique set of circumstances have to occur to trigger it.

~~The fix will almost certainly involve an update to my custom blocking plugin code used in Snort and Suricata. Then the libpfctl library will likely be updated in the next pfSense version release.~~

More Updated Info:
I have some further refinements/updates for the custom blocking modules in both Snort and Suricata. I will be posting new versions of both packages a bit later- after the updates for libpfctl have landed.

bmeeks

@yorke said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

pfsense CE 2.7.0
Nov 17 06:59:00 kernel pid 25245 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

Are you using Legacy Blocking Mode? And if so, is the Kill States option checked (enabled) on the INTERFACE SETTINGS tab?

fireodo

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

bmeeks

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

That's why the Snort and Suricata Signal 11 issue appears most acute for CE 2.7.0 users. But I think it is impacting 23.09 users as well.

I updated today to 2.7.1 CE and reading your explanations I reactivated the KILL STATES on the busy interface and will report.

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

I'm not sure of all the logistics of getting that updated library onto individual machines, though. For CE 2.7.1-RC users it could happen with a new incremental build of that version. I don't know how it will work for 23.09 Plus users or CE 2.7.0 users.

fireodo

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

You will likely still experience the bug in 2.7.1 CE at the moment, although the kernel developer is about to commit the fix to libpfctl there shortly. At some point after that, a new libpfctl package will build for 2.7.1 CE and 23.09 Plus.

Ah, OK then I get back to the NO Kill States ... and wait :-)

SteveITS

@bmeeks 2.7.1 released yesterday actually…didn’t notice any posts here but it’s on Netgate’s blog.

Perhaps this and the URL Alias bug will be sufficient to quickly produce a point release.

bmeeks

@SteveITS said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

2.7.1 released yesterday actually…didn’t notice any posts here but it’s on Netgate’s blog.

I've been buried head down chasing the IDS/IPS package bugs and have not checked the release status.

I do know the kernel developer and author of all the recent libpfctl library changes is working to post the fixes and get them into at least 2.7.1 and 23.09. That is all happening this morning.

bmeeks

I've updated the original post at the top of this thread with new information.

I'll briefly repeat that here --

A fix has been identified and implemented for the bug exposed by the custom blocking module changes in the most recent Snort and Suricata package updates. The fix requires the publishing of a new libpfctl library package and then rebuilding impacted packages where libpfctl is a build dependency. That will happen soon in the pfSense CE 2.7.1 and pfSense Plus 23.09 branches.

Not sure at this point if the fix can be fully migrated back to CE 2.7.0 because of the kernel age difference between 2.7.0 CE and the just releasted 2.7.1 CE. Some kernel functions used by the updated libpfctl library code are not present in the 2.7.0 CE kernel. It may turn out that the only fix for 2.7.0 CE users is to update to 2.7.1 CE after the fixed packages are present there. But this question is still being looked at by the Netgate developer team.

fireodo

@bmeeks OFF Topic:

Thanks for your effort and work chasing that annoying bug! (has to be said)

NogBadTheBad

@bmeeks Is this the new version that's just come out today, if it is its still dumping core:-

Nov 17 15:58:20 kernel pid 93766 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 17 15:58:19 suricata 92214 [634254] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 17 15:58:19 php 82334 [Suricata] Suricata START for LAN(igb0)...

SteveITS

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

fix requires the publishing of a new libpfctl library package

I might be low on coffee but does this mean it would be distributed as part of the Suricata/Snort packages, and not a pfSense version update?

Thanks,

bmeeks

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks Is this the new version that's just come out today, if it is its still dumping core:-

Nov 17 15:58:20 kernel pid 93766 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 17 15:58:19 suricata 92214 [634254] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 17 15:58:19 php 82334 [Suricata] Suricata START for LAN(igb0)...

No, there is no new version yet. You are likely seeing the old code if you are on 23.09 Plus. There was a problem with the package builder for 23.09, so what was the "new" Suricata package for 2.7.0 and 2.7.1 CE users did not get into 23.09 at the same time. 23.09 continued with the old 7.0.0 package until probably this morning. They were still working on the 23.09 package builder server yesterday, but expected it to be working last night.

So, long story is you do not have the fixed package. In fact, it may not get posted until either later tonight, over the weekend, or potentially it might be Monday. Not sure right now.

bmeeks

@SteveITS said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

fix requires the publishing of a new libpfctl library package

I might be low on coffee but does this mean it would be distributed as part of the Suricata/Snort packages, and not a pfSense version update?

Thanks,

Yes, indirectly. Because libpfctl is now a build dependency for the Snort and Suricata binaries, when Snort or Suricata is rebuilt the updated libpfctl will be used.

At some point over the next four days (since this is a Friday), new versions of the Snort and Suricata packages will show up for 2.7.1 CE and 23.09 Plus users. Because we are spanning a weekend, things may pause over Saturday and Sunday and pick up on Monday. That will be Netgate's call. I am working now to get my changes in.

computerhousecalls

2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT

After update last week, snort status on interfaces keeps crashing so far every four to five mins. Snort version 4.1.6-13 is using legacy blocking with kill states enabled. At this point I decided to switch to Suricata 7.0.2 and still the same thing. Suricata would also crash about every five mins. So then I switched back to snort and disabled killstates. Then the service was not error 11 status. So as to the Original Note it is affecting both Snort and Suricata packages.

bmeeks

@computerhousecalls said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT

After update last week, snort status on interfaces keeps crashing so far every four to five mins. Snort version 4.1.6-13 is using legacy blocking with kill states enabled. At this point I decided to switch to Suricata 7.0.2 and still the same thing. Suricata would also crash about every five mins. So then I switched back to snort and disabled killstates. Then the service was not error 11 status. So as to the Original Note it is affecting both Snort and Suricata packages.

Yes, both packages are impacted when using Legacy Blocking Mode. A fix has been indentified and is on the way. Just taking a little time for final extra testing and packaging things up.

computerhousecalls

@bmeeks Thank you & Thank you again.

JonathanLee

I have 23.05.01 and with the old version of snort it works fine. When I stay at 23.05.01 and update snort the core dump bug starts.
I run a sg-2100max. I sure you already have that info. I just didn't know if anyone has used it on 23.05.01 yet

Gerard64

@bmeeks

I upgrade to 2.7.1 this morning and Snort didn't stop anymore not once.
So seams all is good again.

Thank you man you are the best

bmeeks

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

I have 23.05.01 and with the old version of snort it works fine. When I stay at 23.05.01 and update snort the core dump bug starts.
I run a sg-2100max. I sure you already have that info. I just didn't know if anyone has used it on 23.05.01 yet

Anything earlier than 23.09 will have the defective libpfctl library version on it. If you update to a Snort package version after 4.1.6_9 you will hit the bug, because beginning with 4.1.6_11 the updated custom blocking module code that calls functions in the defective libpfctl library was included. Same thing applies to pfSense CE.

JonathanLee

@bmeeks thanks for the reply. I can assure you I still see it, however much less than the new version of Snort. I am running the .11 and it does seem very stable without the core crashes. As soon as I update the snort package it crashes every couple mins when adjusting supress lists. After I downgraded I have no more logs for core dumps. I am stable with the version before it.

I went back to the old version.

The errors in logs are from when I had the updated snort.