Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

[bmeeks]

Attention users of both Snort and Suricata on pfSense.

Updated Nov 17, 2023:
The bug has been identified and fixed. It was in the libpfctl library that is bundled with FreeBSD. A new version of the library (which exists as a package) will be built and made available. Packages built after that with a libpfctl dependency will be bundled with the fixed version. I have some updates coming for both Snort and Suricata in their custom blocking plugins. Those will also be posted later today. Users should look for new package versions for both Snort and Suricata over the next couple of days.

Updated Nov 20, 2023:
Afternoon update:
A method has been identified to get this fix out without requiring a pfSense point release. We've been able to force the Snort and Suricata binary modules to link with the libpfctl library package contained in Ports instead of using the native module bundled with pfSense. This means a package update can be deployed for just Snort and Suricata, and during installation the new libpfctl package will be downloaded and installed along with the rest of Snort or Suricata. This will put a "fixed" library on the firewall, and Snort and Suricata will then use the "fixed" library and ignore the other one.

Morning update:
My previous statement about the bug residing in the libpfctl library distributed with FreeBSD was not 100% correct. The libpfctl library in FreeBSD is distributed as a separate package, and binaries that need it link against that package version. It is different in pfSense. The libpfctl in pfSense is distributed as a native component of pfSense and not as a package. That means laying down a new version is not simple. It would require a new pfSense version. The Netgate team is looking for alternatives including forcing the Snort and Suriata packages to link with the net/libpfctl package version of libpfctl library instead of the built-in version bundled currently with pfSense.

This means the update containing the fix is not yet available for users to install. There seems to be some confusion with users because some are not seeing the bug manifest at all, some are seeing it very rarely, and others can't seem to run either package for more than a few minutes. This variability is caused by what the bug is and where it resides in the library's code. It takes a combination of conditions to happen concurrently for the bug to trigger. Those conditions do not always line up in the needed pattern on every user's firewall- thus the randomness of the bug and user experiences with it.

Updated Nov 21, 2023:
The fix for this issue in both the Snort and Suricata packages was merged overnight (relative to US Eastern Time) into the pfSense CE 2.7.1 branch and the pfSense Plus 23.09 branch.

The 2.7.1 CE updates are in place and available to users. There is a problem with package builds in the 23.09 branch that is unrelated to the Snort and Suricata fixes. So, the updated packages are not yet showing up for Plus 23.09 users. The changes are in place in that repo, but for unrelated reasons package building is failing there. The Netgate guys are working on it.

Updated Nov 22, 2023:
As of 7:00 PM US Eastern Time the updated Snort and Suricata packages for pfSense Plus 23.09 are not yet available. I had email communication early this morning from Netgate letting me know they were continuing to work on the 23.09 package builder problem. They had hoped to have it fixed today (Nov 22, 2023), but if that goal was not met it would be Monday, November 27th, before work would resume due to the long Thanksgiving Holiday weekend here in the US.

Updated Nov 22, 2023:
The new packages built and were successfully deployed for pfSense Plus 23.09 users overnight.

=====================================
Original Note is Below:
It appears a bug has crept into the Legacy Blocking Mode code used as a custom plugin compiled into both Snort and Suricata on pfSense. The bug manifests as the binary daemon crashing with a Signal 11 fault and producing a core dump.

This bug likely first appeared approximately two weeks ago (I think). I believe the bug is related to some updates done in the custom plugin code to adapt to recent kernel changes in the latest FreeBSD version used in pfSense Plus. The changes in FreeBSD were to parts of the pf firewall engine code and specifically were in the area around killing open firewall states. The custom plugin calls the kernel's "kill states" function when it blocks an IP address and the Kill States option is enabled on the INTERFACE SETTINGS tab of the package GUI. That option is enabled by default, as normally you want to immediately kill (or clear) any open states the firewall has for an offender's IP address that Snort or Suricata wishes to block.

It appears the recent changes may be at fault because turning off the Kill States option in the GUI allows Snort to run without faulting. But running with the option disabled is not ideal for the reason mentioned above.

I have opened Redmine Issue #14986 to track this problem for Snort. I strongly suspect Suricata has the same problem, but have not created a separate Redmine for it yet. Pending what we find for Snort, I will make sure Suricata is also updated if required. The Netgate developer team is aware and engaged on finding and fixing this problem. Their kernel developers are going to have a look for me. I will post additional updates as I have them.

Unfortunately, if you are experiencing this problem, your only option is to either not use Legacy Blocking Mode, or if using it, uncheck the Kill States option in the GUI on the INTERFACE SETTINGS tab. If you disable that option, you must restart Snort after saving the change in order for the binary to see the change. Because this problem is related to a change in the FreeBSD kernel, simply rolling back the Snort package version would not fix the problem. The new plugin code is required on the new FreeBSD version, and the bug appears to be in that required new plugin code.

Note for Suricata Users: the problem described above is NOT related to the potential Hyperscan library issue in Suricata 7.0.0. That is a completely separate problem that hopefully will be fixed in the forthcoming 7.0.2 update.

[NogBadTheBad]

@bmeeks I wonder if this is my issue with suricata in legacy mode dumping core.

If I disable blocking it works fine.

[Bob.Dig]

Suricata-only-User here, so this is only Plus and not 2.7.1.RC? I see no problems on CE RC with Kill States active, after I had to reinstall Suricata though.
I had running Suricata on the new Plus for a short time. But I changed (back) my Switch to a combo 1G/10G one and with that, Suricata manged to block my local-IPs (again). So it must have working than. But with that I uninstalled Suricata (again).

[bmeeks]

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks I wonder if this is my issue with suricata in legacy mode dumping core.

If I disable blocking it works fine.

More specifically, what if you disable just the Kill States option under the Legacy Blocking Mode section on the INTERFACE SETTINGS tab? Enable Legacy Mode Blocking but uncheck the Kill States option. Save the change and restart Suricata.

Do that for me as an experiement. I'm trying to isolate and/or confirm if the issue is where I suspect. It is not a condition you want to run in permanently, though, as killing any open states for an offender's IP address is desirable.

Since I do not have a pfSense Plus test environment, I have not yet been able to reproduce these Signal 11 faults with either Suriata or Snort. I am in email communication with the Netgate kernel developer, and he is looking at the code changes he made earlier. His suspicion is-- like mine-- that there is a lurking issue in the new changes related to libpfctl revisions in FreeBSD itself.

[bmeeks]

@Bob-Dig said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Suricata-only-User here, so this is only Plus and not 2.7.1.RC?

For now, I think that is correct. I have only 2.7.0 CE Release that is fully operable in my test setups, and I cannot seem to reproduce the problem there. I have not yet upgraded that environment to 2.7.1-RC.

The vast majority (if not all) of the current reports seem to be coming from Plus users. But I'm still trying to get an accurate inventory of operating systems associated with the Signal 11 faults.

[Bob.Dig]

@bmeeks I will reinstall Suricata again here on plus because I ditched that switch in the meantime (again), one probe more, I will report what I find.

[bmeeks]

@Bob-Dig said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks I will reinstall Suricata again here on plus because I ditched that switch in the meantime (again), one probe more, I will report what I find.

Just had a Snort user report he is seeing the Signal 11 fault and core dump on 2.7.0-RELEASE, so maybe it is more widespread. All the pfSense versions now have the same flavor of the custom blocking plugin, and nearly all of the code in that plugin is identical for both Snort and Suricata.

[fireodo]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

The vast majority (if not all) of the current reports seem to be coming from Plus users. But I'm still trying to get an accurate inventory of operating systems associated with the Signal 11 faults.

Hi, here pfsense CE 2.7.0 Snort 4.1.6_13 exit with signal 11 (coredumped).
When kill states are off - no crash.

[bmeeks]

@fireodo said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Hi, here pfsense CE 2.7.0 Snort 4.1.6_13 exit with signal 11 (coredumped).
When kill states are off - no crash.

Thanks. So it is more widespread. I will feed this back to the Netgate kernel developer. We definitely found one typo in the new code, but does not seem it should be causing the Signal 11 fault.

[NogBadTheBad]

@bmeeks Suricata works fine with the kill states unticked.

[bmeeks]

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks Suricata works fine with the kill states unticked.

Thanks for that report. It seems to further confirm the problem is solely within the new "kill states" changes. That is where we are currently focusing our efforts this morning.

[Bob.Dig]

I have a block here (Kill States enabled), everything seems to be good, so maybe Suricata isn't affected. I don't use any snort rules though.

Spoiler

[bmeeks]

@Bob-Dig said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

I have a block here (Kill States enabled), everything seems to be good, so maybe Suricata isn't affected. I don't use any snort rules though.

Give it some running time. The fault is not always instantaneous for other users. Some are seeing up to 30 minutes of runtime before a fault.

The specific rules (Snort VRT versus Emerging Threats or others) should not play any role here. The custom blocking plugin code is generally ignorant of the rule specifics. It is just concerned with pulling out the offending IP from the data passed by the main binary and then deciding whether to block the IP or not based on any configured pass list.

From evidence gathered thus far, the bug appears to need multiple factors to trigger. I say this because it does not seem to be reproducible with any degree of certainty.

[InstanceExtension]

@bmeeks Add me to the list of 2.7.0 CE (just downgraded from Plus on Oct 31st) using Snort legacy mode and seeing core dumps. Mine all seems to occur just after a rules update when Snort is restarted, Does not occur with each rules update and restart though. Has occurred 3 times since Nov 1st.

[Bob.Dig]

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Give it some running time. The fault is not always instantaneous for other users. Some are seeing up to 30 minutes of runtime before a fault.

Still nothing to report here for Suricata.

Spoiler

[NogBadTheBad]

@Bob-Dig You're running In-line mode aren't you ?

[Bob.Dig]

@NogBadTheBad No, legacy.

[bmeeks]

Update -- I was finally able to experience the Signal 11 core dump on my CE 2.7.0-RELEASE testing machine. It took about an hour for the event to trigger. Seemed to happen when it attempted to kill open states.

I am now building a debug-enabled version of Snort for further testing to see if I can trigger the fault again. Having a debug-enabled version will help track down what's happening. Since the fault happens randomly, this may take a bit to work out.

[Bob.Dig]

@bmeeks Just another Datapoint regarding Suricata. I let it run over two hours on my VPS with pfSense CE RC on WAN. Hundreds of blocks, still no problem.

[bmeeks]

@Bob-Dig said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks Just another Datapoint regarding Suricata. I let it run over two hours on my VPS with pfSense CE RC on WAN. Hundreds of blocks, still no problem.

Is this with Kill States checked or unchecked on the INTERFACE SETTINGS tab?