Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

bmeeks

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Will the update work on 23.05.01??

No. The kernel there does not contain some of the necessary functions. But there should not be a problem with the Snort version in 23.05.1. Everything there on the binary side is using the old code.

The problem surfaced when Snort was updated to cope with changes that appeared in 23.09 Plus and 2.7.1 CE.

SteveITS

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Will the update work on 23.05.01??

If you're on 23.05 don't install a package from 23.09...change your update branch to Previous Stable per my sig.

slu

@bmeeks

I installed today 4.1.6_14 and there was also a libpfctl upgrade, do this version fix the issue?

[1/3] Fetching snort-2.9.20_7.pkg: .......... done
[2/3] Fetching pfSense-pkg-snort-4.1.6_14.pkg: .......... done
[3/3] Fetching libpfctl-0.8.pkg: . done
Checking integrity... done (0 conflicting)
[1/3] Installing libpfctl-0.8...
[1/3] Extracting libpfctl-0.8: ...... done
[2/3] Upgrading snort from 2.9.20_5 to 2.9.20_7...
[2/3] Extracting snort-2.9.20_7: .......... done
[3/3] Upgrading pfSense-pkg-snort from 4.1.6_13 to 4.1.6_14...

fireodo

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:
@bmeeks

I installed today 4.1.6_14 and there was also a libpfctl upgrade

Here too 4.1.6_14 (under pfsense 2.7.1) - no issues so far ... keep observing.

bmeeks

Yes, the 4.1.6_14 version of the Snort package should correct the Signal 11 crashing issue when Kill States is enabled with Legacy Mode Blocking. There is also an update for Suricata to 7.0.2_1 that fixes the same problem in Suricata's Legacy Blocking Mode.

The developer working with me on this fix resides in Europe, and he merged the fixes during his working hours- some of which are still during my bedtime here in the Eastern US. So, just now seeing the news.

slu

@bmeeks
so cool, thank you @bmeeks and Netgate.

Like the way to communicate here directly and open.

bmeeks

The 2.7.1 CE updates are in place and available to users. There is a problem with package builds in the 23.09 branch that is unrelated to the Snort and Suricata fixes. So, the updated packages are not yet showing up for Plus 23.09 users. The changes are in place in that repo, but for unrelated reasons package building is failing there. The Netgate guys are working on it.

bmeeks

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

[1/3] Installing libpfctl-0.8...
[1/3] Extracting libpfctl-0.8: ...... done

These two lines are the actual fix for the problem. The libpfctl library that comes bundled with pfSense natively is version 0.4 and it has the bug. That buggy library is still there for now, but packages that need to do libpfctl things have been recompiled and instructed to use the package version of that library now stored in /usr/local/lib/ instead of the system-bundled version in /usr/lib/.

In the next release of pfSense (whenever that happens), the bundled library will be removed and ports recompiled to use only the package version of libpfctl. This will make any future updates to the library easier. Packages that need libfpctl functionality will automatically install that library package if it is not already present-- or update the installed version if necessary.

JonathanLee

@bmeeks when will 23.09 plus users get the update? Anytime soon or should users go back the their old boot environments?

bmeeks

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks when will 23.09 plus users get the update? Anytime soon or should users go back the their old boot environments?

Whenever the 23.09 package builder server successfully builds all of the packages and copies them over to the 23.09 repo web server. I have no inside information on what's wrong. All I was told is that no packages are succesfully building on that infrastructure. right now.

The 2.7.1 CE builders are working fine, so the Snort and Suricata updates are available there.

In today's world, with all the encrypted traffic on networks, I would not consider the IDS/IPS important enough to warrant rolling back to 23.05.1. I would simply disable the IDS/IPS until the package update becomes available for 23.09.

JonathanLee

@bmeeks thanks for the info. I personally see this package as a reason to roll back as it is a work horse for me. I'll have to roll back again.

computerhousecalls

bmeeks Thank you so much for all of your hard work. I am able to finally provide an update to signal 11 snort issue. Pfsense 2.7.1 with snort 4.1.6_14 appears to now be working correctly. I have pushed all updates and package updates and so far for over 40mins the service has been running. Thanks again I hope you have a happy thanksgiving too.

JonathanLee

bmeeks

It's 7:06 PM US Eastern Time, and I just checked with my SG-5100 running Plus 23.09 and the updated Snort and Suricata packages are still not available. I had an earlier email communication from Netgate advising they were continuing work to resolve the 23.09 package builder problems. Their hope was to get things resolved today. Apparently that did not have happened (unless the packages build overnight).
~~
Due the long Thanksgiving Holiday weekend here in the US, work on the problem will likely not resume until Monday, November 27th. pfSense Plus 23.09 users will have to be patient a little longer.

Update: the new packages were built overnight. Updates are available now on pfSense 23.09 for both Snort and Suricata. This update should correct the Signal 11 crash when using Legacy Mode Blocking with Kill States enabled.

It will NOT make any difference in Suricata if you are experiencing the HyperScan "Fatal: hyperscan returned error -1" problem.

ronv42

I just checked this morning from my homelab's self-built protectli running 23.09 and there is a update. Happy Thanksgiving for those in the USA.

NogBadTheBad

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

bmeeks

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

I need some more troubleshooting info here to help me identify the issue.

If you turn off Legacy Mode does the problem go away?
If you leave Legacy Blockling Mode enabled but uncheck the Kill States option does the problem go away?
Are there any other errors in the suricata.log file?

NogBadTheBad

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

I need some more troubleshooting info here to help me identify the issue.

If you turn off Legacy Mode does the problem go away?

Yes

If you leave Legacy Blockling Mode enabled but uncheck the Kill States option does the problem go away?

Block Offenders On
Kill States Off

Core dumps

Are there any other errors in the suricata.log file?

Just deleted the interface and recreated it with all the defaults and no core dump.

Restart suricata from the GUI and a core dump.

From both interface logs:-

02312 - Suricata-Main] 2023-11-24 19:19:14 Info: logopenfile: http-log output device (regular) initialized: http.log
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: detect: 1 rule files processed. 370 rules successfully loaded, 0 rules failed
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: detect: 370 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 191 inspect application layer, 108 are decoder event only
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: runmodes: Using 1 live device(s).
[135891 - RX#01-pppoe0] 2023-11-24 19:19:14 Info: pcap: pppoe0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[135891 - RX#01-pppoe0] 2023-11-24 19:19:14 Info: pcap: pppoe0: snaplen set to 1518
[102312 - Suricata-Main] 2023-11-24 19:19:14 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[135891 - RX#01-pppoe0] 2023-11-24 19:20:15 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[135893 - W#02] 2023-11-24 19:20:39 Error: spm-hs: Hyperscan returned fatal error -1.

[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: threshold-config: Threshold config parsed: 43 rule(s) found
[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: detect: 370 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 191 inspect application layer, 108 are decoder event only
[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: runmodes: Using 1 live device(s).
[135845 - RX#01-igb0] 2023-11-24 19:17:21 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[135845 - RX#01-igb0] 2023-11-24 19:17:21 Info: pcap: igb0: snaplen set to 1518
[212407 - Suricata-Main] 2023-11-24 19:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[135845 - RX#01-igb0] 2023-11-24 19:18:05 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[135848 - W#03] 2023-11-24 19:20:38 Error: spm-hs: Hyperscan returned fatal error -1.
[1099191 - W#04] 2023-11-24 19:20:38 Error: spm-hs: Hyperscan returned fatal error -1.

Looks like its Hyperscan causing the issue that you mentioned further up the thread.

bmeeks

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Looks like its Hyperscan causing the issue.

Agreed, but the additional details are what I needed to verify.

It's important for Suricata users to understand there are two likely completely unrelated bugs impacting them. One was the Legacy Blocking Mode bug that also affected Snort. That bug is fixed as of version 7.0.2_1 of the Suricata package. It was nailed down to a problem in the libpfctl library distributed with pfSense.

The potential HyperScan bug (that is still just a theoretical cause, but circumstantial evidence strongly points there) is a totally different thing unrelated to Legacy Blocking Mode. But users have intermixed the two bug reports in this thread about the common Legacy Blocking Mode problem that Snort and Suricata shared.

So, that's why I asked the clarifying questions about Legacy Blocking Mode and turning off the Kill States option. I needed to verify which bug you were likely still seeing. I was pretty sure it was not the Legacy Blocking Mode bug, but just needed confirmation from you.

If other Suricata users read this thread and find this reply, please determine which bug is currently impacting you: (1) the Legacy Blocking Mode bug, which so far as we know at this point is fixed; or (2) the HyperScan related bug which is not positively identified as reproducible and also not verified as fixed. You can immediately rule out anything related to Legacy Blocking Mode by simply turning off blocking and seeing if Suricata still experiences the crash. If it still crashes with Legacy Blocking Mode disabled, then the bug reported in this thread is not what you are experiencing.

bmeeks

Locking this thread for now as I am highly confident the original bug discussed here was fixed as of version 4.1.6_14 of the Snort package and version 7.0.2_1 of the Suricata package.

If you are having Signal 11 or Signal 10 crashes with Suricata, please report those in this thread instead: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem.