Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module

JonathanLee · Nov 19, 2023, 3:42 AM

@bmeeks thanks for the reply. I can assure you I still see it, however much less than the new version of Snort. I am running the .11 and it does seem very stable without the core crashes. As soon as I update the snort package it crashes every couple mins when adjusting supress lists. After I downgraded I have no more logs for core dumps. I am stable with the version before it.

I went back to the old version.
login-to-view

The errors in logs are from when I had the updated snort.

login-to-view

JonathanLee · Nov 20, 2023, 6:20 PM

Will the update work on 23.05.01??

bmeeks · Nov 20, 2023, 6:33 PM

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Will the update work on 23.05.01??

No. The kernel there does not contain some of the necessary functions. But there should not be a problem with the Snort version in 23.05.1. Everything there on the binary side is using the old code.

The problem surfaced when Snort was updated to cope with changes that appeared in 23.09 Plus and 2.7.1 CE.

SteveITS · Nov 20, 2023, 7:18 PM

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

Will the update work on 23.05.01??

If you're on 23.05 don't install a package from 23.09...change your update branch to Previous Stable per my sig.

slu · Nov 21, 2023, 10:55 AM

@bmeeks

I installed today 4.1.6_14 and there was also a libpfctl upgrade, do this version fix the issue?

[1/3] Fetching snort-2.9.20_7.pkg: .......... done
[2/3] Fetching pfSense-pkg-snort-4.1.6_14.pkg: .......... done
[3/3] Fetching libpfctl-0.8.pkg: . done
Checking integrity... done (0 conflicting)
[1/3] Installing libpfctl-0.8...
[1/3] Extracting libpfctl-0.8: ...... done
[2/3] Upgrading snort from 2.9.20_5 to 2.9.20_7...
[2/3] Extracting snort-2.9.20_7: .......... done
[3/3] Upgrading pfSense-pkg-snort from 4.1.6_13 to 4.1.6_14...

fireodo · Nov 21, 2023, 11:36 AM

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:
@bmeeks

I installed today 4.1.6_14 and there was also a libpfctl upgrade

Here too 4.1.6_14 (under pfsense 2.7.1) - no issues so far ... keep observing.

bmeeks · Nov 21, 2023, 1:16 PM

Yes, the 4.1.6_14 version of the Snort package should correct the Signal 11 crashing issue when Kill States is enabled with Legacy Mode Blocking. There is also an update for Suricata to 7.0.2_1 that fixes the same problem in Suricata's Legacy Blocking Mode.

The developer working with me on this fix resides in Europe, and he merged the fixes during his working hours- some of which are still during my bedtime here in the Eastern US. So, just now seeing the news.

slu · Nov 21, 2023, 2:21 PM

@bmeeks
so cool, thank you @bmeeks and Netgate.

Like the way to communicate here directly and open.

bmeeks · Nov 21, 2023, 2:28 PM

The 2.7.1 CE updates are in place and available to users. There is a problem with package builds in the 23.09 branch that is unrelated to the Snort and Suricata fixes. So, the updated packages are not yet showing up for Plus 23.09 users. The changes are in place in that repo, but for unrelated reasons package building is failing there. The Netgate guys are working on it.

bmeeks · Nov 21, 2023, 3:43 PM

@slu said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

[1/3] Installing libpfctl-0.8...
[1/3] Extracting libpfctl-0.8: ...... done

These two lines are the actual fix for the problem. The libpfctl library that comes bundled with pfSense natively is version 0.4 and it has the bug. That buggy library is still there for now, but packages that need to do libpfctl things have been recompiled and instructed to use the package version of that library now stored in /usr/local/lib/ instead of the system-bundled version in /usr/lib/.

In the next release of pfSense (whenever that happens), the bundled library will be removed and ports recompiled to use only the package version of libpfctl. This will make any future updates to the library easier. Packages that need libfpctl functionality will automatically install that library package if it is not already present-- or update the installed version if necessary.

JonathanLee · Nov 22, 2023, 1:16 AM

@bmeeks when will 23.09 plus users get the update? Anytime soon or should users go back the their old boot environments?

bmeeks · Nov 22, 2023, 1:22 AM

@JonathanLee said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@bmeeks when will 23.09 plus users get the update? Anytime soon or should users go back the their old boot environments?

Whenever the 23.09 package builder server successfully builds all of the packages and copies them over to the 23.09 repo web server. I have no inside information on what's wrong. All I was told is that no packages are succesfully building on that infrastructure. right now.

The 2.7.1 CE builders are working fine, so the Snort and Suricata updates are available there.

In today's world, with all the encrypted traffic on networks, I would not consider the IDS/IPS important enough to warrant rolling back to 23.05.1. I would simply disable the IDS/IPS until the package update becomes available for 23.09.

JonathanLee · Nov 22, 2023, 1:25 AM

@bmeeks thanks for the info. I personally see this package as a reason to roll back as it is a work horse for me. I'll have to roll back again.

computerhousecalls · Nov 22, 2023, 10:39 PM

bmeeks Thank you so much for all of your hard work. I am able to finally provide an update to signal 11 snort issue. Pfsense 2.7.1 with snort 4.1.6_14 appears to now be working correctly. I have pushed all updates and package updates and so far for over 40mins the service has been running. Thanks again I hope you have a happy thanksgiving too.

JonathanLee · Nov 22, 2023, 10:45 PM

bmeeks · Nov 23, 2023, 1:16 PM

It's 7:06 PM US Eastern Time, and I just checked with my SG-5100 running Plus 23.09 and the updated Snort and Suricata packages are still not available. I had an earlier email communication from Netgate advising they were continuing work to resolve the 23.09 package builder problems. Their hope was to get things resolved today. Apparently that did not have happened (unless the packages build overnight).
~~
Due the long Thanksgiving Holiday weekend here in the US, work on the problem will likely not resume until Monday, November 27th. pfSense Plus 23.09 users will have to be patient a little longer.

Update: the new packages were built overnight. Updates are available now on pfSense 23.09 for both Snort and Suricata. This update should correct the Signal 11 crash when using Legacy Mode Blocking with Kill States enabled.

It will NOT make any difference in Suricata if you are experiencing the HyperScan "Fatal: hyperscan returned error -1" problem.

ronv42 · Nov 23, 2023, 12:24 PM

I just checked this morning from my homelab's self-built protectli running 23.09 and there is a update. Happy Thanksgiving for those in the USA.

login-to-view

NogBadTheBad · Nov 23, 2023, 7:51 PM

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

bmeeks · Nov 24, 2023, 3:55 PM

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

I need some more troubleshooting info here to help me identify the issue.

If you turn off Legacy Mode does the problem go away?
If you leave Legacy Blockling Mode enabled but uncheck the Kill States option does the problem go away?
Are there any other errors in the suricata.log file?

NogBadTheBad · Nov 24, 2023, 7:29 PM

@bmeeks said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

@NogBadTheBad said in Important Snort and Suricata Package Announcement -- probable bug in Legacy Blocking Module:

It's still dumping core in Legacy mode:-

Nov 23 19:49:17 kernel pid 59990 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 23 19:49:15 suricata 58667 [100237] <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
Nov 23 19:49:15 php 48436 [Suricata] Suricata START for LAN(igb0)...
Nov 23 19:49:14 php 48436 [Suricata] Building new sid-msg.map file for LAN...

I need some more troubleshooting info here to help me identify the issue.

If you turn off Legacy Mode does the problem go away?

Yes

If you leave Legacy Blockling Mode enabled but uncheck the Kill States option does the problem go away?

Block Offenders On
Kill States Off

Core dumps

Are there any other errors in the suricata.log file?

Just deleted the interface and recreated it with all the defaults and no core dump.

Restart suricata from the GUI and a core dump.

From both interface logs:-

02312 - Suricata-Main] 2023-11-24 19:19:14 Info: logopenfile: http-log output device (regular) initialized: http.log
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: detect: 1 rule files processed. 370 rules successfully loaded, 0 rules failed
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: detect: 370 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 191 inspect application layer, 108 are decoder event only
[102312 - Suricata-Main] 2023-11-24 19:19:14 Info: runmodes: Using 1 live device(s).
[135891 - RX#01-pppoe0] 2023-11-24 19:19:14 Info: pcap: pppoe0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[135891 - RX#01-pppoe0] 2023-11-24 19:19:14 Info: pcap: pppoe0: snaplen set to 1518
[102312 - Suricata-Main] 2023-11-24 19:19:14 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[135891 - RX#01-pppoe0] 2023-11-24 19:20:15 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[135893 - W#02] 2023-11-24 19:20:39 Error: spm-hs: Hyperscan returned fatal error -1.

[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: threshold-config: Threshold config parsed: 43 rule(s) found
[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: detect: 370 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 191 inspect application layer, 108 are decoder event only
[212407 - Suricata-Main] 2023-11-24 19:17:21 Info: runmodes: Using 1 live device(s).
[135845 - RX#01-igb0] 2023-11-24 19:17:21 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[135845 - RX#01-igb0] 2023-11-24 19:17:21 Info: pcap: igb0: snaplen set to 1518
[212407 - Suricata-Main] 2023-11-24 19:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[135845 - RX#01-igb0] 2023-11-24 19:18:05 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[135848 - W#03] 2023-11-24 19:20:38 Error: spm-hs: Hyperscan returned fatal error -1.
[1099191 - W#04] 2023-11-24 19:20:38 Error: spm-hs: Hyperscan returned fatal error -1.

Looks like its Hyperscan causing the issue that you mentioned further up the thread.