After upgrade to pf+ 23.09 Surricata says it's starting but..
-
Running on a Dell PowerEdge R350 with IX Intel 10-Gigabit X540-AT2 nics with the following settings:
- Hardware Checksum Offloading: Enabled
- Hardware TCP Segmentation Offloading: Disabled
- Hardware Large Receive Offloading: Disabled
Things were running fine until the 23.09 update. I tried the above remedies without success. Full uninstall, reboot, reinstall (making sure config was not saved). It definitely happens as soon as you enable blocking. We're using legacy mode and only block SRC ip's on a couple of specific LAN segments with only emerging-p2p.rules enabled. Here's a copy of the brief suricata.log showing the error at the end.
[100377 - Suricata-Main] 2023-11-14 11:06:51 Notice: suricata: This is Suricata version 7.0.0 RELEASE running in SYSTEM mode [100377 - Suricata-Main] 2023-11-14 11:06:51 Info: cpu: CPUs/cores online: 8 [100377 - Suricata-Main] 2023-11-14 11:06:51 Info: suricata: Setting engine mode to IDS mode by default [100377 - Suricata-Main] 2023-11-14 11:06:51 Info: app-layer-htp-mem: HTTP memcap: 67108864 [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Creating automatic firewall interface IP address Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:755a to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface bge0 IPv6 address fe80:0000:0000:0000:c65a:b1ff:fea8:aebb to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface bge0 IPv4 address 10.0.0.33 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.653 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.653 IPv4 address 172.16.48.1 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.897 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:755a to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.897 IPv4 address 12.146.29.52 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.895 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:755a to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.895 IPv4 address 12.220.13.22 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.896 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:755a to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.896 IPv4 address 12.220.13.39 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.892 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:755a to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix1.892 IPv4 address 0.0.0.0 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.651 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.401 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.402 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.600 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.647 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.647 IPv4 address 172.16.64.1 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.648 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.650 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.650 IPv4 address 172.16.128.1 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.652 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.654 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.660 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.661 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.700 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Adding firewall interface ix0.711 IPv6 address fe80:0000:0000:0000:a236:9fff:fe7c:7558 to automatic interface IP Pass List. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: logopenfile: alert-pf output device (regular) initialized: block.log [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_40419_ix0.650/passlist. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_40419_ix0.650/passlist processed: Total entries parsed: 9, IP addresses/netblocks/aliases added to No Block list: 8, IP addresses/netblocks ignored because they were covered by existing entries: 1. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=src kill-state=yes block-drops-only=no passlist-debugging=no [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: logopenfile: fast output device (regular) initialized: alerts.log [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: alert-syslog: Syslog output initialized [107064 - ] 2023-11-14 11:06:51 Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started. [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: detect: 1 rule files processed. 119 rules successfully loaded, 0 rules failed [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: threshold-config: Threshold config parsed: 0 rule(s) found [100428 - Suricata-Main] 2023-11-14 11:06:51 Info: detect: 119 signatures processed. 0 are IP-only rules, 81 are inspecting packet payload, 37 inspect application layer, 0 are decoder event only [100428 - Suricata-Main] 2023-11-14 11:06:52 Info: runmodes: Using 1 live device(s). [107065 - RX#01-ix0.650] 2023-11-14 11:06:52 Info: pcap: ix0.650: running in 'auto' checksum mode. Detection of interface state will require 1000 packets [107065 - RX#01-ix0.650] 2023-11-14 11:06:52 Info: pcap: ix0.650: snaplen set to 1518 [100428 - Suricata-Main] 2023-11-14 11:06:52 Notice: threads: Threads created -> RX: 1 W: 8 FM: 1 FR: 1 Engine started. [107065 - RX#01-ix0.650] 2023-11-14 11:06:52 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used [107067 - W#02] 2023-11-14 11:08:39 Error: spm-hs: Hyperscan returned fatal error -1.You can see that it runs for just a couple of minutes. A suricata.core file does get produced in the /root/ dir. It eventually fails even if I strip it down to a single LAN.
Curious if someone could post the XML of config export from a known working instance. Perhaps there is a specific setting that needs to flip?
Thanks
-
@PalisadesTahoe:
You are hitting a Hyperscan library error according to this entry from the log you posted:[107067 - W#02] 2023-11-14 11:08:39 Error: spm-hs: Hyperscan returned fatal error -1.What version of the Hyperscan library is installed? It should be 5.4.0 if I recall correctly. You can check what's on your system with this command at a shell prompt:
pkg info | grep hyperscanIf you have altered the Pattern Matching algorithm, try setting it back to "Auto".
-
@bmeeks Thank for taking a look. Pattern Matching is set to Auto and is indeed at version 5.4.0:
[23.09-RELEASE][admin@SRV-pfSense-PT]/root: pkg info | grep hyperscan hyperscan-5.4.0 High-performance multiple regex matching library -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Thank for taking a look. Pattern Matching is set to Auto and is indeed at version 5.4.0:
[23.09-RELEASE][admin@SRV-pfSense-PT]/root: pkg info | grep hyperscan hyperscan-5.4.0 High-performance multiple regex matching libraryI also see you are still running version 7.0.0 of the binary. There is, or will soon be posted, a new 7.0.2 version of the Suricata package. Update to that as soon as it shows up for you.
-
@PalisadesTahoe:
Check out this bug report on the upstream Suricata Redmine: https://redmine.openinfosecfoundation.org/issues/6195. There is a new 7.0.2 version of the Suricata package that should build tonight and show up. That package contains the latest 7.0.2 binary from upstream which contains a fix for the Hyperscan error.Currently you are running the 7.0.0 binary.
-
@bmeeks
I'll give it a go once it shows up, but looking at the bug, it seems related to dealing with a change in a rule pattern after an update which shouldn't be happening to us. But perhaps there are some other bug fixes in 7.0.2 that might remedy this issue. I'll report back after the update is available and applied. Thanks again. Cheers. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
I'll give it a go once it shows up, but looking at the bug, it seems related to dealing with a change in a rule pattern after an update which shouldn't be happening to us. But perhaps there are some other bug fixes in 7.0.2 that might remedy this issue. I'll report back after the update is available and applied. Thanks again. Cheers.
Since you are using only the ET-Open P2P rules, I can try to see if this is reproducible for me in a virtual machine. Unfortunately I don't have a pfSense Plus test environment. I only have 2.7.0 CE and 2.8.0 CE Development virtual environments to test with.
There was a change made in FreeBSD itself back in August that altered how the kill states system function call works. The Legacy Blocking Code attempts to perform that system call as part of killing the states associated with any IP it adds to the blocking table, snort2c. As an experiment, you could try turning off the Kill States option for the interfaces under the INTERFACE SETTINGS tab. That would bypass calling that modified kernel code. You would need to restart Suricata on the interface after changing and saving that setting. If it works then, that gives me a strong hint of where to concentrate troubleshooting.
The consequence of disabling that option is any active states for a blocked IP will remain open and traffic would continue to flow. So, until the open states timed out, that would essentially convert the IPS to more IDS.
-
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks.No, there is nothing much to enable debugging wise unless the Suricata binary daemon is compiled with debugging enabled, and we do not do that on the production builds.
Let's see if the 7.0.2 binary makes any difference with respect to the Hyperscan error.
I really thing the Signal 11 core dump is related to killing states in the Legacy Blocking Module. Seeing that in Snort now, and it has the same code. There was a change in FreeBSD in the kill states kernel system call. A Netgate developer made some changes in the Legacy Blocking Module code to sync up with the FreeBSD changes, but I'm thinking those changes might have introduced a bug.
-
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks.Thank you for this feedback. This bit of info (using AC or AC-BS versus Hyperscan for the pattern matcher setting) probably needs to go back upstream. Might be something particular to FreeBSD, though. For example, I believe there is now a 5.4.1 hyperscan library version out there for other distros, but FreeBSD has (or at least had the last time I checked) only 5.4.0 in FreeBSD-ports.
-
@PalisadesTahoe Thanks for posting your detailed troubleshooting. I just tried AC in my environment and I am also having success getting blocking mode enabled again.
-
@xpxp2002 Spoke too soon. Came back to find my box running 80% CPU (four cores) and 100% mem usage and about 50% swap usage. Stopped all of the Suricata processes, and CPU and memory started recovering right away.
I have yet to see the Suricata 7.0.2 package show up in Package Manager. For now, I'm just going to run without Suricata. It seems to be ok with blocking mode disabled, but it's just too risky to keep trying these types of changes with this version and potentially have the system run out of memory and drop offline.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
I have yet to see the Suricata 7.0.2 package show up in Package Manager.
What version of pfSense are you running? I expected the new binary to be deployed to all pfSense versions by now.
Also, just to note, one of the issues with some of the non-hyperscan pattern matchers is they can become huge memory hogs. Same thing happens in Snort. The default choice is always the best choice. But I know in this case it will choose Hyperscan (as that library is available and the best and fastest) and thus trigger the Hyperscan bug.
-
@bmeeks 23.09
-
I've just sent an email to my Netgate developer contact asking that the 7.0.2 package be deployed to the pfSense Plus 23.09 branch. It was supposed to have already been deployed, so I'm not sure what held it up. Maybe some confusion around the potential Kill States bug. But that bug has no relation to the Hyperscan issue. Those are two separate bugs with separate fixes. The Kill States bug has still not been fully identified, but when it is a separate package update will be issued containing that fix.
-
Got a reply from Netgate. They are currently having issues with the pfSense Plus 23.09 package builder infrastructure. That has held up building and deploying several other package updates. Resolution is in progress, and hopefully everything is back on track later today.
Until that issue is resolved, the Suricata 7.0.2 update can't get built for 23.09
. -
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
Yes, the 7.0.2 package should correct the Hyperscan fatal exit error. Here is the upstream commit (change) that fixed it: https://github.com/OISF/suricata/commit/00e00254eae205bad5d4cfbf6c9e69f944faaf69. It was fixed starting with Suricata 7.0.1 from upstream. Currently you have 7.0.0, but the update that is held up from building at the moment is 7.0.2.
This post of mine in another thread explains the Hyperscan error and the fix: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem/15?_=1700148686823.
-
@bmeeks Where to find HyperScan? Is it a kind of rule or tool used by some rules?
