Suricata process dying due to hyperscan problem

asdjklfjkdslfdsaklj

@bmeeks anecdotally, yes. ~8hrs after disabling blocking mode and both LAN PHY Suricata instances are still up.

bmeeks

@asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

@bmeeks anecdotally, yes. ~8hrs after disabling blocking mode and both LAN PHY Suricata instances are still up.

I need to know if you have any VLAN configured on either LAN interface.
Try enabling Blocking Mode on just one of the LAN interfaces and see what happens then.

To help me troubleshoot this, I desparately need you folks having the issue to give me some explicit details when responding. For example, answer question #1 above and also try troubleshooting suggestion #2 above. Then follow up back here with detailed results for each.

I will repeat again for clarity: I am trying to determine if VLANs configured on the crashing interface are related or not. So, tell me if you have VLANs on the interface, and if you do, how many. Then tell me if you can relate the crash to blocking enabled or not.

jowe78

It was a little early to sound the all-clear...

The suricata interfaces died yesterday in the evening, seems to work better, but not working.

However something that is interesting is that I changed to Snort with 7 interfaces (AC-BNFA-NQ), all in blocking mode, and all was up and running in the morning, but ovpn went down and i can't see anything in the log related.

Nov 29 05:21:00 php-cgi 23 servicewatchdog_cron.php: Service Watchdog detected service openvpn stopped. Restarting openvpn (OpenVPN server:)
Nov 29 05:20:48 kernel ovpns1: link state changed to DOWN
Nov 29 05:20:48 kernel pid 35910 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
Nov 29 05:19:00 sshguard 17586 Now monitoring attacks.
Nov 29 05:19:00 sshguard 42196 Exiting on signal.

tylerevers

@bmeeks said in Suricata process dying due to hyperscan problem:

For other users experiencing the Hyperscan crash in Suricata --

Do you have one or more VLANs configured on the interface that crashes?

Does disabling blocking mode on the crashing interface result in a difference in behavior?

If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

Let me help you help the community, kind sir.

Environment

pfSense+ Plus 23.09-RELEASE
suricata 7.0.2_1
Dedicated Bare Metal pfSense+ Plus 23.09-RELEASE box acting as RoaS (Router-on-a-Stick): Xeon E5-1650 v0 @ 3.20 GHz; 40GB DDR3 ECC REG BUF; 120GB SSD boot drive
Intel X520-DA2 with both SFP+ ports connected via LAGG to UniFi USW Pro 48 PoE on Ports 51-52 Aggregate
I have 11 VLANs traversing the LAGG (VLAN 10 to 110 in increments of 10), but only 3 VLANs (30, 50, 60) do I have setup within Suricata.

Answers to Your Questions

Do you have one or more VLANs configured on the interface that crashes?
As detailed above, I run Suricata on 3 out of the 11 VLANs I have in total. Each of these 3 VLANs have their own Interfaces (of course) within Suricata. Only one of these Suricata Interfaces (VLAN 30) is crashing when using Hyperscan, but has run just fine for more than a week or two with AC-KS.

Does disabling blocking mode on the crashing interface result in a difference in behavior?
Please help me understand where this toggle is located as I do not see an entry that says "Blocking Mode" within the GUI. I will test once I understand your request.

Additional Observations

Of the 3 Suricata Interfaces, all had Signature Group Header MPM Context set to Full. For the failing Suricata Interface (VLAN 30), I have set this to Auto and I have returned the Patter Matcher Algorithm to Auto. I will follow-up as soon as I see the Suricata Interface fail.

jowe78

@tylerevers said in Suricata process dying due to hyperscan problem:

Please help me understand where this toggle is located

Under the Suricata interface you have a box. "Block Offenders". Uncheck and you wont block, just monitor.

tylerevers

@jowe78 said in Suricata process dying due to hyperscan problem:

@tylerevers said in Suricata process dying due to hyperscan problem:

Please help me understand where this toggle is located

Under the Suricata interface you have a box. "Block Offenders". Uncheck and you wont block, just monitor.

Thank you. I will wait until I see the Suricata Interface fail with Block Offenders checked and then I shall try with it unchecked.

bmeeks

@tylerevers said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:

For other users experiencing the Hyperscan crash in Suricata --

Do you have one or more VLANs configured on the interface that crashes?

Does disabling blocking mode on the crashing interface result in a difference in behavior?

If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

Let me help you help the community, kind sir.

Environment

pfSense+ Plus 23.09-RELEASE

suricata 7.0.2_1

Dedicated Bare Metal pfSense+ Plus 23.09-RELEASE box acting as RoaS (Router-on-a-Stick): Xeon E5-1650 v0 @ 3.20 GHz; 40GB DDR3 ECC REG BUF; 120GB SSD boot drive

Intel X520-DA2 with both SFP+ ports connected via LAGG to UniFi USW Pro 48 PoE on Ports 51-52 Aggregate

I have 11 VLANs traversing the LAGG (VLAN 10 to 110 in increments of 10), but only 3 VLANs (30, 50, 60) do I have setup within Suricata.

Answers to Your Questions

Do you have one or more VLANs configured on the interface that crashes?
As detailed above, I run Suricata on 3 out of the 11 VLANs I have in total. Each of these 3 VLANs have their own Interfaces (of course) within Suricata. Only one of these Suricata Interfaces (VLAN 30) is crashing when using Hyperscan, but has run just fine for more than a week or two with AC-KS.

Does disabling blocking mode on the crashing interface result in a difference in behavior?
Please help me understand where this toggle is located as I do not see an entry that says "Blocking Mode" within the GUI. I will test once I understand your request.

Additional Observations

Of the 3 Suricata Interfaces, all had Signature Group Header MPM Context set to Full. For the failing Suricata Interface (VLAN 30), I have set this to Auto and I have returned the Patter Matcher Algorithm to Auto. I will follow-up as soon as I see the Suricata Interface fail.

Thank for the detailed reply.

The "Block Mode" toggle is my generic name for the setting on the INTERFACE SETTINGS tab when you can enable or disable blocking. The setting is in the Alert and Block Settings section of the page. The checkbox is called Block Offenders. Unchecking that box removes all future blocking of offender IP addresses (it will not clear any currently existing blocks). There are also two settings for blocking offenders. One uses the netmap kernel device to implement a true inline-IPS mode of operation. But netmap will not work with VLANs or LAGG interfaces at the moment. You would need to run it on just the parent physical interface. Legacy Mode Blocking uses a custom output plugin compiled into the Suricata binary used on pfSense. This plugin calls a pfctl system function to insert offender IP addresses into a firewall table refereced in a hidden built-in blocking rule in pfSense.

tylerevers

@bmeeks said in Suricata process dying due to hyperscan problem:

The "Block Mode" toggle is my generic name for the setting on the INTERFACE SETTINGS tab when you can enable or disable blocking. The setting is in the Alert and Block Settings section of the page. The checkbox is called Block Offenders. Unchecking that box removes all future blocking of offender IP addresses (it will not clear any currently existing blocks). There are also two settings for blocking offenders. One uses the netmap kernel device to implement a true inline-IPS mode of operation. But netmap will not work with VLANs or LAGG interfaces at the moment. You would need to run it on just the parent physical interface. Legacy Mode Blocking uses a custom output plugin compiled into the Suricata binary used on pfSense. This plugin calls a pfctl system function to insert offender IP addresses into a firewall table refereced in a hidden built-in blocking rule in pfSense.

Thank you for the insights and explanation. I have wanted to use true inline-IPS for some time, but I knew of the tradeoff and I simply cannot give up VLANs/LAGG.

NogBadTheBad

@bmeeks said in Suricata process dying due to hyperscan problem:

For other users experiencing the Hyperscan crash in Suricata --

Do you have one or more VLANs configured on the interface that crashes?

Yes on my LAN interface
PPPOE on my WAN interface

Does disabling blocking mode on the crashing interface result in a difference in behavior?

Yes for no blocking, ok

Set to AC and blocking ok

Set to Auto and Core dumps

If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

Currently set not to block so I can rebaseline my suppression list.

masons

@bmeeks , I think I may have found a reliable way to reproduce the issue.

Environment

Two separate VMs.

VM1

pfSense CE 2.7.0
4 vCPUs on KVM
AES-NI CPU Crypto: Yes
Suricata 7.0.2_1
LAN interface has multiple VLANs, but Suricata is only running on one of the VLAN interfaces (PC)
WAN interface is running Suricata

VM2

pfSense CE 2.7.1
4 vCPUs on KVM
AES-NI CPU Crypto: Yes
Suricata 7.0.2_1
LAN interface has multiple VLANs, but Suricata is only running on one of the VLAN interfaces (PC)
WAN interface is running Suricata

How to reproduce the issue

Start the Suricata service
Check the Suricata interfaces
- WAN will be running
- PC will not be running
suricata.log for the PC Suricata instance does not show the Hyperscan log error
System log shows

pid 1810 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

How to get the PC instance running

Stop the Suricata service
Go to Diagnostic --> Command Prompt and enter

elfctl -e +noaslr /usr/local/bin/suricata

Start the Suricata service
Check the Suricata interfaces, both WAN and PC will be running
suricata.log for the PC Suricata instance does not show the Hyperscan log error
System log shows no errors

I can cycle back and forth between +noaslr and -noaslr and the behaviour is completely repeatable. I've had one VM running with +noaslr for one day and both Suricata instances have remained up the whole time.

I know that I previously reported that the fix didn't appear to solve the problem, but it's worth noting that in my previous report I was seeing the Hyperscan log entry and it was the WAN interface that failed, not the PC interface. In this post I'm not seeing the Hyperscan log entry and Suricata instances have remained running for much longer.

bmeeks

@masons, @tylerevers, @NogBadTheBad. @jowe78:
Thank you all for the extra information. I will continue to dig into this.

asdjklfjkdslfdsaklj

@bmeeks said in Suricata process dying due to hyperscan problem:

@asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

@bmeeks anecdotally, yes. ~8hrs after disabling blocking mode and both LAN PHY Suricata instances are still up.

I need to know if you have any VLAN configured on either LAN interface.

Try enabling Blocking Mode on just one of the LAN interfaces and see what happens then.

To help me troubleshoot this, I desparately need you folks having the issue to give me some explicit details when responding. For example, answer question #1 above and also try troubleshooting suggestion #2 above. Then follow up back here with detailed results for each.

I will repeat again for clarity: I am trying to determine if VLANs configured on the crashing interface are related or not. So, tell me if you have VLANs on the interface, and if you do, how many. Then tell me if you can relate the crash to blocking enabled or not.

No VLANs.
Enabled blocking mode on LAN 1, disabled bon LAN 2.

Both ran for a few hours, and eventually LAN 1 died (same hyperscan error), while LAN 2 remains up.

tylerevers

@bmeeks

Reconfirm Hyperscan Still Crashes

Block Offenders = On
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

Interface failed with error:

[101378 - W#07] 2023-11-29 12:54:32 Error: spm-hs: Hyperscan returned fatal error -1.

Test with Block Offenders Off
Block Offenders = Off
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

It has been three hours without a crash.

bmeeks

@asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

Both ran for a few hours, and eventually LAN 1 died (same hyperscan error), while LAN 2 remains up.

Okay, now swap the blocking mode around. Disable blocking on LAN 1 and Enable blocking on LAN 2. Let's see if the hyperscan error moves over to LAN 2 and it now crashes while LAN 1 remains stable.

If the problem does not move to LAN 1, then that would tend to take blocking mode out of the picture unless it takes that in combination with something else to trigger the hyperscan crash.

bmeeks

@tylerevers said in Suricata process dying due to hyperscan problem:

@bmeeks

Reconfirm Hyperscan Still Crashes

Block Offenders = On
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

Interface failed with error:
[101378 - W#07] 2023-11-29 12:54:32 Error: spm-hs: Hyperscan returned fatal error -1.
Test with Block Offenders Off
Block Offenders = Off
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

It has been three hours without a crash.

How long does it typically take to crash? Is three hours of runtime quite a bit longer than you were getting with blocking enabled?

tylerevers

@bmeeks said in Suricata process dying due to hyperscan problem:

@tylerevers said in Suricata process dying due to hyperscan problem:
@bmeeks

Reconfirm Hyperscan Still Crashes

Block Offenders = On
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

Interface failed with error:
[101378 - W#07] 2023-11-29 12:54:32 Error: spm-hs: Hyperscan returned fatal error -1.
Test with Block Offenders Off
Block Offenders = Off
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

It has been three hours without a crash.
How long does it typically take to crash? Is three hours of runtime quite a bit longer than you were getting with blocking enabled?

Yes, three hours is in the realm of 3-8x longer (and it still hasn't crashed yet ~9 hours total).

bmeeks

@tylerevers said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:
@tylerevers said in Suricata process dying due to hyperscan problem:
@bmeeks

Reconfirm Hyperscan Still Crashes

Block Offenders = On
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

Interface failed with error:
[101378 - W#07] 2023-11-29 12:54:32 Error: spm-hs: Hyperscan returned fatal error -1.
Test with Block Offenders Off
Block Offenders = Off
Signature Group Header MPM Context = Auto
Pattern Matcher Algorithm = Auto

It has been three hours without a crash.
How long does it typically take to crash? Is three hours of runtime quite a bit longer than you were getting with blocking enabled?
Yes, three hours is in the realm of 3-8x longer (and it still hasn't crashed yet ~9 hours total).

Well, now I need to figure out how in the world the custom blocking module code could possibly interact with the Hyperscan library .

It makes no sense as they are not even remotely related.

chrysmon

@bmeeks Can confirm that in IDS mode (no blocking) suricata has no crashes. In IPS mode it crashes. Hyperscan, no VLANS.

asdjklfjkdslfdsaklj

@bmeeks swapped, same result. Instance on interface w/blocking disabled remains up, other died.

bmeeks

@asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

@bmeeks swapped, same result. Instance on interface w/blocking disabled remains up, other died.

Thank you. This is very helpful. It tells me that somehow the custom blocking module is part of the issue.

I will need to dig into the code and see if something pops out. It will be a few days, though, before I can generate debug versions of the package because the ESXi host that contained all my pfSense package builders and private testing repo crashed and burned last Sunday morning due to a power blip and my UPS failing at the same time. Something is weird with the UPS. It shows the battery as good, but if power blips it drops the load. I will need to get a new one. I've started the process of rebuilding my test environment on that host, but it's going to take a few days. Also have some other non-related obligations over the next 4 days that interfere with the effort.