Netgate hardware

bmeeks

Yes, Netgate tests pfSense on all of their hardware. They do not test packages, though. Packages in the pfSense system are for the most part created and maintained by communtiy volunteers. Snort and Suricata fall into that category. I am the creator/maintainer for those packages. But I am not employed by Netgate nor do I receive any monetary benefit from them for the packages. I volunteer my time and make code contributions in the spirit of open-source software.

The recent Suricata bug was actually traced to a bug in a FreeBSD library (libpfctl) included with pfSense. Some changes in Suricata and Snort to transition over to using that library for certain legacy operations exposed a latent bug in the library.

Read the updates in this thread: https://forum.netgate.com/topic/184112/important-snort-and-suricata-package-announcement-probable-bug-in-legacy-blocking-module.

Pay particular attention to the recent Updates under the red-highlighted headers. This issue is corrected for CE users, and also for Plus 23.09 users but that package builder is currently experiencing an issue and the deployment of the fix for 23.09 is still delayed a bit. Netgate is working on that now.

coxhaus

@bmeeks
Thank you for your time and effort to keep Snort and Suricata alive and going.

chrysmon

@bmeeks Thank you for your answer, you are 100% correct (of course). Seems to be a deadlock on the pid file of Suricata. However the Services Watchdog does not restart it as it should.

I have a question about pfSense hardware (not necessary Netgate) and you are very suitable person to respond. How many CPU cores and what CPU speed are required for pfSense to do the following:

inter VLAN routing on 2 SFP+ 10Gbit interfaces and other 4 1Gbit including WAN. There are 5 VLANs on each SFP+. There are firewall rules everywhere. No 10Gbit endpoints pass through pfSense. The network has about 500 MAC addresses.
Internet connection 150-200 Mbps symmetric.
OpenVPN (2-3 connections simultaneously all the day long, occasionally others). There are 5 OpenVPN servers.
Packages: Suricata, pfBlockerNG, Wireguard, Zabbix and some others not relevant here.

Thanks again, sorry for not posted in other thread.

bmeeks

@chrysmon said in Netgate hardware:

However the Services Watchdog does not restart it as it should.

Never ever never use Service Watchdog with Suricata or Snort!

Service Watchdog is a very primitive package. It does not understand how Suricata and Snort work internally, it does not understand how to monitor them when running on multiple interfaces, and it does not understand that both of them periodically restart themselves (after scheduled rules updates, for example). This means Service Watchdog foolishly will try to restart the Suricata or Snort daemons when they are already in the middle of restarting themselves. This will lead to a chaotic situation eventually. I've lost count of how many times I've posted in threads on this forum to never monitor the IDS/IPS packages with Service Watchdog.

I am tempted in the next update of both packages to include code in the package to detect if Service Watchdog is configured for monitoring Suricata and automatically refuse to start Suriata and log a message why if that is true.

Service Watchdog should really only be used when actively troubleshooting something or for very limited workarounds for some unique bug while awaiting a fix. It should NEVER be used on a routine basis for anything. If you have services randomly stopping, you need to find the root cause and fix it. Using Service Watchdog is a very crude hammer and can lead to other problems. Services should never just randomly stop. If they do, find out why and fix the actual problem. Don't cover it up with a Service Watchdog bandaid.

bmeeks

@chrysmon said in Netgate hardware:

I have a question about pfSense hardware (not necessary Netgate) and you are very suitable person to respond. How many CPU cores and what CPU speed are required for pfSense to do the following:

inter VLAN routing on 2 SFP+ 10Gbit interfaces and other 4 1Gbit including WAN. There are 5 VLANs on each SFP+. There are firewall rules everywhere. No 10Gbit endpoints pass through pfSense. The network has about 500 MAC addresses.
Internet connection 150-200 Mbps symmetric.
OpenVPN (2-3 connections simultaneously all the day long, occasionally others). There are 5 OpenVPN servers.
Packages: Suricata, pfBlockerNG, Wireguard, Zabbix and some others not relevant here.

Sorry, but I have no idea how answer those questions. You might want to take this up with the Netgate sales team.

chrysmon

@bmeeks Sorry for the Watchdog, I'm new in the forum.
The question is a CPU suggestion for the configuration specified. Because you know each package is multi core or not, etc.

bmeeks

@chrysmon said in Netgate hardware:

Because you know each package is multi core or not, etc.

No, I only know about the two packages I manage. Suricata is multithreaded while Snort is not. Those are the only ones I know about as those are the only two I maintain.

stephenw10

I will add that we do test packages especially the popular packages like Suricata. But we cannot test every package in every configuration on all hardware types. Unfortunately.

bmeeks

@stephenw10 said in Netgate hardware:

I will add that we do test packages especially the popular packages like Suricata. But we cannot test every package in every configuration on all hardware types. Unfortunately.

Thanks @stephenw10. I did not know that the Netgate team ran formal tests with Suricata.

I just assumed the occasional tickets that appear on Redmine with Netgate folks as the author were the result of client troubleshooting efforts by the paid support Help Desk staff.

stephenw10

The testing we can do there is fairly limited. But, for example, if Suricata failed to install or failed to start with a basic config that should be caught by release testing.

More informally a number of us run test configs for Snort and Suricata with more complex configs. But that combinations of config and architecture are large!

chrysmon

@stephenw10 Maybe for longer than a day I had no Suricata crashes. Today I updated the package and it crashed twice since then. Version 7.0.2_1. I restart manually after deleting the /var/run/suricata_whatever

bmeeks

@chrysmon said in Netgate hardware:

@stephenw10 Maybe for longer than a day I had no Suricata crashes. Today I updated the package and it crashed twice since then. Version 7.0.2_1. I restart manually after deleting the /var/run/suricata_whatever

Are you seeing the same HyperScan error as the users are reporting in this thread: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem?

If so, please post your current configuration and any log messages from the pfSense system log (under STATUS > SYSTEM LOGS) and the suricata.log (under LOGS VIEW in the Suricata GUI) into a new message in that thread. I would like to consolidate all the Suricata "stops" into a common thread if possible as I think they are all related to something with the HyperScan library and Suricata's interaction with it.

chrysmon

@bmeeks Yes, The last log entry in the STATUS > SYSTEM LOGS is:

suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.

This is from the suricata.log:

[107176 - Suricata-Main] 2023-11-25 12:48:26 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: cpu: CPUs/cores online: 12
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: suricata: Setting engine mode to IDS mode by default
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: app-layer-htp-mem: HTTP memcap: 671088640
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: alert-pf output device (regular) initialized: block.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_5401_igb0/passlist.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_5401_igb0/passlist processed: Total entries parsed: 20, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 2.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=src kill-state=yes block-drops-only=yes passlist-debugging=no
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: fast output device (regular) initialized: alerts.log
[209444 - ] 2023-11-25 12:48:26 Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: http-log output device (regular) initialized: http.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: stats output device (regular) initialized: stats.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-syslog: Syslog output initialized
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: eve-log output device (regular) initialized: eve.json
[100456 - Suricata-Main] 2023-11-25 12:48:26 Warning: output-json-alert: HTTP body logging has been configured, however, metadata logging has not been enabled. HTTP body logging will be disabled.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: output-json-email-common: Going to log the md5 sum of email subject
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 2 rule files processed. 44008 rules successfully loaded, 0 rules failed
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 44017 signatures processed. 1282 are IP-only rules, 6728 are inspecting packet payload, 35719 inspect application layer, 109 are decoder event only
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'ET.GenericPhish_Adobe' is checked but not set. Checked in 2023048 and 0 other sigs
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'is_ssh_client_kex' is checked but not set. Checked in 2001977 and 1 other sigs
[100456 - Suricata-Main] 2023-11-25 12:49:09 Info: runmodes: Using 1 live device(s).
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: snaplen set to 14180
[100456 - Suricata-Main] 2023-11-25 12:49:10 Notice: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1 Engine started.
[209486 - RX#01-igb0] 2023-11-25 12:49:14 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[209498 - W#12] 2023-11-25 20:21:08 Error: spm-hs: Hyperscan returned fatal error -1.

I'm running Suricata only on the WAN interface, in IPS Mode (Legacy), the Pattern Matcher Algorithm set to Auto. Do you need any other information about configuration? Maybe the part from the backup file?

bmeeks

@chrysmon said in Netgate hardware:

@bmeeks Yes, The last log entry in the STATUS > SYSTEM LOGS is:

suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.

This is from the suricata.log:

[107176 - Suricata-Main] 2023-11-25 12:48:26 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: cpu: CPUs/cores online: 12
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: suricata: Setting engine mode to IDS mode by default
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: app-layer-htp-mem: HTTP memcap: 671088640
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: alert-pf output device (regular) initialized: block.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_5401_igb0/passlist.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_5401_igb0/passlist processed: Total entries parsed: 20, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 2.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=src kill-state=yes block-drops-only=yes passlist-debugging=no
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: fast output device (regular) initialized: alerts.log
[209444 - ] 2023-11-25 12:48:26 Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: http-log output device (regular) initialized: http.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: stats output device (regular) initialized: stats.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-syslog: Syslog output initialized
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: eve-log output device (regular) initialized: eve.json
[100456 - Suricata-Main] 2023-11-25 12:48:26 Warning: output-json-alert: HTTP body logging has been configured, however, metadata logging has not been enabled. HTTP body logging will be disabled.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: output-json-email-common: Going to log the md5 sum of email subject
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 2 rule files processed. 44008 rules successfully loaded, 0 rules failed
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 44017 signatures processed. 1282 are IP-only rules, 6728 are inspecting packet payload, 35719 inspect application layer, 109 are decoder event only
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'ET.GenericPhish_Adobe' is checked but not set. Checked in 2023048 and 0 other sigs
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'is_ssh_client_kex' is checked but not set. Checked in 2001977 and 1 other sigs
[100456 - Suricata-Main] 2023-11-25 12:49:09 Info: runmodes: Using 1 live device(s).
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: snaplen set to 14180
[100456 - Suricata-Main] 2023-11-25 12:49:10 Notice: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1 Engine started.
[209486 - RX#01-igb0] 2023-11-25 12:49:14 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[209498 - W#12] 2023-11-25 20:21:08 Error: spm-hs: Hyperscan returned fatal error -1.

I'm running Suricata only on the WAN interface, in IPS Mode (Legacy), the Pattern Matcher Algorithm set to Auto. Do you need any other information about configuration? Maybe the part from the backup file?

No, this is obviously the HyperScan issue described in the thread I linked. It's right here in the log:

suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.

But I was wanting information about this bug posted in the other thread I linked so that there aren't half a dozen other threads scattered around the forum about the same issue. It makes it hard for me to track who has what problem and what shared information they might have if there are lots of different threads all about the same basic issue. Much easier to keep track when all the comments and reports about a given issue are in the same thread.

Please post anything else you have about this issue in the thread I linked earlier. Here is the direct link again: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem.