Netgate hardware
-
@chrysmon said in Netgate hardware:
However the Services Watchdog does not restart it as it should.
Never ever never use Service Watchdog with Suricata or Snort!
Service Watchdog is a very primitive package. It does not understand how Suricata and Snort work internally, it does not understand how to monitor them when running on multiple interfaces, and it does not understand that both of them periodically restart themselves (after scheduled rules updates, for example). This means Service Watchdog foolishly will try to restart the Suricata or Snort daemons when they are already in the middle of restarting themselves. This will lead to a chaotic situation eventually. I've lost count of how many times I've posted in threads on this forum to never monitor the IDS/IPS packages with Service Watchdog.
I am tempted in the next update of both packages to include code in the package to detect if Service Watchdog is configured for monitoring Suricata and automatically refuse to start Suriata and log a message why if that is true.
Service Watchdog should really only be used when actively troubleshooting something or for very limited workarounds for some unique bug while awaiting a fix. It should NEVER be used on a routine basis for anything. If you have services randomly stopping, you need to find the root cause and fix it. Using Service Watchdog is a very crude hammer and can lead to other problems. Services should never just randomly stop. If they do, find out why and fix the actual problem. Don't cover it up with a Service Watchdog bandaid.
-
@chrysmon said in Netgate hardware:
I have a question about pfSense hardware (not necessary Netgate) and you are very suitable person to respond. How many CPU cores and what CPU speed are required for pfSense to do the following:
inter VLAN routing on 2 SFP+ 10Gbit interfaces and other 4 1Gbit including WAN. There are 5 VLANs on each SFP+. There are firewall rules everywhere. No 10Gbit endpoints pass through pfSense. The network has about 500 MAC addresses.
Internet connection 150-200 Mbps symmetric.
OpenVPN (2-3 connections simultaneously all the day long, occasionally others). There are 5 OpenVPN servers.
Packages: Suricata, pfBlockerNG, Wireguard, Zabbix and some others not relevant here.Sorry, but I have no idea how answer those questions. You might want to take this up with the Netgate sales team.
-
@bmeeks Sorry for the Watchdog, I'm new in the forum.
The question is a CPU suggestion for the configuration specified. Because you know each package is multi core or not, etc. -
@chrysmon said in Netgate hardware:
Because you know each package is multi core or not, etc.
No, I only know about the two packages I manage. Suricata is multithreaded while Snort is not. Those are the only ones I know about as those are the only two I maintain.
-
I will add that we do test packages especially the popular packages like Suricata. But we cannot test every package in every configuration on all hardware types. Unfortunately.
-
@stephenw10 said in Netgate hardware:
I will add that we do test packages especially the popular packages like Suricata. But we cannot test every package in every configuration on all hardware types. Unfortunately.
Thanks @stephenw10. I did not know that the Netgate team ran formal tests with Suricata.
I just assumed the occasional tickets that appear on Redmine with Netgate folks as the author were the result of client troubleshooting efforts by the paid support Help Desk staff.
-
The testing we can do there is fairly limited. But, for example, if Suricata failed to install or failed to start with a basic config that should be caught by release testing.
More informally a number of us run test configs for Snort and Suricata with more complex configs. But that combinations of config and architecture are large!
-
@stephenw10 Maybe for longer than a day I had no Suricata crashes. Today I updated the package and it crashed twice since then. Version 7.0.2_1. I restart manually after deleting the /var/run/suricata_whatever
-
@chrysmon said in Netgate hardware:
@stephenw10 Maybe for longer than a day I had no Suricata crashes. Today I updated the package and it crashed twice since then. Version 7.0.2_1. I restart manually after deleting the /var/run/suricata_whatever
Are you seeing the same HyperScan error as the users are reporting in this thread: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem?
If so, please post your current configuration and any log messages from the pfSense system log (under STATUS > SYSTEM LOGS) and the
suricata.log(under LOGS VIEW in the Suricata GUI) into a new message in that thread. I would like to consolidate all the Suricata "stops" into a common thread if possible as I think they are all related to something with the HyperScan library and Suricata's interaction with it. -
@bmeeks Yes, The last log entry in the STATUS > SYSTEM LOGS is:
suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.
This is from the suricata.log:
[107176 - Suricata-Main] 2023-11-25 12:48:26 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: cpu: CPUs/cores online: 12
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: suricata: Setting engine mode to IDS mode by default
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: app-layer-htp-mem: HTTP memcap: 671088640
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: alert-pf output device (regular) initialized: block.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_5401_igb0/passlist.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_5401_igb0/passlist processed: Total entries parsed: 20, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 2.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=src kill-state=yes block-drops-only=yes passlist-debugging=no
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: fast output device (regular) initialized: alerts.log
[209444 - ] 2023-11-25 12:48:26 Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: http-log output device (regular) initialized: http.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: stats output device (regular) initialized: stats.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-syslog: Syslog output initialized
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: eve-log output device (regular) initialized: eve.json
[100456 - Suricata-Main] 2023-11-25 12:48:26 Warning: output-json-alert: HTTP body logging has been configured, however, metadata logging has not been enabled. HTTP body logging will be disabled.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: output-json-email-common: Going to log the md5 sum of email subject
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 2 rule files processed. 44008 rules successfully loaded, 0 rules failed
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 44017 signatures processed. 1282 are IP-only rules, 6728 are inspecting packet payload, 35719 inspect application layer, 109 are decoder event only
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'ET.GenericPhish_Adobe' is checked but not set. Checked in 2023048 and 0 other sigs
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'is_ssh_client_kex' is checked but not set. Checked in 2001977 and 1 other sigs
[100456 - Suricata-Main] 2023-11-25 12:49:09 Info: runmodes: Using 1 live device(s).
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: snaplen set to 14180
[100456 - Suricata-Main] 2023-11-25 12:49:10 Notice: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1 Engine started.
[209486 - RX#01-igb0] 2023-11-25 12:49:14 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[209498 - W#12] 2023-11-25 20:21:08 Error: spm-hs: Hyperscan returned fatal error -1.I'm running Suricata only on the WAN interface, in IPS Mode (Legacy), the Pattern Matcher Algorithm set to Auto. Do you need any other information about configuration? Maybe the part from the backup file?
-
@chrysmon said in Netgate hardware:
@bmeeks Yes, The last log entry in the STATUS > SYSTEM LOGS is:
suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.
This is from the suricata.log:
[107176 - Suricata-Main] 2023-11-25 12:48:26 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: cpu: CPUs/cores online: 12
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: suricata: Setting engine mode to IDS mode by default
[107176 - Suricata-Main] 2023-11-25 12:48:26 Info: app-layer-htp-mem: HTTP memcap: 671088640
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: alert-pf output device (regular) initialized: block.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_5401_igb0/passlist.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Pass List /usr/local/etc/suricata/suricata_5401_igb0/passlist processed: Total entries parsed: 20, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 2.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-pf: pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=src kill-state=yes block-drops-only=yes passlist-debugging=no
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: fast output device (regular) initialized: alerts.log
[209444 - ] 2023-11-25 12:48:26 Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: http-log output device (regular) initialized: http.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: stats output device (regular) initialized: stats.log
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: alert-syslog: Syslog output initialized
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: logopenfile: eve-log output device (regular) initialized: eve.json
[100456 - Suricata-Main] 2023-11-25 12:48:26 Warning: output-json-alert: HTTP body logging has been configured, however, metadata logging has not been enabled. HTTP body logging will be disabled.
[100456 - Suricata-Main] 2023-11-25 12:48:26 Info: output-json-email-common: Going to log the md5 sum of email subject
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 2 rule files processed. 44008 rules successfully loaded, 0 rules failed
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[100456 - Suricata-Main] 2023-11-25 12:48:43 Info: detect: 44017 signatures processed. 1282 are IP-only rules, 6728 are inspecting packet payload, 35719 inspect application layer, 109 are decoder event only
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'ET.GenericPhish_Adobe' is checked but not set. Checked in 2023048 and 0 other sigs
[100456 - Suricata-Main] 2023-11-25 12:48:43 Warning: detect-flowbits: flowbit 'is_ssh_client_kex' is checked but not set. Checked in 2001977 and 1 other sigs
[100456 - Suricata-Main] 2023-11-25 12:49:09 Info: runmodes: Using 1 live device(s).
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[209486 - RX#01-igb0] 2023-11-25 12:49:09 Info: pcap: igb0: snaplen set to 14180
[100456 - Suricata-Main] 2023-11-25 12:49:10 Notice: threads: Threads created -> RX: 1 W: 12 FM: 1 FR: 1 Engine started.
[209486 - RX#01-igb0] 2023-11-25 12:49:14 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[209498 - W#12] 2023-11-25 20:21:08 Error: spm-hs: Hyperscan returned fatal error -1.I'm running Suricata only on the WAN interface, in IPS Mode (Legacy), the Pattern Matcher Algorithm set to Auto. Do you need any other information about configuration? Maybe the part from the backup file?
No, this is obviously the HyperScan issue described in the thread I linked. It's right here in the log:
suricata 17648 [209498] <Error> -- Hyperscan returned fatal error -1.But I was wanting information about this bug posted in the other thread I linked so that there aren't half a dozen other threads scattered around the forum about the same issue. It makes it hard for me to track who has what problem and what shared information they might have if there are lots of different threads all about the same basic issue. Much easier to keep track when all the comments and reports about a given issue are in the same thread.
Please post anything else you have about this issue in the thread I linked earlier. Here is the direct link again: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem.
