Suricata not starting and blank log

bmeeks

The suricata.log file for an interface is wiped clean each time a start attempt is made, so bear that in mind when looking at the file. However, in most cases the failed start attempt will leave a log file containing the error that caused the abort. If that is not happening, then usually the problem is failure of the binary to start at all due to missing libraries. Your test at the command-line indicates missing libraries are not the cause.

If you run this command, does it come up empty with no running Suricata processes?

ps -ax | grep suricata

Are there any Suricata PID files in /var/run?

If you get no errors when executing

suricata -V

then the binary itself can start just fine.

You system log snippet indicates you are using VLANs. Do you actually have VLANs defined on your WAN?

At this point I'm rapidly running out of ideas. One last point, make sure you are trying with Legacy Blocking Mode first (I suspect you are since you did not say otherwise, but just reminding).

kesawi

@bmeeks there's no running Suricata process, no PID file and no errors when executing.

My ISP requires a VLAN on the WAN and it is defined. My WAN is set to use the VLAN.

In relation to Legacy Blocking Mode, I've used the defaults when setting up Suricata. I've had a look and am not sure where the blocking mode is set. Suricata is configured to alert only and I haven't enabled blocking.

bmeeks

@kesawi said in Suricata not starting and blank log:

@bmeeks there's no running Suricata process, no PID file and no errors when executing.

My ISP requires a VLAN on the WAN and it is defined. My WAN is set to use the VLAN.

In relation to Legacy Blocking Mode, I've used the defaults when setting up Suricata. I've had a look and am not sure where the blocking mode is set. Suricata is configured to alert only and I haven't enabled blocking.

Okay, I missed the part about alert only so disregard the blocking question. You set that on the INTERFACE SETTINGS tab, though.

I am really baffled at this point why you don't get a start. There are two ways to start Suricata. One is via the GUI on the INTERFACES tab, and the other is via a shell script in /usr/local/etc/rc.d. So when you click the Start icon in the GUI what happens?

kesawi

@bmeeks I've been able to get a little further.

I ran /usr/local/etc/rc.d/suricata onestart which produced the following output:

Starting suricata.
[100735] 31/3/2020 -- 11:02:20 - (suricata.c:1084) <Notice> (LogVersion) -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode

It stopped straight away but populated /var/run/log/suricata.log with the following:

[100735] 31/3/2020 -- 11:02:20 - (suricata.c:1084) <Notice> (LogVersion) -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
[100735] 31/3/2020 -- 11:02:20 - (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 8
[100781] 31/3/2020 -- 11:02:21 - (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- fast output device (regular) initialized: fast.log
[100781] 31/3/2020 -- 11:02:21 - (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- eve-log output device (regular) initialized: eve.json
[100781] 31/3/2020 -- 11:02:21 - (util-logopenfile.c:474) <Info> (SCConfLogOpenGeneric) -- stats output device (regular) initialized: stats.log
[100781] 31/3/2020 -- 11:02:21 - (util-conf.c:162) <Info> (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket
[100781] 31/3/2020 -- 11:02:21 - (detect-engine-loader.c:230) <Warning> (ProcessSigFiles) -- [ERRCODE: SC_ERR_NO_RULES(42)] - No rule files match the pattern /var/lib/suricata/rules/suricata.rules
[100781] 31/3/2020 -- 11:02:21 - (detect-engine-loader.c:345) <Warning> (SigLoadSignatures) -- [ERRCODE: SC_ERR_NO_RULES_LOADED(43)] - 1 rule files specified, but no rule was loaded at all!
[100781] 31/3/2020 -- 11:02:21 - (util-threshold-config.c:1126) <Info> (SCThresholdConfParseFile) -- Threshold config parsed: 0 rule(s) found
[100781] 31/3/2020 -- 11:02:21 - (detect-engine-build.c:1416) <Info> (SigAddressPrepareStage1) -- 0 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 0 inspect application layer, 0 are decoder event only
[101058] 31/3/2020 -- 11:02:21 - (source-ipfw.c:345) <Error> (ReceiveIPFWThreadInit) -- [ERRCODE: SC_ERR_IPFW_SOCK(81)] - Can't create divert socket: Protocol not supported
[100781] 31/3/2020 -- 11:02:21 - (util-conf.c:162) <Info> (ConfUnixSocketIsEnable) -- Running in live mode, activating unix socket
[100781] 31/3/2020 -- 11:02:21 - (unix-manager.c:132) <Info> (UnixNew) -- Using unix socket file '/var/run/suricata/suricata-command.socket'
[100781] 31/3/2020 -- 11:02:21 - (unix-manager.c:150) <Info> (UnixNew) -- Created socket directory /var/run/suricata/
[100781] 31/3/2020 -- 11:02:21 - (tm-threads.c:2125) <Error> (TmThreadWaitOnThreadInit) -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "RX-8000" failed to initialize: flags 0145
[100781] 31/3/2020 -- 11:02:21 - (suricata.c:3073) <Error> (main) -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...

I then attempted to start Suricata on the WAN interface through the GUI in the Interfaces menu. The cog spun, a green tick appeared under the status and then it changed to a red cross. The following is not in /var/log/suricata/suricata_igb5.10050183/suricata.log:

31/3/2020 -- 11:04:12 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
31/3/2020 -- 11:04:12 - <Info> -- CPUs/cores online: 8
31/3/2020 -- 11:04:12 - <Info> -- HTTP memcap: 67108864
31/3/2020 -- 11:04:12 - <Notice> -- using flow hash instead of active packets
31/3/2020 -- 11:04:12 - <Info> -- fast output device (regular) initialized: alerts.log
31/3/2020 -- 11:04:12 - <Info> -- http-log output device (regular) initialized: http.log
31/3/2020 -- 11:04:12 - <Info> -- stats output device (regular) initialized: stats.log
31/3/2020 -- 11:04:13 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'byte_math'.
31/3/2020 -- 11:04:13 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET 4786 (msg:"SERVER-OTHER Cisco Smart Install invalid init discovery message denial of service attempt"; flow:to_server,established; content:"|00 00 00|"; depth:3; content:"|00 00 00 07|"; within:4; distance:5; fast_pattern; content:"|00 00 00 01|"; within:4; distance:4; byte_math:bytes 4,offset 0,oper +,rvalue 8,result sub_len_plus_eight,relative; byte_test:4,!=,sub_len_plus_eight,-8,relative; metadata:policy balanced-ips drop, policy connectivity-ips drop, policy max-detect-ips drop, policy security-ips drop; reference:cve,2018-0171; reference:url,tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180328-smi2; classtype:attempted-dos; sid:46468; rev:1;)" from file /usr/local/etc/suricata/suricata_50183_igb5.100/rules/suricata.rules at line 717
31/3/2020 -- 11:04:13 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
31/3/2020 -- 11:04:13 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Citrix ADC and Gateway arbitrary code execution attempt"; flow:to_server,established; content:"/vpns/"; fast_pattern:only; content:"NSC_USER:"; http_raw_header; content:"../"; within:10; http_raw_header; metadata:policy balanced-ips drop, policy connectivity-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:cve,2019-19781; reference:url,support.citrix.com/article/CTX267027; classtype:web-application-attack; sid:52620; rev:1;)" from file /usr/local/etc/suricata/suricata_50183_igb5.100/rules/suricata.rules at line 796
31/3/2020 -- 11:04:13 - <Info> -- 2 rule files processed. 899 rules successfully loaded, 2 rules failed
31/3/2020 -- 11:04:13 - <Info> -- Threshold config parsed: 0 rule(s) found
31/3/2020 -- 11:04:13 - <Info> -- 899 signatures processed. 0 are IP-only rules, 153 are inspecting packet payload, 282 inspect application layer, 103 are decoder event only
31/3/2020 -- 11:04:13 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.class|file.jar' is checked but not set. Checked in 31540 and 0 other sigs
31/3/2020 -- 11:04:13 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 0 other sigs
31/3/2020 -- 11:04:16 - <Info> -- Using 1 live device(s).
31/3/2020 -- 11:04:16 - <Info> -- using interface igb5.100
31/3/2020 -- 11:04:16 - <Info> -- running in 'auto' checksum mode. Detection of interface state will require 1000ULL packets
31/3/2020 -- 11:04:16 - <Info> -- Set snaplen to 1518 for 'igb5.100'
31/3/2020 -- 11:04:16 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
31/3/2020 -- 11:04:16 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
31/3/2020 -- 11:04:16 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
31/3/2020 -- 11:04:16 - <Info> -- RunModeIdsPcapAutoFp initialised
31/3/2020 -- 11:04:16 - <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#08" failed to initialize: flags 0145
31/3/2020 -- 11:04:16 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...

Doubled Stream Memory Cap on the WAN interface from 64MB to 128MB and started Suricata and this time it worked. Log file gives:

31/3/2020 -- 11:15:07 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
31/3/2020 -- 11:15:07 - <Info> -- CPUs/cores online: 8
31/3/2020 -- 11:15:07 - <Info> -- HTTP memcap: 67108864
31/3/2020 -- 11:15:07 - <Notice> -- using flow hash instead of active packets
31/3/2020 -- 11:15:07 - <Info> -- fast output device (regular) initialized: alerts.log
31/3/2020 -- 11:15:07 - <Info> -- http-log output device (regular) initialized: http.log
31/3/2020 -- 11:15:07 - <Info> -- stats output device (regular) initialized: stats.log
31/3/2020 -- 11:15:07 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'byte_math'.
31/3/2020 -- 11:15:07 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET 4786 (msg:"SERVER-OTHER Cisco Smart Install invalid init discovery message denial of service attempt"; flow:to_server,established; content:"|00 00 00|"; depth:3; content:"|00 00 00 07|"; within:4; distance:5; fast_pattern; content:"|00 00 00 01|"; within:4; distance:4; byte_math:bytes 4,offset 0,oper +,rvalue 8,result sub_len_plus_eight,relative; byte_test:4,!=,sub_len_plus_eight,-8,relative; metadata:policy balanced-ips drop, policy connectivity-ips drop, policy max-detect-ips drop, policy security-ips drop; reference:cve,2018-0171; reference:url,tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180328-smi2; classtype:attempted-dos; sid:46468; rev:1;)" from file /usr/local/etc/suricata/suricata_50183_igb5.100/rules/suricata.rules at line 717
31/3/2020 -- 11:15:07 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
31/3/2020 -- 11:15:07 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Citrix ADC and Gateway arbitrary code execution attempt"; flow:to_server,established; content:"/vpns/"; fast_pattern:only; content:"NSC_USER:"; http_raw_header; content:"../"; within:10; http_raw_header; metadata:policy balanced-ips drop, policy connectivity-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:cve,2019-19781; reference:url,support.citrix.com/article/CTX267027; classtype:web-application-attack; sid:52620; rev:1;)" from file /usr/local/etc/suricata/suricata_50183_igb5.100/rules/suricata.rules at line 796
31/3/2020 -- 11:15:08 - <Info> -- 2 rule files processed. 899 rules successfully loaded, 2 rules failed
31/3/2020 -- 11:15:08 - <Info> -- Threshold config parsed: 0 rule(s) found
31/3/2020 -- 11:15:08 - <Info> -- 899 signatures processed. 0 are IP-only rules, 153 are inspecting packet payload, 282 inspect application layer, 103 are decoder event only
31/3/2020 -- 11:15:08 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.class|file.jar' is checked but not set. Checked in 31540 and 0 other sigs
31/3/2020 -- 11:15:08 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 0 other sigs
31/3/2020 -- 11:15:11 - <Info> -- Using 1 live device(s).
31/3/2020 -- 11:15:11 - <Info> -- using interface igb5.100
31/3/2020 -- 11:15:11 - <Info> -- running in 'auto' checksum mode. Detection of interface state will require 1000ULL packets
31/3/2020 -- 11:15:11 - <Info> -- Set snaplen to 1518 for 'igb5.100'
31/3/2020 -- 11:15:11 - <Info> -- RunModeIdsPcapAutoFp initialised
31/3/2020 -- 11:15:11 - <Notice> -- all 9 packet processing threads, 2 management threads initialized, engine started.
31/3/2020 -- 11:15:12 - <Info> -- No packets with invalid checksum, assuming checksum offloading is NOT used

Ran a speed test and confirmed CPU usage spiked which indicates Suricata is scanning. Are there any sites I can use to test whether detection is working?

bmeeks

You used the wrong file in /usr/local/etc/rc.d. The correct command would be this one:

/usr/local/etc/rc.d/suricata.sh start

By using that other script Suricata has no idea where its proper config file lives nor which interfaces to start on. The shell script I referenced above is created by the GUI code.

Your other problem is the stream_memcap problem that happens on machines with high core count CPUs. That is a common known problem, but most pfSense users don't hit that because they don't use high core count CPUs.

The SC_ERR_INVALID_SIGNATURE and other similar errors are expected when you attempt to use certain Snort rules with Suricata. There are some Snort keywords and rule options that Suricata neither supports nor decodes properly. For those, it will print an error in the startup log and skip loading that rule.

The best way to test Suricata is to use a suite such as Kali Linux to scan your firewall. A virtual machine created with a Kali Linux image is what I use. There are a number of options for testing Suricata using the tools within Kali.

kesawi

@bmeeks thanks for your help.

I have a number of VLANs running for my internal networks on lagg0. Is it possible for me to assign Suricata to the lagg0 interface in promiscuous mode and monitor the VLANs rather than creating separate interfaces in Suricata for each VLAN? I haven't assigned lagg0 to an interface under the Interfaces\Interface Assignments menu which is why I suspect it isn't appearing in the Suricata GUI.

Do I need to assign lagg0 to an interface? If I do, can I leave the IPv4 and IPv6 configuration as none? What value would I need to increase Interface PCAP Snaplen to in Suricata?

bmeeks

Monitoring VLANs via the parent interface is really the preferred method when it works for your setup. This kind of setup would not work, though, if you wanted different rules for the various VLANs.

You can create a Suricata interface on any enabled interface you have in pfSense. I'm not so sure that using lagg0 is going to work out well, though. But you can certainly try. That is a special kind of interface and not really the same as saying using a physical NIC.

The Snaplen should always match your Ethernet frame plus VLAN tags. The default value is usually sufficient to capture both of those, but you can increase it if you want to experiment.

kesawi

Will it place an increased load on the CPU if I monitor each VLAN interface separately rather than from the parent interface? I assume that as each interface will require a separate instance of suricate so there may be some additional overhead but the traffic volume would be the same in each case? I assume the RAM usage would increase as each instance will require its own memcap limits?

bmeeks

@kesawi said in Suricata not starting and blank log:

Will it place an increased load on the CPU if I monitor each VLAN interface separately rather than from the parent interface? I assume that as each interface will require a separate instance of suricate so there may be some additional overhead but the traffic volume would be the same in each case? I assume the RAM usage would increase as each instance will require its own memcap limits?

Of course. Each active Suricata instance requires CPU cycles to execute. So more CPU utilization and obviously more RAM consumed with multiple Suricata instances. Your OS will spend time and energy allocating time slices to the various running Suricata binaries, so lots of context switching will be happening as well.

crugeman

Evening, I am having similar issues with not starting and an empty log. When I run the
/usr/local/bin/suricata -v command I get:

Shared object "libibverbs.so.1" not found, required by "libpcap.so.1"

How do I go about installing the package or source it? I can't find it in the package manager.

kesawi

@crugeman I initially had the same issue. I resolved this by completely uninstalling suricata, upgrading pfSense from the previous version 2.4.4-RELEASE-p3 I was running to the latest 2.4.5-RELEASE which contains the library, and then installing suricata.

bmeeks

@crugeman said in Suricata not starting and blank log:

Evening, I am having similar issues with not starting and an empty log. When I run the
/usr/local/bin/suricata -v command I get:

Shared object "libibverbs.so.1" not found, required by "libpcap.so.1"

How do I go about installing the package or source it? I can't find it in the package manager.

What version of pfSense are running and on what kind of hardware? Is it a Netgate appliance, and if so, which one?

If you are on pfSense-2.4.4_p3 and tried to upgrade Suricata, you are hosed until you upgrade your firewall to pfSense-2.4.5. There are several warnings in the pfSense upgrade docs about updating pfSense BEFORE you update packages whenever a new pfSense version is available. That's because new pfSense versions frequently come with FreeBSD ports trees (where packages come from) that are based on newer libraries. That's the case here. The "current packages" repo has been recompiled for use on pfSense-2.4.5 which is based on FreeBSD 11.3/STABLE.

If you absolutely can't upgrade your firewall to 2.4.5, then go read through this thread and follow the information there at your own risk: https://forum.netgate.com/topic/151709/2-4-5-update-caution/43.

kesawi

@bmeeks said in Suricata not starting and blank log:

If you are on pfSense-2.4.4_p3 and tried to upgrade Suricata, you are hosed until you upgrade your firewall to pfSense-2.4.5.

If suricata is being installed for the first time, rather than an upgrade, then it still downloads the current version which is incompatible with releases prior to 2.4.5. There should be a warning in the description in package manager.

bmeeks

@kesawi said in Suricata not starting and blank log:

@bmeeks said in Suricata not starting and blank log:

If you are on pfSense-2.4.4_p3 and tried to upgrade Suricata, you are hosed until you upgrade your firewall to pfSense-2.4.5.

If suricata is being installed for the first time, rather than an upgrade, then it still downloads the current version which is incompatible with releases prior to 2.4.5. The should be a warning in the description in package manager.

Unfortunately the pkg utility in FreeBSD does not work that way so far as I know. The same kind of issue exists in Linux where if you specify the wrong package repo version you can break software.

The pkg configuration files in pfSense maintain two different repo pointers. One points to where to fetch the base OS files, while the other points to where to fetch packages. It's that second one that is pointing to the pfSense-2.4.5 files, I believe. But I don't profess to be a pkg expert.

crugeman

Thanks for the quick response. I'm on pfSense-2.4.4_p3 right now. Will update tomorrow and report back.

crugeman

Thanks @bmeeks and @kesawi for your help! Upgrade complete, installed Suricata and everything up and running.

Now on to the tunning! :)

cromulon

I'm having a similar issue, however, not the same symptoms as @crugeman. when I run the /usr/local/etc/rc.d/suricata onestart command I get this output:

Starting suricata.
Shared object "libibverbs.so.1" not found, required by "libpcap.so.1"
/usr/local/etc/rc.d/suricata: WARNING: failed to start suricata

I see the error, but I'm not sure how to fix it on pfsense. I'm running 2.4.4_p3. suricata installed is 5.0.2

EDIT:

After reading the entire thread I think I understand the problem in my case.

I upgraded pfsense to 2.4.5, prior to that I made a backup. Upgrade was smooth. I noticed though afterwards I was getting a lot of issues with the NIC and/or System with large file transfers (downloading from web, or uploading to server across the LAN). On console, i noticed repeatedly seeing re0: watchdog timeout. I know this is because realtek nics suck. However, I didn't have as much as a problem on 2.4.4_p3. My decision was to roll back. Made another backup (this is froma 2.4.5 machine after all packages had been installed).

Fast forward to installation of 2.4.4_p3 and restoring. everything downloads, and mostly everything starts fine. However, Suricata doesn't start.

It wasn't until this thread and another did I realize, that suricata 5.0.2 should not be run on 2.4.4_p3 due to library dependencies. I take it, that I must upgrade to 2.4.5, then get a not so crappy nic. If I'm wrong please guide me to the light.

bmeeks

@cromulon:
Yes, you can only run the new Suricata package on pfSense-2.4.5 due to changes in some supporting libraries. Right now you have a highly confused pfSense pkg configuration because it has a 2.4.4_p3 base OS but some incorrect libraries downloaded and installed because of the Suricata package that is compiled for pfSense-2.4.5 only.

If you check the Hardware subform here on the Netgate forums you will find a link for downloading and installing a Realtek binary driver that fixes the watchdog timer issue. Or even better, install an Intel NIC and forget the crappy Realtek.

cromulon

@bmeeks

I'm not even going to waste my time on realtek, tomorrow I'm ordering the intel PRO/1000 PT 4-port. Think that's a mighty fine investment.

Also, I just upgraded back to 2.4.5 and the suricata package is back to running. I'm glad I at least reported my problem, and mentioned where I found answers. Often times when looking around the web for issues similar to mine I see the thread die with "nevermind I found a fix". Also, thanks for the quick response, @bmeeks