(SOLVED) Suricata Interfaces have to be manually Restarted

Raffi_

Yes, it's an IPV4 address. Are there any other logs that might help?

This isn't something a system reboot might fix? I like to avoid that, but I might give it a try if all else fails.

To clarify, I'm not sure if I might have two different problems at least according to the suricata error logs.
Problem 1, when my WAN2 IP changes, I get the tree null errors. I'm not sure how that error manifests itself in terms of software issues. Is that what prevented my alert from not being blocked?

Problem 2, when suricata performs a rules update, the status on both my LAN and OPT1 interface go from the green check to the red x and each has to be manually restarted. That was indicated in the log by the "[ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2"

In case it's relevant, here are my packages.

bose301s

Mine seems to be working now, will be keeping an eye on it.

digdug3

I have the same problem, Suricata is not restarting after an update on all the interfaces. Also tried reinstalling the package.
I can start the interfaces manually.

MD5: c962d5d995867c5baf3136035a34fac7

bmeeks

@digdug3 said in Suricata Interfaces have to be manually Restarted:

I have the same problem, Suricata is not restarting after an update on all the interfaces. Also tried reinstalling the package.
I can start the interfaces manually.
MD5: c962d5d995867c5baf3136035a34fac7

What messages do you see in the suricata.log file for the interfaces that do not restart? I must know what errors are printing to even know where to begin looking for the problem.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Yes, it's an IPV4 address. Are there any other logs that might help?

This isn't something a system reboot might fix? I like to avoid that, but I might give it a try if all else fails.

To clarify, I'm not sure if I might have two different problems at least according to the suricata error logs.
Problem 1, when my WAN2 IP changes, I get the tree null errors. I'm not sure how that error manifests itself in terms of software issues. Is that what prevented my alert from not being blocked?

Problem 2, when suricata performs a rules update, the status on both my LAN and OPT1 interface go from the green check to the red x and each has to be manually restarted. That was indicated in the log by the "[ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2"

In case it's relevant, here are my packages.

Let's first make sure that you don't have a duplicate or zombie Suricata process running. Stop all Suricata instances via the GUI controls, and then open a shell session on the firewall and run this command:

ps -ax | grep suricata

You should see only a single line display (the grep command). If you see any other lines with suricata in them, then kill those processes and try restarting Suricata from the GUI.

Also, at any time in the past have you run a command to manually install Suricata from the shell prompt? Something like pkg install pfSense-pkg-suricata? Don't do that now, but if you ever did it in the past I found when testing a Snort change a while back that the old copy of the binary installed by the manual pkg install process hangs around even when you do a delete and reinstall from the GUI.

bmeeks

@bose301s said in Suricata Interfaces have to be manually Restarted:

Mine seems to be working now, will be keeping an eye on it.

Thank you for the feedback. Glad it seems to be working for you. As I mentioned in a previous post, I can't make it fail for me on either a 2.4.4. or a 2.5 pfSense machine.

digdug3

@bmeeks
Here is the log:

18/6/2019 -- 06:13:53 - <Notice> -- Signal Received.  Stopping engine.
18/6/2019 -- 06:13:53 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2 
18/6/2019 -- 06:13:53 - <Info> -- time elapsed 29316.584s
18/6/2019 -- 06:13:53 - <Info> -- (RX#01-vtnet1) Packets 433047, bytes 110221491
18/6/2019 -- 06:13:53 - <Info> -- (RX#01-vtnet1) Pcap Total:433049 Recv:433049 Drop:0 (0.0%).
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- Alerts: 0
18/6/2019 -- 06:13:53 - <Info> -- cleaning up signature grouping structure... complete
18/6/2019 -- 06:13:53 - <Notice> -- Stats for 'vtnet1':  pkts: 433047, drop: 0 (0.00%), invalid chksum: 0

And another one:

18/6/2019 -- 06:13:55 - <Notice> -- Signal Received.  Stopping engine.
18/6/2019 -- 06:13:55 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2 
18/6/2019 -- 06:13:55 - <Info> -- time elapsed 29297.555s
18/6/2019 -- 06:13:55 - <Info> -- (RX#01-vtnet0) Packets 19190012, bytes 10423765489
18/6/2019 -- 06:13:55 - <Info> -- (RX#01-vtnet0) Pcap Total:19194208 Recv:19189936 Drop:4272 (0.0%).
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- Alerts: 0
18/6/2019 -- 06:13:55 - <Info> -- cleaning up signature grouping structure... complete
18/6/2019 -- 06:13:55 - <Notice> -- Stats for 'vtnet0':  pkts: 19190012, drop: 4272 (0.02%), invalid chksum: 0

And here the manual start of that last interface:

18/6/2019 -- 08:28:37 - <Notice> -- This is Suricata version 4.1.4 RELEASE
18/6/2019 -- 08:28:37 - <Info> -- CPUs/cores online: 4
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for MPM
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for SPM
18/6/2019 -- 08:28:37 - <Info> -- HTTP memcap: 67108864
18/6/2019 -- 08:28:37 - <Notice> -- using flow hash instead of active packets
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
--- CUT ---
18/6/2019 -- 08:28:37 - <Info> -- alert-pf output device (regular) initialized: block.log
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Pass List /usr/local/etc/suricata/suricata_38383_vtnet0/passlist parsed: 31 IP addresses loaded.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Firewall interface IP address change notification monitoring thread started.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf output initialized, pf-table=snort2c  block-ip=both  kill-state=on  block-drops-only=on
18/6/2019 -- 08:28:37 - <Info> -- fast output device (regular) initialized: alerts.log
18/6/2019 -- 08:28:37 - <Info> -- http-log output device (regular) initialized: http.log
18/6/2019 -- 08:28:37 - <Info> -- stats output device (regular) initialized: stats.log
18/6/2019 -- 08:28:37 - <Info> -- Syslog output initialized
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for SPM
18/6/2019 -- 08:28:37 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'sip_method'.
--- CUT ---
18/6/2019 -- 08:28:39 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert udp $EXTERNAL_NET any -> $SIP_SERVERS $SIP_PORTS (msg:"PROTOCOL-VOIP Known SIP scanner User-Agent detected"; flow:to_server; sip_header; content:"User-Agent: sip-scan"; fast_pattern:only; metadata:policy max-detect-ips drop, service sip; reference:url,blog.kolmisoft.com/sip-attack-friendly-scanner/; classtype:attempted-recon; sid:48309; rev:2;)" from file /usr/local/etc/suricata/suricata_38383_vtnet0/rules/suricata.rules at line 21407
18/6/2019 -- 08:28:39 - <Info> -- 2 rule files processed. 22211 rules successfully loaded, 203 rules failed
18/6/2019 -- 08:28:39 - <Warning> -- [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2000419, gid 1: unknown rule
--- CUT ---
18/6/2019 -- 08:28:39 - <Warning> -- [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200004, gid 1: unknown rule
18/6/2019 -- 08:28:39 - <Info> -- Threshold config parsed: 16 rule(s) found
18/6/2019 -- 08:28:40 - <Info> -- 22223 signatures processed. 11 are IP-only rules, 8729 are inspecting packet payload, 15311 inspect application layer, 0 are decoder event only
18/6/2019 -- 08:28:40 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.GenericPhish_Excel' is checked but not set. Checked in 2023046 and 0 other sigs
--- CUT ---
18/6/2019 -- 08:28:40 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.docm' is checked but not set. Checked in 43975 and 1 other sigs
18/6/2019 -- 08:28:44 - <Info> -- Using 1 live device(s).
18/6/2019 -- 08:28:44 - <Info> -- using interface vtnet0
18/6/2019 -- 08:28:44 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
18/6/2019 -- 08:28:44 - <Info> -- Set snaplen to 1518 for 'vtnet0'
18/6/2019 -- 08:28:44 - <Info> -- RunModeIdsPcapAutoFp initialised
18/6/2019 -- 08:28:44 - <Notice> -- all 5 packet processing threads, 2 management threads initialized, engine started.

Had to cut some "flowbit error messages" and "pass list" to keep the post short

bmeeks

@digdug3: I suspect you may have the same underlying problem as another user in a similar thread. He had multiple instances of Suricata running on the same interface, and they were running with memory resident copies of the old binary.

Those PCAP_DISPATCH errors look suspiciously like what I would expect if multiple Suricata instances were trying to execute on the same interface. PCAP might get confused. You can walk through the steps below to check for and kill any duplicate runaway processes, or you can try rebooting the firewall.

Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.
Open a shell prompt on the firewall and execute this command:

ps -ax | grep suricata

It should return only a single line showing the grep command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the single grep command.
Now execute this command at the shell prompt:

ls /usr/local/bin/suricata*

It should find no file by that name. If it does, then delete the file it finds.
Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.

digdug3

@bmeeks Thanks for the info. Indeed there were some old instances still running after package deletion. Had to kill them using

kill -9 PID

A normal kill did not work.

After package reinstalling all interface came up automatically (as expected).
For your info, the running processes after package deletion:

ps -ax | grep suricata

76850  -  INs      6:14.59 /usr/local/bin/suricata -i vtnet1 -D -c /usr/local/etc/suricata/suricata_22491_vtnet1/suricata.yaml --pidfile /var/run/suricata_vtnet122491.pid
77805  -  INs     17:18.04 /usr/local/bin/suricata -i vtnet0 -D -c /usr/local/etc/suricata/suricata_38383_vtnet0/suricata.yaml --pidfile /var/run/suricata_vtnet038383.pid
78845  -  INs     15:48.93 /usr/local/bin/suricata -i vtnet3 -D -c /usr/local/etc/suricata/suricata_21975_vtnet3/suricata.yaml --pidfile /var/run/suricata_vtnet321975.pid
79709  -  INs      5:29.14 /usr/local/bin/suricata -i vtnet0.10 -D -c /usr/local/etc/suricata/suricata_4336_vtnet0.10/suricata.yaml --pidfile /var/run/suricata_vtnet0.104336.pid
22120  0  S+       0:00.00 grep suricata

And the ls:

ls /usr/local/bin/suricata*
ls: No match.

I think this fixes the not starting of the interfaces, if not, I'll let you know. Thank you.

Raffi_

@bmeeks Success! Thanks a lot for all the help on this. I was having the exact same issue as @digdug3. Thanks to the info you posted about killing the processes that solved it.

And thank you @digdug3 for the hint on how to kill those processes. I was having the same issue as you with killing them.

FYI, I had a bunch of the processes running which had to be killed. I did not grab the output but it was something like 5 per interface. I think each time I uninstalled and reinstalled, it started new processes for each interface.

I can confirm with pretty high confidence that both of my issues are solved.

Previously when I forced an update, that would cause the issue of suricata not starting back up on the interfaces.
Now, when I do that there is no error in the log and suricata was still running on both interfaces.

The other tree null issues is gone as well. Previously I was able to reproduce that by rebooting the 4G LTE modem and when it got a new address, those errors would show up.
Now, there are no errors related to that.

I'll mark this as solved.

Thanks again!
Raffi

bmeeks

Glad everything is working again. I was pretty certain I had fixed that bug, and the fact you were still seeing it was truly puzzling. I finally thought to have you check for runaway processes, and that turned out to be the problem. Those runaway processes would all be running memory-resident versions of the old Suricata binary (the one with the bug).

Runaway processes can occur from time to time with the way so many things kick off a pfSense "restart all packages" command. So when you see weirdness or recurring problems, it's not a bad idea to pull a Microsoft Windows routine and just reboot the firewall. That will for sure clear out any runaway processes.

digdug3

@bmeeks Would it be possible to check for those processes during the uninstallation of the package? So you can warn the user?

Raffi_

@digdug3 said in (SOLVED) Suricata Interfaces have to be manually Restarted:

@bmeeks Would it be possible to check for those processes during the uninstallation of the package? So you can warn the user?

That's a good idea. How about auto-kill any found processes during the uninstall? Would that be possible?

bmeeks

@digdug3 said in (SOLVED) Suricata Interfaces have to be manually Restarted:

@bmeeks Would it be possible to check for those processes during the uninstallation of the package? So you can warn the user?

It tries to do that now, but does so by looking for a PID file in /var/run. A crashed/runaway process may no longer have a valid PID file in /var/run.

I can look at some other approaches using pgrep maybe to find all suricata processes and then force kill them. Me with Suricata and Snort, and the former package maintainers before me for Snort, have been struggling with multiple instances getting launched forever. It stems somewhat from the way pfSense itself will sometimes issue multiple "restart all packages" commands in response to WAN IP changes or changes in the WAN interface state or problems with a gateway. These multiple "restart all packages" commands can lead to multiple instances of Suricata (or Snort) running on an interface.

Raffi_

@bmeeks said in (SOLVED) Suricata Interfaces have to be manually Restarted:

@digdug3 said in (SOLVED) Suricata Interfaces have to be manually Restarted:

@bmeeks Would it be possible to check for those processes during the uninstallation of the package? So you can warn the user?

It tries to do that now, but does so by looking for a PID file in /var/run. A crashed/runaway process may no longer have a valid PID file in /var/run.

I can look at some other approaches using pgrep maybe to find all suricata processes and then force kill them. Me with Suricata and Snort, and the former package maintainers before me for Snort, have been struggling with multiple instances getting launched forever. It stems somewhat from the way pfSense itself will sometimes issue multiple "restart all packages" commands in response to WAN IP changes or changes in the WAN interface state or problems with a gateway. These multiple "restart all packages" commands can lead to multiple instances of Suricata (or Snort) running on an interface.

That sounds like a pain to deal with. It would be great if another approach could be implemented.