Suricata unwanted blocking

Stewart

I’ve had this off and on problem with Suricata running on pfSense where it will block IPs that exist on the pass list. I’ve researched and posted in the forums before but have had limited success in permanently resolving the error. I've restarted the entire Suricata service as well as on each interface. I can check and see the IP is indeed in the pass list. If I (on the most recent one with the problem) run

cat /usr/local/etc/suricata/suricata_28922_igb1/passlist

Then I can see the IP in the list (it’s actually the IP/32 since that’s how it’s put it in case I need to expand the IP range). What else can I check or try? Yesterday I disabled the rule entirely so when it happened this morning it wound up blocking a whitelisted IP using a disabled rule. It just feels like there is a disconnect between pfSense and Suricata. As you can see below, in the Alerts page on pfSense it even has the indicators that the rule is disabled and the alerts for it suppressed.

The Alert:

The Block:

The Pass List

I really don't know where to go at this point.
pfSense: 2.4.5-p1
Suricata: 5.0.2_3

bmeeks

My bet is you have a duplicate Suricata instance running on the same interface, and that instance is not going to see or honor any changes you make in the Pass List. Duplicate instances can occur via several mechanisms. First, if you have Service Watchdog installed on your system, DO NOT let it monitor or mess with Suricata. Suricata and Snort are quite complex in the way the operate, and both packages routinely shut themselves down and automatically restart as part of periodic rules updates. A package like Service Watchdog sees the daemon go down and immediately sends a restart command without realizing that the IDS/IPS package might simply be restarting on its own after a rules update. Thus two instances of the IDS/IPS wind up getting launched on the same interface, and one of them is going to totally ignore anything you do in the GUI. Another mechanism that can cause duplicate instances on an interface is pfSense issuing the "restart all packages" command several times in quick succession. Things that can cause this are bouncing interfaces (WAN going down and back up, for example, or VPN tunnels changing, etc.).

Here is how to check for duplicate processes --

Execute this command from a shell prompt either directly via the firewall console or via SSH:

ps -ax | grep suricata

That command should show exactly one running Suricata process for each configured interface. If you see duplicate lines in the output of the above command, then you have multiple processes on the same interface. To get rid of them, do this:

In the GUI on the INTERFACES tab, stop all Suricata instances. Go back to the shell prompt and run this command again:

ps -ax | grep suricata

Note the Process ID of any remaining Suricata instances. For each one you see, run this command:

kill -9 <pid>

where <pid> is the Process ID identified in the earlier step.

Now go back to the GUI and start Suricata on the configured interfaces. It should honor your Pass List and disabled rules now. About the only way Suricata can possibly still alert on a disabled rule is if there is a duplicate process running that the GUI is not aware of (because the GUI did not start it).

Stewart

@bmeeks Thanks for the reply. There may some merit to that idea. I started top and then went in to stop the service. It spun for a bit and then showed that it was still running. The processes still showed in top as well. I stopped it again and after a minute or so it finally showed that it stopped and all the processes cleared out of top. I then started it again and it hasn't blocked again, yet. I'm not exactly sure what that indicates but it could be that telling in telling it to restart it wound up starting a new process without successfully stopping the old one first. Next time I see it again I'll give your suggestion a shot and reply to this thread.

bmeeks

Almost always, when Snort or Suricata seem to ignore changes to a Pass List or disabling of a rule, it indicates a duplicate instance is running on the interface. The GUI has control of one, but not the other due to different process IDs. Everything the GUI does is governed by the process ID. So if the GUI did not start that second process, it does not know what the Process ID is and can't control it.

Also, Snort and Suricata read their respective configuration file at startup. When the GUI codes makes certain changes (like enabling or disabling a rule), a SIGUSR2 signal is sent to the running Suricata interface instance by using the process ID the GUI obtained when it launched that instance. But if there is another instance running on the interface with a different process ID, it won't see the signal to reload its configuration and thus will ignore the change to the enabled or disabled rule (or altered Pass List content).

Stewart

@bmeeks That makes a ton of sense. So, when I encounter this in the future I should see Suricata processes still running even after stopping the service. In that case I would follow your

ps -ax | grep suricata
kill -9 <pid>

And then restart the service. Is that correct?

As an aside, I've always just done kill <pid>. Why the -9? I've tried to read the manual but I'm not clear on it. I've understood that kill <pid> ends the task and kill -s HUP <PID> sends the shutdown command to the process so it can close gracefully. I could be way off, though.

bmeeks

@Stewart said in Suricata unwanted blocking:

@bmeeks That makes a ton of sense. So, when I encounter this in the future I should see Suricata processes still running even after stopping the service. In that case I would follow your
ps -ax | grep suricata
kill -9 <pid>
And then restart the service. Is that correct?

As an aside, I've always just done kill <pid>. Why the -9? I've tried to read the manual but I'm not clear on it. I've understood that kill <pid> ends the task and kill -s HUP <PID> sends the shutdown command to the process so it can close gracefully. I could be way off, though.

Yes, you are correct on the sequence of commands.

The "kill -9" version means "kill the SOB no matter what, even if it does not acknowledge we are killing it" ... . I had some failures in the past where the plain "kill <pid>" version would fail to kill an errant process. The "-9" switch worked, though. "kill <pid>" sends a SIGTERM to the process and waits for the process to acknowledge. The process is free to ignore the command if it wants to. The "-9" switch is an unconditional kill command that forces the program to end and cannot be ignored by the process.

Suricata is different from many other apps. When you send it a SIGHUP, that is the code for "refresh all log files". You send it SIGHUP when you have rotated the logs. To stop Suricata normally, you send it SIGTERM. If it ignores the SIGTERM, then you can go the "-9" route.

AppXprt

@bmeeks
There may actually be multiple suricata processes running in which case you could kill each PID of each suricata individual process or you can kill all of them easier in one command with pkill:
pkill suricata