Suricata sometimes block

NickFree

Hi,
I have a problem using Suricata: all works fine for days until something goes wrong and the traffic blocked.
So, i restart Suricata and then it works. This problem appears every 7-10days and i don't know why.
Do you have some ideas?

bmeeks

You gave us such a wealth of information about your problem that it is going to take us several days to sort through all of it and positively identify the root cause for you... .

Seriously, and I apologize for the initial sarcasm, but I could not help myself.... We have nothing to go on other than every 7 to 10 days you get a block. How can we troubleshoot with only that limited amount of information?

Which blocking mode are you using (Inline or Legacy)?
Which rule SID is blocking?
To save us from having to look it up, is the rule from the Emerging Threats or Snort Subscriber rules?
Is the rule perhaps a false positive in your environment?
What hardware are you running pfSense on?
What version of pfSense and the Suricata package are you running?

Simply restarting Suricata does not remove blocked hosts. So we need some more detail there. Perhaps if you are running Inline IPS mode with a NIC that either does not support or poorly supports netmap, that could be an issue. Stopping and starting Suricata in that instance would cycle the NIC's netmap connection and restore connectivity. But if that is your issue, it's not really Suricata "blocking". It would be your NIC driver crashing with netmap.

NickFree

@bmeeks no problem, sorry for my generic first post :)

So:

inline mode
I cannot find a specific rule that is blocking, this is the problem. Simply, all traffic does not flow anymore throught the pfSense. No alerts reported. Is seems some kind of "full block" of the engine.
pfSense is running on VmWare / Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz / AES-NI CPU Crypto: Yes (active)
pfSense 2.4.4-RELEASE-p3 (amd64) / FreeBSD 11.2-RELEASE-p10
Suricata 4.1.5

The netmap can be an idea (VMWare...). But I did run this pfSense a lot of months before to install Suricata, and we didn't have any problems. Only after Suricata setup the problem has started.

Thanks for your help!

bmeeks

@NickFree said in Suricata sometimes block:

@bmeeks no problem, sorry for my generic first post :)

So:

inline mode

I cannot find a specific rule that is blocking, this is the problem. Simply, all traffic does not flow anymore throught the pfSense. No alerts reported. Is seems some kind of "full block" of the engine.

pfSense is running on VmWare / Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz / AES-NI CPU Crypto: Yes (active)

pfSense 2.4.4-RELEASE-p3 (amd64) / FreeBSD 11.2-RELEASE-p10

Suricata 4.1.5

The netmap can be an idea (VMWare...). But I did run this pfSense a lot of months before to install Suricata, and we didn't have any problems. Only after Suricata setup the problem has started.

Thanks for your help!

netmap incompatibility is likely your issue. The symptom of that problem is either a total loss of network connectivity or (sometimes) a box crash.

Suricata with Inline IPS Mode uses a native FreeBSD feature called netmap. You can Google it to learn about it. It is a kernel-level device that allows a user land application to perform high-speed network operations by injecting itself between the physical NIC driver and the network stack in the kernel. In order for this to work well, the physical NIC driver must fully support netmap operation. Only a tiny handful do, and even those still sometimes need some tuneables configured. Check out this Sticky Post at the top of the IDS/IPS sub-forum. These are the only drivers officially supporting netmap operation: em, igb, ixgb, ixl, lem, re and cxgbe.

You may experience better long-term reliability by switching to Legacy Mode blocking and then ticking the "Block DROPS Only" option. Using the "Block DROPS Only" option will allow you to still have some rules just alert while only rules you have changed to DROP will actually block. So the combination of Legacy Mode blocking along with "Block DROPS Only" emulates Inline IPS operation for the most part. Of course it's not exactly the same because Legacy Mode uses libpcap and blocks not by actually dropping packets but instead by inserting the offender's IP address into a pf table for the firewall to block.

NickFree

@bmeeks said in Suricata sometimes block:

ou may experience better long-term reliability by switching to Legacy Mode blocking and then ticking the "Block DROPS Only" option.

Thank you for your explanation!

I've changed to Legacy + Block DROPS only, and will check in some days if this will be better.

Just a couple of related questions:

it inserts the IP into the pf table, but this entry will be permanent, or will expiry after a while?
my environment runs "em" nics. Does make sense I try to make it stable using suggestions in the post you pointed, or it will better to avoid it and run as Legacy?

thanks

bmeeks

@NickFree
Once an IP address is inserted into the pf table, the block stays in place until the IP is removed. There are several ways for the IP to get removed. All but one are manual. For that one automatic process, you need to set the Remove Blocked Hosts Interval drop-down on the GLOBAL SETTINGS tab to some value other than Never. I recommend a short interval such as 15 minutes, 30 minutes or at most one hour. That setting creates a cron task that runs every 5 minutes. The task checks the IP addresses in the table and removes them if a given IP address has not seen additional traffic during the interval configured (for instance, a given blocked IP has not seen new traffic for the last 15 minutes if you have the 15-minute interval selected).

By the way, that pf table name is snort2c. You can examine its contents using the DIAGNOSTICS > TABLES menu option in pfSense.

You can certainly try the Sticky Post settings with your em driver in the virtual machine. They may help. Some updates to the netmap code within the Suricata binary is coming with the release of Suricata 5.0.0 last week. The upstream developer rewrote the section of Suricata code that deals with netmap. I'm working on bringing Suricata 5.0 to pfSense in the near future.

NickFree

After 1 week, pfsense crashed and it's impossible to access on web GUI.
So, I restart it through console and after I stop suricata.
I don't find any logs :(
Do you have any idea?

bmeeks

@NickFree said in Suricata sometimes block:

After 1 week, pfsense crashed and it's impossible to access on web GUI.
So, I restart it through console and after I stop suricata.
I don't find any logs :(
Do you have any idea?

Can you be more specific with the statement "I don't find any logs"?

What kind of logs? pfSense system logs, Suricata logs, dmesg.boot logs? What logs did you not find and where did you look for them?
Are you running any other packages besides Suricata? If so, what are they?
Legacy Mode has never been known to cause pfSense to lockup. Inline IPS Mode, yes, but never Legacy Mode. Are you positive Suricata is the culprit? You may have another hardware or package issue.
Before rebooting the firewall, did you try simply restarting the Web Configurator from the console option?
Did the firewall stop routing all traffic, or did just the GUI login fail?
Try leaving Suricata disabled for a week and see if you still get a crash.

NickFree

@bmeeks

1: any log reporting something that can explain why it did stop responding
2: only open-vm-tools and zabbix-agent-42. They are up from many months and no problem before I introduced Suricata
3: I cannot be sure. PFSense is on a VMWare installed on a primary cloud provider, so I think it cannot be "hardware". I point to Suricata because I used this firewall a lot of months without any problem, until I started using Suricata (and nothing other change).
4: yes. Nothing changed.
5: all traffic stopped. No way to contact any server behind the firewall, nor to contact pfSense.
6: yes. I stopped it yesterday and this is my current plan :(

thanks

NickFree

Uptime 21 Days 16 Hours 14 Minutes 42 Seconds

no more problems after I disabled suricata... :(