Suricata Not Blocking legacy mode

bmeeks

The only thing to do at this point would be to compile a special debug version of the Suricata binary, copy your Pass List IPs (all 158 of them) into the configuration, and then generate some packets having the SRC/DST IP addresses you posted earlier that are not blocking. It would take manually stepping through the binary code using the gdb debugger package to see what's really going on.

bmeeks

It will take me some time to compile the debug binary and set up the necessary debugging environment. Give me a few days to set this up and test. I will see if there is a bug in the code someplace. It could be in my custom blocking plugin, or it could be within the Radix Tree code that comes from Suricata upstream.

I will post back what I can find out, but it will be several days.

bmeeks

In the meantime, you might find that Inline IPS mode on the WAN interface will work well for you. You can implement your "Pass List" by using PASS rules in the Custom Rules category available on the RULES tab. I posted a link earlier to the Suricata documentation on using PASS rules.

everfree

I test Inline mode 3 years ago, i use intel ix0 nic, run it, system crash finally. 3 years ago, i use name(ntct) and discuss with you. Finally I select use Legacy Block Mode, and it works stablely. My Firewall has 2-3G bps traffic Realtime, and 800,000 states. 172 elementary and junior high schools traffic pass through this firewall.

bmeeks

@everfree said in Suricata Not Blocking legacy mode:

I test Inline mode 3 years ago, i use intel ix0 nic, run it, system crash finally. 3 years ago, i use name(ntct) and discuss with you. Finally I select use Legacy Block Mode, and it works stablely. My Firewall has 2-3G bps traffic Realtime, and 800,000 states. 172 elementary and junior high schools traffic pass through this firewall.

Inline IPS Mode is picky as it depends on the NIC driver adequately supporting the FreeBSD netmap driver. A user created a tutorial for tuning some NIC parameters to improve netmap performance. Here is a link to the Sticky Post: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces. Even though the material primarily talks about em and igb drivers, some of the parameters are likely applicable to the ix driver as well.

Suricata 5.x, when it comes out, will have a rewritten netmap driver interface. That code has been rewritten by the upstream Suricata developer team. Maybe it will perform better with Inline IPS Mode..

I will attempt to reproduce and debug the issue you are having with Legacy Mode Blocking and Pass Lists.

bmeeks

@everfree:
I was able to compile a debug version of the Suricata binary and test for your issue in a VMware virtual machine. Unfortunately, I have thus far been unable to reproduce your problem. Here are the steps I used while attempting to replicate your setup along with the results of my testing.

Installed Suricata in Legacy Blocking Mode on the WAN interface of a VMware virtual machine. The WAN interface of this VM had the IP address 192.168.10.21 (an IP within my private LAN). This is a pfSense-2.5-DEVEL snapshot virtual machine.
I copied and pasted your Pass List from one of your earlier posts (the one containing 158 entries) into the Suricata configuration. I assigned that Pass List to the WAN Suricata interface on the test virtual machine.
I created the following custom rule within Suricata to test the alerting and blocking of traffic from two of the IP addresses that did not block for you (158.85.63.185 and 4.16.74.232). I used this simple rule so that any traffic sourced from either of those two IP addresses would trigger the alert and subsequent block.

Custom Rule:

alert ip [158.85.63.185, 4.16.74.232] any -> $HOME_NET any (msg:"Test rule triggered"; reference:url,doc.emergingthreats.net/bin/view/Main/CompromisedHosts; classtype:misc-attack; sid:2500000; rev:5175; metadata:affected_product Any, attack_target Any, deployment Perimeter, tag COMPROMISED, signature_severity Major, created_at 2011_04_28, updated_at 2019_08_31;)

I fired up my Kali Linux "blackhat" virtual machine and used nmap to scan the target victim virtual machine (the one at 192.168.10.21). I used the "source address spoof" option in nmap to make my traffic appear to come from one of the two IP addresses that did not block for you. Here are the two nmap commands I used:

nmap -e eth0 -S 158.85.63.185 192.168.10.21
nmap -e eth0 -S 4.16.74.232 192.168.10.21

Those commands, when executed, resulted in these alerts on the target virtual machine (there is a lag of a few seconds between the first alert and the block in each case because I was manually stepping through the code in the debugger one line per step):

SuricataBlockTest_158.85.63.185_alerts.png

SuricataBlockTest_4.16.74.232_alerts.png

Those alerts then also resulted in these blocks (note that I cleared out all previous alerts and blocks when executing each test, so the screenshots are from two separate test runs: one with each IP address):

SuricataBlockTest_158.85.63.185_blocks.png

SuricataBlockTest_4.16.74.232_blocks.png

So as you can see above, the IP addresses that failed for you worked as expected in my testing. At this point I'm not sure what else I can try. Perhaps you can repeat my tests with your own Kali Linux virtual machine and see if you get the same result. You can try with my custom rule. I chose the custom rule route because I don't have the software necessary to generate an actual malware signature such as would be required to trigger the ET MALWARE rule you have in your screenshots earlier in this thread. However, which particular rule triggers should not matter when it comes to blocking. The blocking code works only with the packet IP addresses (either SRC, DST or BOTH as configured on the INTERFACE SETTINGS tab).

everfree

Block Both

bmeeks

@everfree :
I replicated a bridge setup that I think is like yours and I still could not reproduce your issue using the previous two IP addresses I tested with (158.85.63.185 and 4.16.74.232).

On your live firewall, have you actually checked the snort2c table to verify if the IP addresses that should be blocked are in fact missing from there as well? You check this under DIAGNOSTICS > TABLES and choosing snort2c as the table to display.

I agree that in your screen capture above, the "non-highlighted" IP addresses should indeed have been blocked, but apparently they were not. Or at least the PHP code in the GUI thinks they are not and thus is not showing the red X beside them.

I really have no idea what could be happening. Your firewall has a lot of traffic if your realtime load is 2-3 Gbps. I'm beginning to wonder if it is a loading issue. I have not been able to reproduce your problem in my test setup, but I cannot simulate that level of load in my test lab. Suricata being a multithreaded application makes debugging more difficult. There is also a possibility that the internal signal flow within the Suricata binary is not working correctly under load. There are two separate modules at work. One is stock from upstream and that module does the logging (writes the alerts to a specified log file). The other module is a custom output plugin I wrote that is supposed to get a copy of every alerting packet. Maybe that is not always happening under heavy load. Maybe the logging module gets the packet but my blocking module (the custom output plugin) does not get a copy of every single packet and thus misses the opportunity to block the traffic. Just theorizing here on possibilities. I will keep looking into the binary code to see if I spot someplace a problem like you are experiencing could creep in.

everfree

where is the code about the custom output plugin??

I don't think it is a loading issue, because I can use it before.

bmeeks

@everfree said in Suricata Not Blocking legacy mode:

where is the code about the custom output plugin??

I don't think it is a loading issue, because I can use it before.

The patch *.diff file is here: https://github.com/pfsense/FreeBSD-ports/blob/devel/security/suricata/files/patch-alert-pf.difflink text.

If you want to examine the final patched source code, download a copy of the Suricata binary source tree version 4.1.4 from here. Untar that archive into an empty directory, then download the patch-alert-pf.diff file linked above and apply the patch to the source code tree. After doing all that, the source for the custom blocking plugin will be in the src/alert-pf.c and src/alert-pf.h files.

I've examined the code in the custom blocking plugin, and I have yet to find a way for the problem you describe to manifest itself. With me using your exact Pass List values, that should have reproduced the problem if it was something directly within the custom module. The fact it did not leads me to theorize it might be a load issue. Perhaps, under heavy load, my custom blocking plugin is not really seeing all the traffic. The alert log is taken directly from Suricata's alert-fast log module's output. So it would be theoretically possible for an alert to get logged but no block happen if the logging module saw the alert but my custom blocking plugin did not.

bmeeks

@everfree said in Suricata Not Blocking legacy mode:

where is the code about the custom output plugin??

I don't think it is a loading issue, because I can use it before.

But there have also been quite a number of changes within other parts of the Suricata binary over the last few years upstream that are not directly part of the custom blocking plugin used on pfSense. This makes it hard to nail down what might be the culprit; especially when the problem is not reproducible in a test environment.

everfree

This post is deleted!

nn14

This post is deleted!

nn14

Hi bmeeks:
Do you know how to confirm that the custom blocking plugin may lose alerts?

bmeeks

I have submitted a Pull Request with the custom blocking module changes that should hopefully address the "no blocks" issue identified in this thread. I've asked that the pull request be merged this Monday, September 30th. So a new Suricata package (version 4.1.5) should show up for the pfSense-2.4.4_p3 RELEASE branch sometime Monday.

everfree

Yes, it works, it's back back back.
thanks. bmeeks.

bmeeks

@everfree said in Suricata Not Blocking legacy mode:

Yes, it works, it's back back back.
thanks. bmeeks.

You're welcome. I'm still puzzled why that variable was not always getting set to NULL in the SCRadixFindKeyBestMatchIPv4() function when the IP was not in a Pass List. I need to study that function carefully to see what's going on. Might be a bug within that code that needs reporting upstream.

nn14

Dear bmeeks:
We appreciate your effort to solve this issue,
Thanks for your significant contribution to this community.
Thank you!

everfree

still have some loss, sad >.<

everfree

Still waiting, hope it will be fixed.