Suricata pass list ignored

j.koopmann

Hi,

I have a working suricate setup. Legacy mode on LAN interface. Blocking is on BOTH. On the LAN interface I have several VLANs configured. The pass list has all local networks including the VLAN networks. However if an attack is detected, source (outside) and dst (internal IP on one of the VLANs) are blocked.

Yes, I restarted suricata. And the interface. Several times. And yes I did check if several suricata processes are running. Choosing "default" as a pass list does not change this. Could this be related to the VLANs?

Regards
JP

bmeeks

So on the BLOCKED tab you see both IP addresses listed? That means the external one and the internal VLAN address?

If the VLAN subnet is properly configured in the Pass List, and that Pass List is assigned to the interface (this does not automatically happen just because you create a Pass List, you still must explicitly choose it in the drop-down selector in the Pass List area of the INTERFACE EDIT tab), then only the external IP address should be showing up on the BLOCKED tab. The most common error seen with an issue like you are describing is the user forgetting to explicitly assign the custom Pass List on the INTERFACES EDIT tab for the chosen interface. Another common error is forgetting to restart Suricata after changing the Pass List assignment. However, you state you have restarted Suricata.

One other more rare occurence is the existance of a duplicate Suricata instance on the interface. If that is the case, then the duplicate instance is outside the control of the GUI and that instance will not respond to GUI changes. You can check for this by running this command from a shell prompt on the firewall:

ps -ax | grep suricata

You should see exactly one running Suricata instance for each interface. If you see more, then you have a duplicate and will need to kill it. The easiest way to do this is to stop all running Suricata instances on the INTERFACES tab in the GUI, then go back to the shell prompt and run the command above again. Note the PID (process ID) of any remaining Suricata process (or processes) and kill each of them using

kill -9 <pid>

where <pid> is the process ID of the Suricata process.

j.koopmann

@bmeeks thanks. I am aware of all of these facts and triple checked. I killed all suricata processes on the cli. I assigned the pass list. The subnet is part of the pass list. And still both sides incl the IP on the block list are blocked. That is why I came here...

j.koopmann

To clarify:

@bmeeks said in Suricata pass list ignored:

So on the BLOCKED tab you see both IP addresses listed? That means the external one and the internal VLAN address?

Yes.

If the VLAN subnet is properly configured in the Pass List,

It is. The correct /24 is in the Pass List. It also shows under "View List" in Suricata -> Interface -> LAN -> Pass List.

and that Pass List is assigned to the interface (this does not automatically happen just because you create a Pass List, you still must explicitly choose it in the drop-down selector in the Pass List area of the INTERFACE EDIT tab)

Was this way all along.

, then only the external IP address should be showing up on the BLOCKED tab.

Correct. It should. I can replicate this any time. Also the internal private IP which is in the pass list is blocked.

The most common error seen with an issue like you are describing is the user forgetting to explicitly assign the custom Pass List on the INTERFACES EDIT tab for the chosen interface.

Not here

Another common error is forgetting to restart Suricata after changing the Pass List assignment. However, you state you have restarted Suricata.

Indeed.

One other more rare occurence is the existance of a duplicate Suricata instance on the interface. If that is the case, then the duplicate instance is outside the control of the GUI and that instance will not respond to GUI changes.

I have read this in other threads as well which is why I manually killed all suricata processes on the pfsense CLI and restarted. Still same result.

I appreciate you taking the time to help. Did not want to sound rude. It is just that I already followed all of these trails after intensive study and debugging. And I am aware that most likely 50+% of people asking this very same question had not done this. :-)

I also checked that the pastlist file in the correct subdirectory unter /usr/local/etc/suricata is correct and shows the network to ignore. It does.

Still the problem persists. Any other idea?

j.koopmann

@bmeeks I just added the VLAN subnet to the ALIAS and in addition the IP in question as a /32 as well. NOW the passlist works. This suggests a bug somewhere....

Before (not working):

192.168.200.0/24
192.168.202.0/24
192.168.205.0/24
192.168.255.0/24

Now (working):

192.168.200.0/24
192.168.202.0/24
192.168.205.0/24
192.168.205.250/32
192.168.255.0/24

Maybe something in the matching of the IPs to networks is incorrect?

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks I just added the VLAN subnet to the ALIAS and in addition the IP in question as a /32 as well. NOW the passlist works. This suggests a bug somewhere....

Before (not working):
192.168.200.0/24
192.168.202.0/24
192.168.205.0/24
192.168.255.0/24
Now (working):
192.168.200.0/24
192.168.202.0/24
192.168.205.0/24
192.168.205.250/32
192.168.255.0/24
Maybe something in the matching of the IPs to networks is incorrect?

Hmm... might well be. Unfortunately that would likely be internal to the Suricata binary itself. More specifically, it would point to a problem with the integral Radix Tree functionality provided by the base binary. The custom plugin used on pfSense for blocking makes use of the Radix Tree utility code within the base Suricata binary to both store and subsequently lookup IP addresses on a Pass List. I will need to create a debug version of the Suricata binary, install it on a virtual machine, and then try to duplicate your issue using the IP address ranges.

That Radix Tree code is quite complicated, and to be honest I have never been able to fully follow its logic. So it is possible that some part of it is not working correctly in the base Suricata binary.

j.koopmann

@bmeeks said in Suricata pass list ignored:

I will need to create a debug version of the Suricata binary, install it on a virtual machine, and then try to duplicate your issue using the IP address ranges.

Thanks very much.

Let me know if I can help (e.g. by installing a debug version on my pfsense and replicate it for you).

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks said in Suricata pass list ignored:

I will need to create a debug version of the Suricata binary, install it on a virtual machine, and then try to duplicate your issue using the IP address ranges.

Thanks very much.

Let me know if I can help (e.g. by installing a debug version on my pfsense and replicate it for you).

Thanks, but installing a debug version on pfSense is not really possible unless you have your own private packages repository and pfSense package builder to create the package. That's because of library dependencies and the need to have everything matched up in order for pkg to actually install the package.

I have your IP address ranges in the posts on this thread, so that should give me what I need to reproduce the issue.

I fought something similar to this about a year ago for a user in Asia and we never got to the bottom of it. It was also related to the Radix Tree code within Suricata. It's difficult to explain in a few words, but the Radix Tree is a device for storing data like network IP subnets and addresses and then looking them back up rapidly. It also can tell if a given IP address or subnet already exists in the radix table by way of being nested within another existing stored subnet. It rapidly gets complicated to explain, and the actual C source code is even more complicated. Hence the issues with troubleshooting it. And that is not code I created, and it's not super well documented either.

j.koopmann

@bmeeks Just had the same issue again. This time it was my production system on the LAN network (which 100% is part of the pass list). I will add the most important devices as /32 for the time being.

What strikes me: If this is a problem, this seems to be a problem with the default pass list as well. This is a pfsense specific implementation is it not? In case it is, I believe this really needs fixing and I am surprised it is not exploding more often. :-)

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks Just had the same issue again. This time it was my production system on the LAN network (which 100% is part of the pass list). I will add the most important devices as /32 for the time being.

What strikes me: If this is a problem, this seems to be a problem with the default pass list as well. This is a pfsense specific implementation is it not? In case it is, I believe this really needs fixing and I am surprised it is not exploding more often. :-)

The blocking module used on pfSense (and the related Pass List feature) is unique to the Suricata package on pfSense. The blocking plugin is custom code I created that is compiled into the Suricata binary as a patch. The module is based on the same basic code used to provide blocking in the Snort package on pfSense. The standalone stock binary package for Suricata has no blocking capability at all (well, unless you configure it for inline IPS using the netmap kernel device). The blocking module works by extracting the IP addresses from alerts and then making a FreeBSD system call to insert those IP addresses into a pre-existing pf firewall table. That table name is snort2c, and it is created by the pfSense GUI code at startup and automatically included as part of some hidden block rules in the pf firewall configuration.

The Pass List functionality is logic that stores IP addresses or subnets within a memory structure for later quick lookup so that the IP addresses pulled from an alerting packet can be compared against the list. IP addresses from alerts that are contained in the Pass List memory structure are not added to the snort2c table and thus not blocked.

In both Suricata and Snort the custom module code uses built-in functionality from the stock binary to store the Pass Lists addresses and later look them up. However, the two IDS package binaries (Snort and Suricata) have totally different logic for implementing that lookup function. Suricata uses a construct called a Radix Tree. Supposedly the tree logic is smart enough to calculate if a given /32 IP is either directly stored in the table, or if it is encompassed by an already stored subnet (for instance, a /24 or something else). I'm beginning to wonder if sometimes that lookup is getting fooled. In the previous case I discussed, I was never able to duplicate the original poster's issue in my test virtual machine. Even when using his exact Pass List. It is that Radix Tree code that is somewhat poorly documented and very difficult to follow when debugging.

I have personally never been able to duplicate these issues when users have sporadically reported them. Not being able to duplicate a bug makes it extremely hard to identify the root cause and fix it (if it exists). As you may imagine, there are many different skill levels among the package users, and often times what they report as a bug turns out to be something they have configured incorrectly. Not saying that is your case, just saying that when you maintain a package as a volunteer contributor there are lots of, shall we say, "challenges" with support. For starters there is no physical way I can have on hand and test for all the various configurations users may have. It very well could be something unique to your configuration.

The above does not mean I won't look for the bug if one exists, but hopefully it illustrates the difficulty in identifying and fixing something that you can't reliably replicate. This is double difficult when dealing with multithreaded code like the Suricata IDS engine.

j.koopmann

@bmeeks said in [Suricata pass list ignored]

Let's start with: I totally get what you are saying. I was a package maintainer myself.

The blocking module used on pfSense (and the related Pass List feature) is unique to the Suricata package on pfSense. The blocking plugin is custom code I created that is compiled into the Suricata binary as a patch. The module is based on the same basic code used to provide blocking in the Snort package on pfSense. The standalone stock binary package for Suricata has no blocking capability at all (well, unless you configure it for inline IPS using the netmap kernel device). The blocking module works by extracting the IP addresses from alerts and then making a FreeBSD system call to insert those IP addresses into a pre-existing pf firewall table. That table name is snort2c, and it is created by the pfSense GUI code at startup and automatically included as part of some hidden block rules in the pf firewall configuration.

That is what I expected or gathered from previous codes.

The Pass List functionality is logic that stores IP addresses or subnets within a memory structure for later quick lookup so that the IP addresses pulled from an alerting packet can be compared against the list. IP addresses from alerts that are contained in the Pass List memory structure are not added to the snort2c table and thus not blocked.

And the Pass List is part of your custom code or partly Suricata? If it is entirely your code it is up to you which type of pass list implementation you use.

Even if it is not your code and you are reacting to alerts from Suricata only it should be relatively easy to import the already existing pass list (you or whoever writes them to disc) into a memory structure and make that your code never issues a pf change it the IP matches.

In both Suricata and Snort the custom module code uses built-in functionality from the stock binary to store the Pass Lists addresses and later look them up.

And my question is: Why? Are you forced to use Suricata implementation logic for the custom module?

As you may imagine, there are many different skill levels among the package users,

I hear you!

and often times what they report as a bug turns out to be something they have configured incorrectly. Not saying that is your case, just saying that when you maintain a package as a volunteer contributor there are lots of, shall we say, "challenges" with support. For starters there is no physical way I can have on hand and test for all the various configurations users may have. It very well could be something unique to your configuration.

Hence my offer to install a debug package. I understand that this is not easily possible. But I could exchange some precompiled binaries/modules on the command line manually bypassing pkg if it helps. As mentioned: I can very easily reproduce this here within 5 minutes. A simple curl to one of my Webservers --> boom. :-)

Regards
JP

bmeeks

@j-koopmann:

The Pass List is entirely part of the custom code. Suricata itself has no concept of, nor use for, the Pass List functionality. Neither does Snort, by the way.

The reason a Pass List even exists is because of the "big hammer" approach used for blocking. This all goes back to the Snort package, so a little history to help you better understand.

Snort many many moons ago had an added plugin feature called snortsam. It was a FreeBSD thing and not related to pfSense directly. It also worked with a number of firewall engines from several operating systems. Details can be found here: http://www.snortsam.net/. That feature on FreeBSD relied on the use of the ipfw firewall engine. And it worked by extracting the IP addresses from Snort alerts and then making system calls to insert those addresses into a firewall rule that would then block subsequent traffic from that host. Key here is the word "subsequent", because this all works by using libpcap to send copies of packets to Snort for analysis. Snort then may generate an alert on the copied packet (or packets in the event a larger stream of data). So that first packet (or packets) made it all the way through the firewall. However, subsequent packets in that stream could be blocked by the new firewall rule (provided the user configured the option to clear states for that IP session when an "alert" was detected and the offending IP was handed off to the firewall to be put in the blocking rule).

pfSense uses pf instead of ipfw for the firewall engine. ipfw is used within the Captive Portal code. So someone in the very distant past created a plugin for Snort called "spoink" that worked similar to the snortsam plugin, but this new one worked with pf instead of ipfw. It accomplished the same thing, but just used the pf firewall engine.

So what does a Pass List have to do with this? Because the "blocking" worked by actually putting a host's IP address in a firewall rule, ALL subsequent traffic to or from that host would be blocked so long as the IP existed in the firewall rule. There was no granularity to the blocking. You could not block just the "bad traffic" but let the other packets through. It was an all or nothing proposition. This could potentially be a bad thing if Snort, for example, put your web server's IP in the blocking firewall rule. So was born the idea of a list of hosts that the blocking module would never block -- the Pass List.

So when the Suricata package came along, I put the same functionality into it (the Pass List) for the same reason. Now when using Inline IPS Mode where individual packets are literally dropped by the kernel netmap device and the firewall is not involved in any manner, there is no need for a Pass List as nobody's IP address ever gets put anywhere. Individual packets are either passed or dropped by the netmap device as it routes them from the NIC driver to the kernel depending on instructions from Suricata. And by the way, that same technology is now available in Snort.

Finally, let's loop around to how the Pass List works. To keep from reinventing the wheel, I simply used the existing functions with the Snort and Suricata binaries for storing IP addresses or subnets in a memory table and then later testing whether a given IP address matches a previously stored address or is contained within a previously stored subnet. This functionality exists within Suricata and Snort because the rules evaluation engine needs it (the ability to match addresses to either discrete values or determine if an address resides within a stored subnet).

The code for matching IP addresses against a Pass List can get a little tricky when you start needing to figure out whether an IP is within a subnet. So long as we are talking a small number of subnets, it's not too bad. But some users have gone crazy with adding stuff to Pass Lists and have hundreds and hundreds of IP subnets, many of which overlap each other. So rather than trying to write my own code to handle all of that, I used existing code that was already right there. And with all my testing (and my subsequent testing), it has always worked for me. I've never had an issue with it in either Snort nor Suricata.

If you want to examine the code, or compile your own version for debugging, here is the link on Github: https://github.com/pfsense/FreeBSD-ports/tree/devel/security/suricata. The custom plugin patch is in the files sub-folder. It's called patch-alert-pf.diff. Apply that patch to an extracted stock source code directory from Suricata-5.0.4 and that's all you need. I would be happy for another set of eyes to go over that code. Maybe they will see something I have missed.

j.koopmann

@bmeeks I guess I am just "oversimplifying" the task to match an IP address to a white list. It was my expectation that most likely a dozen of libraries exist that should be able to do this. I agree that once a crazy amount of pass-list entries come into play, this will make a difference.

I doubt that my set of eyes will make a big difference in analyzing the code esp. if it depends on a complex not well documented library. I fully trust that you are a million times better at this than I. :-)

My suggestion was: If you build a debug version with the output you need and you are able to replicate the issue --> super. If you have a debug version but cannot replicate the issue I come into play.

FreeBSD pfSense.koopmann.local 11.3-STABLE FreeBSD 11.3-STABLE #243 abf8cba50ce(RELENG_2_4_5): Tue Jun 2 17:53:37 EDT 2020 root@buildbot1-nyi.netgate.com:/build/ce-crossbuild-245/obj/amd64/YNx4Qq3j/build/ce-crossbuild-245/sources/FreeBSD-src/sys/pfSense amd64

as long as the executable runs on this platform and you drop me a link to the binary (or I provide an upload space) I can exchange the executable, reproduce things and send all debug output to you. Or you can take a look via TeamViewer etc.

Regards
JP

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks I guess I am just "oversimplifying" the task to match an IP address to a white list. It was my expectation that most likely a dozen of libraries exist that should be able to do this. I agree that once a crazy amount of pass-list entries come into play, this will make a difference.

Well, you could say I did use a ready-made library to do this. I am using the code already existing in the Suricata binary. I saw no need to bulk up my plugin module by either creating the Pass List logic from scratch, or importing yet another third-party library into the mix. And like I said, this code has worked. And every time I have tested it, it continues to work. For that other user I mentioned (which was early last year, if I am recalling correctly), I was never able to reproduce his issue either. I did try some things differently in examining the returned value from the built-in function call, but since I could not reproduce his issue I really couldn't test my theory. I built the altered code anyway and he tested; and if I recall, it sometimes worked for him and other times did not. English was not his primary language so there was some difficulty in communicating with each other through the translations. But it worked everytime for me. He was a user with a ton of network subnets defined on his Pass List (like maybe a hundred as I recall). He sent me his exact Pass List, and I pasted it into a list in a virtual machine, and then configured Kali Linux to spoof IP addresses on his pass list while scanning the WAN interface of my Kali machine. I would get alerts and no blocks (as desired when the IPs were on the Pass List). Removing the networks from the Pass List resulted in blocks in the virtual machine.

If the Pass List code was fundamentally flawed, I would expect to see dozens and dozens of posts here about it. After all, there are over 24,000 installations of the Snort and Suricata packages around the world. This is how many unique hits the Snort folks told me they see daily on the rules update servers where the curl user agent identifies the client as pfSense.

I am quite busy at the moment on another totally unrelated project, so it will be a while before I can look into this. But I will. In the meantime, you are welcome to dive in using the code link I provided. You should be able, if you have a FreeBSD-11.3/STABLE machine with the standard build packages in place, to create a pkg *.tgz file and then copy it over to pfSense and install it for testing. The key is having the matching version of the pfSense underlying OS. In this case that is FreeBSD-11.3/STABLE.

Cool_Corona

I encountered the same and just rebooted the box. Problem solved.

bmeeks

@cool_corona said in Suricata pass list ignored:

I encountered the same and just rebooted the box. Problem solved.

If a reboot solved the problem, that would point to you having a duplicate process on an interface. In that instance, the duplicate process will not see nor respond to any Pass List changes. That was a common problem in the past, but more rare now. However, if anyone uses Service Watchdog with the IDS/IPS packages, getting duplicate processes can happen. So can frequent up/down cycles of an interface (especially the WAN) which cause the pfSense subsystem to issue a series of "restart all packages" commands in quick succession.

j.koopmann

@bmeeks after not running into this for months it sort of slipped my mind. Now my VPN Client connection was dropped several times and during debugging I found out I was hit once more by this. The /24 of the client network is in the PASS_LIST but was ignored.

I updated to pfsense 2.5.2 and Suricata 6.0.0_10. Have you had any chance whatsoever to look into this. Reboot does NOT solve this problem.And I just rechecked: Only one process. The entry in the pass list is there for months and several reboots had been performed.

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks after not running into this for months it sort of slipped my mind. Now my VPN Client connection was dropped several times and during debugging I found out I was hit once more by this. The /24 of the client network is in the PASS_LIST but was ignored.

I updated to pfsense 2.5.2 and Suricata 6.0.0_10. Have you had any chance whatsoever to look into this. Reboot does NOT solve this problem.And I just rechecked: Only one process. The entry in the pass list is there for months and several reboots had been performed.

I have tried and tried to reproduce this issue in my test virtual machines for Suricata over the years. I have NEVER had a pass list entry not work properly in all of my testing.

Not saying you aren't seeing the problem, but if I can't reproduce it, I can't troubleshoot or fix it. I have even had other users send me their exact pass lists, which I pasted into my test VMs, and still was unable to reproduce the issue.

j.koopmann

@bmeeks i appreciate this.

How can I help? I should be able to reproduce this somehow but how would that help you?

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks i appreciate this.

How can I help? I should be able to reproduce this somehow but how would that help you?

Unfortunately it likely can't, because what would be needed is a full debug build of the Suricata package, a copy of the binary source code, and then loading it all up into the gdb debugger and stepping through the code to see where the problem is happening.

You would not want all the gdb debugging stuff and associated libraries installed on a production firewall (or I would not want that).

The Pass List feature borrows functionality from an existing shared resource in the Suricata binary -- a Radix tree. That tree is used to store, and later search for, the IP networks and/or individual host IPs pulled from the Pass List text file.

When an alert is logged, the custom blocking plugin pulls out the IP addresses and then compares them to the Pass List. It does that by searching the Radix tree to see if the IP from the alert is "covered" by an existing Radix tree entry. If not, the IP is added to the blocked table. If the Radix tree search indicates the IP is "covered" by an existing entry, the IP is not added to the blocked table. If the Radix tree code was faulty, it would be most logical for it to fail for every lookup -- not just one randomly. The only other possibility I can imagine is perhaps it is a threading problem that crops up randomly due to Suricata's multithreaded nature. That might be why I have never been able to duplicate the issue. My test VM traffic load and number of threads may not match yours.

j.koopmann

@bmeeks What about putting some debug outputs around this in one of the future versions? We could then see if the radix tree match is unsuccessful or if it is working and it is a threading problem.

bmeeks

@j-koopmann said in Suricata pass list ignored:

@bmeeks What about putting some debug outputs around this in one of the future versions? We could then see if the radix tree match is unsuccessful or if it is working and it is a threading problem.

That is an idea for the future. I am hopeful the move to iflib in FreeBSD 12, and some coming improvements to netmap support in Suricata, will lead to the custom Legacy Blocking Mode module used on pfSense being abandoned. It is not an ideal solution. It's too big of a hammer to block all traffic from a host because of a single alert. Better to use the Inline IPS Mode and just selectively drop bad packets instead of completely blocking the host IP.

Before the move to the iflib wrapper for NIC drivers in FreeBSD 12, your particular NIC hardware driver had to be patched to support netmap operation (and thus support inline IPS mode in Suricata). That limited the configurations where you could use netmap. Now, iflib wraps the netmap support up natively in FreeBSD and relieves the NIC driver from having to worry about it. There are perhaps still a few rough patches with iflib and netmap, but those should get smoothed out in future FreeBSD updates. But over time I'm hopeful that netmap and Inline IPS Mode will become how you "block" in Suricata. And the Legacy Blocking Mode and its custom module will disappear.

DaniloZ

Here is the Redmine link:
https://redmine.pfsense.org/issues/12899