Suricata is blocking LAN and WAN IPs

bmeeks · Oct 1, 2021, 1:17 PM

Sadly I have thus far been unable to replicate this problem yet again. So it must be something in my test environment that is preventing the problem from happening (blocking a Pass List IP address).

One problem I have in my current test setup is that I don't have enough, nor the right types, of hardware to configure real VLANs. I don't have a managed switch. Just have never needed one here at home. Probably need to get one eventually. I also don't have a bunch of extra base metal hardware either. I usually test with VMware Workstation and multiple VMs configured in there. But I can't get VLANs working in VMware Workstation. The instant I put a VLAN on one of the virtual interfaces of a VM, it stops passing traffic to any other VM.

I did configure an interface to use all of your IP address subnets, though, and still don't get blocks of addresses on the Pass List. I'm stumped about what the cause might be. At this point I wonder if it might be a multithread concurrent access issue. But if so, I really can't imagine what problem that would cause since the threads are only reading from the Pass List table. Once startup is complete, nothing writes to the Pass List again.

I will build a debug version of Suricata and step through the blocking module code line-by-line to see if something presents itself.

xm4rcell0x · Oct 1, 2021, 1:56 PM

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

bmeeks · Oct 1, 2021, 4:07 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

It may be that the Pass List table in memory is being updated by one thread at the same time another thread is trying to read from it. That could cause a problem where an IP that is actually on the Pass List is getting reported back as not being in the list. This would also explain why I have trouble duplicating the bug because my VM is stable after coming up, and no firewall interface IPs change nor do the interfaces themselves cycle up and down. And my traffic load is very light. So I would likely not have the conditions to trigger the bug.

I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???

xm4rcell0x · Oct 1, 2021, 5:32 PM

@bmeeks said in Suricata is blocking LAN and WAN IPs:

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.

bmeeks · Oct 1, 2021, 10:09 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks said in Suricata is blocking LAN and WAN IPs:

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.

It's a bit more complicated than those particular IPs getting blocked. I've received your PM with your email address and will respond there with some more details when I send the test package.

Bob.Dig · Oct 2, 2021, 8:54 AM

With the new version I wanted to give it another try. But hell no, Suricata is still blocking me.

I also wouldn't mind some progress here.

xm4rcell0x · Oct 2, 2021, 8:55 AM

@bob-dig yes, the _3 version doesn't have any fix for this problem. Later in the day I'll try the new binary for bmeeks and in a day I'll post back here the results

Bob.Dig · Oct 2, 2021, 9:08 AM

@bmeeks said in Suricata is blocking LAN and WAN IPs:

I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???

I looked at my log and there where some changes but concerning 192.168.1.* I can't see a problem.

2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> adding firewall interface hn1 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> Added IPv4 address 192.168.1.0/24 from assigned Pass List.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> deleted address 192.168.1.1 from automatic firewall interface IP Pass List.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> Received notification of IP address change on interface hn1.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> added address 192.168.1.1 to automatic firewall interface IP Pass List.

I have to run those cron jobs on a daily bases:

2 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.reboot 	
4 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.dyndns.update 	
8 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.reload_all

PS: Running it on WAN seems to work for now, tomorrow I will know more...

xm4rcell0x · Oct 2, 2021, 11:32 AM

@bmeeks bad news

suricata.log

bmeeks · Oct 2, 2021, 1:02 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks bad news

suricata.log

Well, crap! I guess it's back to pondering what could be going on. I really thought that change might do it. I so wish I could duplicate this on my test systems. Then I could identify the root cause and fix it instead of having to guess what may be happening.

There just must be something weird going on in Suricata's Radix Tree code, or else I am using that utility code improperly. The documentation on it is sparse.

I think I will try abandoning the Radix Tree and work on copying code over from the Snort binary package. That will take me a bit to get done and tested as not all the same supporting routines that exist in the Snort binary are present in the Suricata binary.

Bob.Dig · Oct 2, 2021, 1:08 PM

@bmeeks For me, I don't run snort because it has no Block On DROP Only mode.

xm4rcell0x · Oct 2, 2021, 1:28 PM

@bob-dig me too.
It's such a big feature in suricata !
@bmeeks can it be implemented in snort package?

bmeeks · Oct 2, 2021, 1:35 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bob-dig me too.
It's such a big feature in suricata !
@bmeeks can it be implemented in snort package?

No, because at the API hook where the custom output plugin gets called by the parent binary, Snort does not make the rule action available to test. Plus, Snort only updates its internal "drop" variables when true inline IPS mode operation is enabled in the Snort DAQ. So the short answer is "no", Snort cannot implement a "block on DROP only" option when using Legacy Mode blocking. The Snort binary is just not plumbed up the same as Suricata.

xm4rcell0x · Oct 2, 2021, 1:37 PM

@bmeeks ok, and what do you think about the "checksum offload" ? Disabling it seems has resolved the OP's problem. @Bob-Dig do you have this option enabled?

Bob.Dig · Oct 2, 2021, 1:56 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@Bob-Dig do you have this option enabled?

Yes,tried both, didn't changed anything.

xm4rcell0x · Oct 2, 2021, 5:02 PM

@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)

Now i would a fresh install but there are some files in

usr/local/etc/suricata

usr/local/etc/snort

/suricata-6.0.3_2.txz

/root/suricata-6.0.3_2.txz

(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.

As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).

bmeeks · Oct 2, 2021, 8:20 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks ok, and what do you think about the "checksum offload" ? Disabling it seems has resolved the OP's problem. @Bob-Dig do you have this option enabled?

All of the hardware options for checksum offloading, TCP Segmentation, and Large Receive should be disabled. Check the corresponding boxes under SYSTEM > ADVANCED > Networking in the pfSense menu. I belive a reboot is required for all of those changes to become effective.

bmeeks · Oct 2, 2021, 8:26 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)

Now i would a fresh install but there are some files in
usr/local/etc/suricata

usr/local/etc/snort

/suricata-6.0.3_2.txz

/root/suricata-6.0.3_2.txz 
(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.

As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).

I sent you a reply via email about removing the files. They are harmless, but removing them is easy. See the email I sent.

For your DROP versus ALERT problem, how did you initially change the rule or rules to DROP? Did you use SID MGMT, or did you click the icon on the ALERTS tab or on the RULES tab when listing the contents of that category? Whichever of those methods you used, simply reverse the process.

On the RULES tab, when listing the contents of a category, there is a reset-to-defaults button that will restore all the rules in the currently selected category to the rule vendor's defaults. That would mean all the rules in the category would be reset to ALERT. If you don't want to do that, go find the SID in question in the list and click the icon under the Action column. In the pop-up dialog, choose the option to restore the default action. Then click Save. Back on the RULES tab, click Apply to send the change to the running Suricata process.

xm4rcell0x · Oct 2, 2021, 9:38 PM

@bmeeks thank you for the email :)
I have changed from the Alert tab, i usually don't change the values from the Rules tab.
If I can help you in any way, let me know Bill!

bmeeks · Oct 3, 2021, 12:24 AM

@xm4rcell0x :

I'm sorry, but I just absolutely cannot duplicate this issue. I've cloned your firewall as closely as possible by including every single IPv4 subnet you have listed in your Pass List. I created 4 interfaces on a virtual machine. I assigned all of your IP subnets to either physical ports on the virtual machine or VLANs defined on a port.

Here is the Pass List I'm using on my test virtual machine. Notice that it explicitly lists the 10.10.20.0/24 subnet which you say is your LAN. I have that IP subnet configured on a firewall interface. The firewall is 10.10.20.1, and I have a Kali Linux virtual machine on the same virtual network at 10.10.20.198 duplicating the IP address of your blocked LAN host.

I have the ET Scan rules enabled with some of them set to DROP. Here are the alerts generated by an nmap scan of the firewall's LAN interface (the 10.10.20.1 IP address) from that Kali Linux machine at IP 10.10.20.198.

I received no block of the Kali Linux host at IP 10.10.20.198. Here is the BLOCKS tab after running the scan (and generating the alerts shown above).

As you see, the Kali Linux host is not being blocked. This is expected as its IP address is within the subnet 10.10.20.0/24 defined in the Pass List.

The direction of traffic will not matter, meaning it is not material whether the LAN host is the source of the traffic, or the destination of the traffic. The blocking plugin simply looks at the IP address in the packet. But just to be sure, I tested three times with the Which IP to Block parameter set to each possible value (SRC, DST, BOTH). They all worked properly. The LAN host was not blocked.