Suricata is blocking LAN and WAN IPs

xm4rcell0x · Sep 30, 2021, 9:12 PM

done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100

[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist

10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wan

Tomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.

bmeeks · Sep 30, 2021, 10:30 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100

[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist

10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wan

Tomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.

Thank you for the additional info. I will duplicate this IP setup in my VMware virtual machine and see if I can replicate the problem. I've made a change in my test system that "might" have an impact, but it would be much more reassuring to have a test case that reliably fails, and then works after my patch is applied.

There was a new Suricata package update posted this afternoon. It addresses another issue with VLANs and I don't expect it to fix your problem, but it would not hurt to test it out just to verify.

bmeeks · Oct 1, 2021, 1:17 PM

Sadly I have thus far been unable to replicate this problem yet again. So it must be something in my test environment that is preventing the problem from happening (blocking a Pass List IP address).

One problem I have in my current test setup is that I don't have enough, nor the right types, of hardware to configure real VLANs. I don't have a managed switch. Just have never needed one here at home. Probably need to get one eventually. I also don't have a bunch of extra base metal hardware either. I usually test with VMware Workstation and multiple VMs configured in there. But I can't get VLANs working in VMware Workstation. The instant I put a VLAN on one of the virtual interfaces of a VM, it stops passing traffic to any other VM.

I did configure an interface to use all of your IP address subnets, though, and still don't get blocks of addresses on the Pass List. I'm stumped about what the cause might be. At this point I wonder if it might be a multithread concurrent access issue. But if so, I really can't imagine what problem that would cause since the threads are only reading from the Pass List table. Once startup is complete, nothing writes to the Pass List again.

I will build a debug version of Suricata and step through the blocking module code line-by-line to see if something presents itself.

xm4rcell0x · Oct 1, 2021, 1:56 PM

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

bmeeks · Oct 1, 2021, 4:07 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

It may be that the Pass List table in memory is being updated by one thread at the same time another thread is trying to read from it. That could cause a problem where an IP that is actually on the Pass List is getting reported back as not being in the list. This would also explain why I have trouble duplicating the bug because my VM is stable after coming up, and no firewall interface IPs change nor do the interfaces themselves cycle up and down. And my traffic load is very light. So I would likely not have the conditions to trigger the bug.

I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???

xm4rcell0x · Oct 1, 2021, 5:32 PM

@bmeeks said in Suricata is blocking LAN and WAN IPs:

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.

bmeeks · Oct 1, 2021, 10:09 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks said in Suricata is blocking LAN and WAN IPs:

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.

I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.

Will you check your suricata.log file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.

I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.

It's a bit more complicated than those particular IPs getting blocked. I've received your PM with your email address and will respond there with some more details when I send the test package.

Bob.Dig · Oct 2, 2021, 8:54 AM

With the new version I wanted to give it another try. But hell no, Suricata is still blocking me.

I also wouldn't mind some progress here.

xm4rcell0x · Oct 2, 2021, 8:55 AM

@bob-dig yes, the _3 version doesn't have any fix for this problem. Later in the day I'll try the new binary for bmeeks and in a day I'll post back here the results

Bob.Dig · Oct 2, 2021, 9:08 AM

@bmeeks said in Suricata is blocking LAN and WAN IPs:

I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???

I looked at my log and there where some changes but concerning 192.168.1.* I can't see a problem.

2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> adding firewall interface hn1 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> Added IPv4 address 192.168.1.0/24 from assigned Pass List.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> deleted address 192.168.1.1 from automatic firewall interface IP Pass List.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> Received notification of IP address change on interface hn1.
2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> added address 192.168.1.1 to automatic firewall interface IP Pass List.

I have to run those cron jobs on a daily bases:

2 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.reboot 	
4 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.dyndns.update 	
8 	7 	* 	* 	* 	root 	/usr/bin/nice -n20 /etc/rc.reload_all

PS: Running it on WAN seems to work for now, tomorrow I will know more...

xm4rcell0x · Oct 2, 2021, 11:32 AM

@bmeeks bad news

suricata.log

bmeeks · Oct 2, 2021, 1:02 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks bad news

suricata.log

Well, crap! I guess it's back to pondering what could be going on. I really thought that change might do it. I so wish I could duplicate this on my test systems. Then I could identify the root cause and fix it instead of having to guess what may be happening.

There just must be something weird going on in Suricata's Radix Tree code, or else I am using that utility code improperly. The documentation on it is sparse.

I think I will try abandoning the Radix Tree and work on copying code over from the Snort binary package. That will take me a bit to get done and tested as not all the same supporting routines that exist in the Snort binary are present in the Suricata binary.

Bob.Dig · Oct 2, 2021, 1:08 PM

@bmeeks For me, I don't run snort because it has no Block On DROP Only mode.

xm4rcell0x · Oct 2, 2021, 1:28 PM

@bob-dig me too.
It's such a big feature in suricata !
@bmeeks can it be implemented in snort package?

bmeeks · Oct 2, 2021, 1:35 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bob-dig me too.
It's such a big feature in suricata !
@bmeeks can it be implemented in snort package?

No, because at the API hook where the custom output plugin gets called by the parent binary, Snort does not make the rule action available to test. Plus, Snort only updates its internal "drop" variables when true inline IPS mode operation is enabled in the Snort DAQ. So the short answer is "no", Snort cannot implement a "block on DROP only" option when using Legacy Mode blocking. The Snort binary is just not plumbed up the same as Suricata.

xm4rcell0x · Oct 2, 2021, 1:37 PM

@bmeeks ok, and what do you think about the "checksum offload" ? Disabling it seems has resolved the OP's problem. @Bob-Dig do you have this option enabled?

Bob.Dig · Oct 2, 2021, 1:56 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@Bob-Dig do you have this option enabled?

Yes,tried both, didn't changed anything.

xm4rcell0x · Oct 2, 2021, 5:02 PM

@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)

Now i would a fresh install but there are some files in

usr/local/etc/suricata

usr/local/etc/snort

/suricata-6.0.3_2.txz

/root/suricata-6.0.3_2.txz

(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.

As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).

bmeeks · Oct 2, 2021, 8:20 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks ok, and what do you think about the "checksum offload" ? Disabling it seems has resolved the OP's problem. @Bob-Dig do you have this option enabled?

All of the hardware options for checksum offloading, TCP Segmentation, and Large Receive should be disabled. Check the corresponding boxes under SYSTEM > ADVANCED > Networking in the pfSense menu. I belive a reboot is required for all of those changes to become effective.

bmeeks · Oct 2, 2021, 8:26 PM

@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:

@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)

Now i would a fresh install but there are some files in
usr/local/etc/suricata

usr/local/etc/snort

/suricata-6.0.3_2.txz

/root/suricata-6.0.3_2.txz 
(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.

As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).

I sent you a reply via email about removing the files. They are harmless, but removing them is easy. See the email I sent.

For your DROP versus ALERT problem, how did you initially change the rule or rules to DROP? Did you use SID MGMT, or did you click the icon on the ALERTS tab or on the RULES tab when listing the contents of that category? Whichever of those methods you used, simply reverse the process.

On the RULES tab, when listing the contents of a category, there is a reset-to-defaults button that will restore all the rules in the currently selected category to the rule vendor's defaults. That would mean all the rules in the category would be reset to ALERT. If you don't want to do that, go find the SID in question in the list and click the icon under the Action column. In the pop-up dialog, choose the option to restore the default action. Then click Save. Back on the RULES tab, click Apply to send the change to the running Suricata process.