Suricata is blocking LAN and WAN IPs
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
I have the exact same problem. Suricata blocks my LAN IPs.
With SNORT i don't have this strange problem and i don't know why.
Already tried to reboot, to stop suricata on each interface and check for zombies processes but nothing.
I have a setup with legacy blocking and Block on Drop only (even without block on drop only i have this problem) and i have VLAN on ix0 interface (but i only have the lan in suricata) . I really don't know what to do .Here my default passlist and the suricata.log and block.log
10.10.10.1/32 10.10.20.0/24 10.10.20.254/32 10.39.156.0/25 WANIP/32 127.0.0.1/32 172.168.69.0/24 192.168.1.0/24 192.168.57.0/24 192.168.70.0/24 192.168.100.0/24 195.43.166.12/32 ::1/128 fe80::92e2:baff:fe4c:c1b4/128 fe80::e9d:92ff:fe5b:cab5/128 fe80::e9d:92ff:fe5b:cab6/128
suricata.log alerts.log blocks.log
P.s. i don't find any similar type of logs in snort, maybe it could
From the
suricata.log
file accompanying your post --30/9/2021 -- 10:36:16 - <Info> -- alert-pf -> Pass List /usr/local/etc/suricata/suricata_13894_ix0/passlist parsed: 0 IP addresses loaded.
So Suricata did not load any local IP addresses into the passlist. Notice the "0 IP addresses loaded" part. The other entries in the log where it mentions "automatic IP interface Pass List" are not the same thing. Those are actual firewall interface IPs. The regular Pass List is where LAN hosts would reside.
So in this instance, blocking of local LAN hosts would certainly occur.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
That's a newer suricata.log.lan made after a fresh install of suricata.
Update: just set up the wan on suricata with legacy block-block on drop only-block both SRC&DST and with this interface i don't have problems with passlist! It blocks only the external IPs!
suricata.log.wanNow look at the
suricata.log
file where it tells you how many IP addresses it loaded from the passed Pass List file:30/9/2021 -- 14:45:47 - <Info> -- alert-pf -> Pass List /usr/local/etc/suricata/suricata_57062_ix0/passlist parsed: 16 IP addresses loaded.
There it found and loaded 16 IP addresses and/or subnets.
-
@bmeeks yes :)
But Suricata still blocks internal IPs on this interface and i really don't know why
On WAN no problems. -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks yes :)
But Suricata still blocks internal IPs on this interface and i really don't know why
On WAN no problems.Are you running VLANs on that interface? If so, what are all of the VLAN subnets defined on that interface?
Also post the content of this file:
/usr/local/etc/suricata/suricata_57062_ix0/passlist
back here. -
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks passlist
Yes i have VLANs on this interface:
192.168.100.0/24 = ix0.100
172.168.69.0/24 = ix0.69
192.168.57.0/24 = ix0.57
LAN = 10.10.20.0/24One other request I should have included previously, but forgot --
Go to the BLOCKS tab and clear all blocks. Remove them all. Then when you experience a new LAN host getting blocked, do this:
Go to DIAGNOSTICS > TABLES and choose the snort2c table from the drop-down list. Screen capture the content of that display of IP addresses in that table and post back here.
Because you have posted just the logs, and the logs don't appear to be time correlated, it's difficult to make heads or tails of what's happening. The content of the snort2c table will always be the current list of what Suricata is blocking. The blocks log will have historical data, and because of the timestamp issues I don't know which blocks were from the run where Suricata loaded zero IP addresses from the Pass List, and which blocks are from the run where it loaded 16 IP addresses from the list.
I would dearly love to find out what is going on as I've had a few users report this behavior over the years. Not a lot, but certainly some. However, I have never in all my testing since I created the Suricata package, been able to duplicate this problem in my test systems. Not once. So if I can't duplicate it, that makes it very hard to troubleshoot, and even harder to actually fix.
-
@bmeeks
i will do all you want! i really want to help you to fix this :) so no problem at all!
suricata alerts
snort2cp.s. i have all the networking offloads disabled but checksum and hardware vlan tagging on this ix0 interface.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks
i will do all you want! i really want to help you to fix this :) so no problem at all!
suricata alerts
snort2cAnd just to be clear, is this the same running instance where the
suricata.log
said it had loaded 16 IP addresses from the passlist file?It's important that Suricata logs that it actually read IP addresses from the passlist file when starting up. You had two posted
suricata.log
files where one showed zero IP addresses loaded, and the other showed 16 IP addresses loaded. Does thesuricata.log
file for this interface currently contain a line saying IP addresses were loaded from the file/usr/local/etc/suricata/suricata_57062_ix0/passlist
? -
@bmeeks
yes i can confirm! the old istance is gone
suricata.log -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks
yes i can confirm! the old istance is gone
suricata.logThanks for the confirmation. I've got an idea, but it will take me a day or so to test it. Still not sure I can reliably reproduce the problem, but I think I see a place in the custom blocking plugin binary code where a section of structure union data could get misinterpreted. I need to figure out a sequence of events and data that reliably trigger the problem to be sure that's really the problem.
-
@bmeeks i'll try to reproduce also on my friend's network in 1 or 2 hours .
I'll put here the results.
Thank you so much :) -
done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wanTomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wanTomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.
Thank you for the additional info. I will duplicate this IP setup in my VMware virtual machine and see if I can replicate the problem. I've made a change in my test system that "might" have an impact, but it would be much more reassuring to have a test case that reliably fails, and then works after my patch is applied.
There was a new Suricata package update posted this afternoon. It addresses another issue with VLANs and I don't expect it to fix your problem, but it would not hurt to test it out just to verify.
-
Sadly I have thus far been unable to replicate this problem yet again. So it must be something in my test environment that is preventing the problem from happening (blocking a Pass List IP address).
One problem I have in my current test setup is that I don't have enough, nor the right types, of hardware to configure real VLANs. I don't have a managed switch. Just have never needed one here at home. Probably need to get one eventually. I also don't have a bunch of extra base metal hardware either. I usually test with VMware Workstation and multiple VMs configured in there. But I can't get VLANs working in VMware Workstation. The instant I put a VLAN on one of the virtual interfaces of a VM, it stops passing traffic to any other VM.
I did configure an interface to use all of your IP address subnets, though, and still don't get blocks of addresses on the Pass List. I'm stumped about what the cause might be. At this point I wonder if it might be a multithread concurrent access issue. But if so, I really can't imagine what problem that would cause since the threads are only reading from the Pass List table. Once startup is complete, nothing writes to the Pass List again.
I will build a debug version of Suricata and step through the blocking module code line-by-line to see if something presents itself.
-
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files. -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.log
file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.It may be that the Pass List table in memory is being updated by one thread at the same time another thread is trying to read from it. That could cause a problem where an IP that is actually on the Pass List is getting reported back as not being in the list. This would also explain why I have trouble duplicating the bug because my VM is stable after coming up, and no firewall interface IPs change nor do the interfaces themselves cycle up and down. And my traffic load is very light. So I would likely not have the conditions to trigger the bug.
I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???
-
@bmeeks said in Suricata is blocking LAN and WAN IPs:
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.log
file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home. -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks said in Suricata is blocking LAN and WAN IPs:
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.log
file for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.It's a bit more complicated than those particular IPs getting blocked. I've received your PM with your email address and will respond there with some more details when I send the test package.
-
With the new version I wanted to give it another try. But hell no, Suricata is still blocking me.
I also wouldn't mind some progress here.
-
@bob-dig yes, the _3 version doesn't have any fix for this problem. Later in the day I'll try the new binary for bmeeks and in a day I'll post back here the results
-
@bmeeks said in Suricata is blocking LAN and WAN IPs:
I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???
I looked at my log and there where some changes but concerning 192.168.1.* I can't see a problem.
2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> adding firewall interface hn1 IPv4 address 192.168.1.1 to automatic interface IP Pass List. 2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> Added IPv4 address 192.168.1.0/24 from assigned Pass List. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> deleted address 192.168.1.1 from automatic firewall interface IP Pass List. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> Received notification of IP address change on interface hn1. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> added address 192.168.1.1 to automatic firewall interface IP Pass List.
I have to run those cron jobs on a daily bases:
2 7 * * * root /usr/bin/nice -n20 /etc/rc.reboot 4 7 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update 8 7 * * * root /usr/bin/nice -n20 /etc/rc.reload_all
PS: Running it on WAN seems to work for now, tomorrow I will know more...
-
@bmeeks bad news
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks bad news
Well, crap! I guess it's back to pondering what could be going on. I really thought that change might do it. I so wish I could duplicate this on my test systems. Then I could identify the root cause and fix it instead of having to guess what may be happening.
There just must be something weird going on in Suricata's Radix Tree code, or else I am using that utility code improperly. The documentation on it is sparse.
I think I will try abandoning the Radix Tree and work on copying code over from the Snort binary package. That will take me a bit to get done and tested as not all the same supporting routines that exist in the Snort binary are present in the Suricata binary.
-
@bmeeks For me, I don't run snort because it has no Block On DROP Only mode.
-
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bob-dig me too.
It's such a big feature in suricata !
@bmeeks can it be implemented in snort package?No, because at the API hook where the custom output plugin gets called by the parent binary, Snort does not make the rule action available to test. Plus, Snort only updates its internal "drop" variables when true inline IPS mode operation is enabled in the Snort DAQ. So the short answer is "no", Snort cannot implement a "block on DROP only" option when using Legacy Mode blocking. The Snort binary is just not plumbed up the same as Suricata.
-
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@Bob-Dig do you have this option enabled?
Yes,tried both, didn't changed anything.
-
@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)Now i would a fresh install but there are some files in
usr/local/etc/suricata usr/local/etc/snort /suricata-6.0.3_2.txz /root/suricata-6.0.3_2.txz
(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks ok, and what do you think about the "checksum offload" ? Disabling it seems has resolved the OP's problem. @Bob-Dig do you have this option enabled?
All of the hardware options for checksum offloading, TCP Segmentation, and Large Receive should be disabled. Check the corresponding boxes under SYSTEM > ADVANCED > Networking in the pfSense menu. I belive a reboot is required for all of those changes to become effective.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks
I don't know why but I've tried to unlock my internal IP, but before that i have changed the Drop action from Drop to Alert.
Cleared the Block Table, restarted suricata on the LAN interface , but it continues to trigger the Drop action instead if the Alert one, so i have uninstalled Suricata (i've followed your instructions)Now i would a fresh install but there are some files in
usr/local/etc/suricata usr/local/etc/snort /suricata-6.0.3_2.txz /root/suricata-6.0.3_2.txz
(also see the email)
How can i remove these files and folders ? I'm not a linux guy but i know there is the "remove" command, but i don't want to mess anything, that's my production environment.As you can see, on the right corner there the Alert action selected, but on the left for some rules it is still on the Drop action (already tried to change from that corner).
I sent you a reply via email about removing the files. They are harmless, but removing them is easy. See the email I sent.
For your DROP versus ALERT problem, how did you initially change the rule or rules to DROP? Did you use SID MGMT, or did you click the icon on the ALERTS tab or on the RULES tab when listing the contents of that category? Whichever of those methods you used, simply reverse the process.
On the RULES tab, when listing the contents of a category, there is a reset-to-defaults button that will restore all the rules in the currently selected category to the rule vendor's defaults. That would mean all the rules in the category would be reset to ALERT. If you don't want to do that, go find the SID in question in the list and click the icon under the Action column. In the pop-up dialog, choose the option to restore the default action. Then click Save. Back on the RULES tab, click Apply to send the change to the running Suricata process.
-
@bmeeks thank you for the email :)
I have changed from the Alert tab, i usually don't change the values from the Rules tab.
If I can help you in any way, let me know Bill! -
I'm sorry, but I just absolutely cannot duplicate this issue. I've cloned your firewall as closely as possible by including every single IPv4 subnet you have listed in your Pass List. I created 4 interfaces on a virtual machine. I assigned all of your IP subnets to either physical ports on the virtual machine or VLANs defined on a port.
Here is the Pass List I'm using on my test virtual machine. Notice that it explicitly lists the 10.10.20.0/24 subnet which you say is your LAN. I have that IP subnet configured on a firewall interface. The firewall is 10.10.20.1, and I have a Kali Linux virtual machine on the same virtual network at 10.10.20.198 duplicating the IP address of your blocked LAN host.
I have the ET Scan rules enabled with some of them set to DROP. Here are the alerts generated by an nmap scan of the firewall's LAN interface (the 10.10.20.1 IP address) from that Kali Linux machine at IP 10.10.20.198.
I received no block of the Kali Linux host at IP 10.10.20.198. Here is the BLOCKS tab after running the scan (and generating the alerts shown above).
As you see, the Kali Linux host is not being blocked. This is expected as its IP address is within the subnet 10.10.20.0/24 defined in the Pass List.
The direction of traffic will not matter, meaning it is not material whether the LAN host is the source of the traffic, or the destination of the traffic. The blocking plugin simply looks at the IP address in the packet. But just to be sure, I tested three times with the Which IP to Block parameter set to each possible value (SRC, DST, BOTH). They all worked properly. The LAN host was not blocked.
-
Another test for you to try for me. What kind of machine is that 10.10.20.198 host? Is it just a workstation -- perhaps Windows?
If so, and you can access a COMMAND prompt on that PC, I want to run this test.
- Go to the RULES tab for the LAN interface in the Suricata GUI on the firewall.
- In the Category drop-down, choose "custom.rules".
- In the blank textbox that opens, copy and paste in this simple rule:
drop icmp any any -> $HOME_NET any (msg:"Test drop of icmp-request"; flow:to_server; classtype:bad-unknown; sid:20010936; rev:1;)
- Click Save when you are done.
- Back on the INTERFACES tab in Suricata, restart the LAN instance (I am assuming that is the 10.10.20.0/24 interface).
- Clear the BLOCKS tab in Suricata.
- Now go to that 10.10.20.198 host, and in a COMMAND prompt window type:
ping 10.10.20.1
or whatever the firewall's LAN IP address is. The idea is to ping the firewall's LAN interface from a host on the LAN to trigger the custom DROP rule.
Check if the host is immediately blocked, or if the first ping succeeds. I would like to establish if it takes some time for the IP to get blocked when triggering DROP rules, or if it gets blocked immediately.
If you can't use the 10.10.20.198 host, then any other host on that network will work as well.
-
I did have one other idea pop into my head after my previous posts above. There is one more place in the custom blocking plugin code I just found where I think some random data could cause the errant blocks.
I am creating an additional "fix" I would like for you to test same as you did the first time. Check your email for another test binary package arriving soon. I will include the installation instructions again in the email.
Thanks
-
@bmeeks Interestingly for me the problem is not occurring if I run those rules on the WAN-Interface with Suricata. Also please note that in my case the "offender" is a local Windows, as those rules are privacy-focused rules and not against attacks from outside.
So maybe this is part of the problem in my case.1:2025275 emerging-info.rules drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET INFO Windows OS Submitting USB Metadata to Microsoft"
1:2027390 emerging-user_agents.rules drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent";
You can trigger those if you (re-)connect a bluetooth device to Windows.
-
@bob-dig me too, on the WAN side no problem at all.
I'm testing the newer binary from @bmeeks and i have some news.
Suricata only blocks the parent interface's IPs, so in my case the subnet 10.10.20.0/24, but not the VLAN on that interface as you can see in the screenshot.
I have also tried to Drop a Rule where the Src is 192.168.1.10 (igb0 interface, that's my ISP modem) and the destination is 10.10.20.201 (my desktop), 192.168.1.10 wasn't blocked.
That's another rule, where the VLAN 69 (an IOT device, ip: 172.168.69.33) wasn't blocked by suricata.
-
@xm4rcell0x Thanks. Kinda the same here, I only noticed it on parent interface or interface without vlans. But also could be a coincidence.
-
@bob-dig said in Suricata is blocking LAN and WAN IPs:
@bmeeks Interestingly for me the problem is not occurring if I run those rules on the WAN-Interface with Suricata. Also please note that in my case the "offender" is a local Windows, as those rules are privacy-focused rules and not against attacks from outside.
So maybe this is part of the problem in my case.1:2025275 emerging-info.rules drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET INFO Windows OS Submitting USB Metadata to Microsoft"
1:2027390 emerging-user_agents.rules drop http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET USER_AGENTS Microsoft Device Metadata Retrieval Client User-Agent";
You can trigger those if you (re-)connect a bluetooth device to Windows.
The type of rule and the direction of traffic flow have no significance. What is happening is that when the IP addresses from the rule are compared to the IPs and subnets stored in the Pass List Radix Tree, the comparison seems to fail for you. I don't know why, because it does not fail for me. When the comparison fails, the blocking code thinks the IP address is not in the Pass List and thus blocks it. That's the problem. At the point this decision is made in the code, it is not looking at anything other than the IP address. Nothing else.
-
So you are getting blocks of hosts residing on the parent interface (the LAN, for example), but not on VLANs defined on that parent? Is that correct?
That can be a hint. It seems I need to work on setting up an environment where I can use actual VLANs for more testing.