Suricata is blocking LAN and WAN IPs
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks yes :)
But Suricata still blocks internal IPs on this interface and i really don't know why
On WAN no problems.Are you running VLANs on that interface? If so, what are all of the VLAN subnets defined on that interface?
Also post the content of this file:
/usr/local/etc/suricata/suricata_57062_ix0/passlistback here. -
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks passlist
Yes i have VLANs on this interface:
192.168.100.0/24 = ix0.100
172.168.69.0/24 = ix0.69
192.168.57.0/24 = ix0.57
LAN = 10.10.20.0/24One other request I should have included previously, but forgot --
Go to the BLOCKS tab and clear all blocks. Remove them all. Then when you experience a new LAN host getting blocked, do this:
Go to DIAGNOSTICS > TABLES and choose the snort2c table from the drop-down list. Screen capture the content of that display of IP addresses in that table and post back here.
Because you have posted just the logs, and the logs don't appear to be time correlated, it's difficult to make heads or tails of what's happening. The content of the snort2c table will always be the current list of what Suricata is blocking. The blocks log will have historical data, and because of the timestamp issues I don't know which blocks were from the run where Suricata loaded zero IP addresses from the Pass List, and which blocks are from the run where it loaded 16 IP addresses from the list.
I would dearly love to find out what is going on as I've had a few users report this behavior over the years. Not a lot, but certainly some. However, I have never in all my testing since I created the Suricata package, been able to duplicate this problem in my test systems. Not once. So if I can't duplicate it, that makes it very hard to troubleshoot, and even harder to actually fix.
-
@bmeeks
i will do all you want! i really want to help you to fix this :) so no problem at all!
suricata alerts
snort2cp.s. i have all the networking offloads disabled but checksum and hardware vlan tagging on this ix0 interface.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks
i will do all you want! i really want to help you to fix this :) so no problem at all!
suricata alerts
snort2cAnd just to be clear, is this the same running instance where the
suricata.logsaid it had loaded 16 IP addresses from the passlist file?It's important that Suricata logs that it actually read IP addresses from the passlist file when starting up. You had two posted
suricata.logfiles where one showed zero IP addresses loaded, and the other showed 16 IP addresses loaded. Does thesuricata.logfile for this interface currently contain a line saying IP addresses were loaded from the file/usr/local/etc/suricata/suricata_57062_ix0/passlist? -
@bmeeks
yes i can confirm! the old istance is gone
suricata.log -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks
yes i can confirm! the old istance is gone
suricata.logThanks for the confirmation. I've got an idea, but it will take me a day or so to test it. Still not sure I can reliably reproduce the problem, but I think I see a place in the custom blocking plugin binary code where a section of structure union data could get misinterpreted. I need to figure out a sequence of events and data that reliably trigger the problem to be sure that's really the problem.
-
@bmeeks i'll try to reproduce also on my friend's network in 1 or 2 hours .
I'll put here the results.
Thank you so much :) -
done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wanTomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
done! also on my friend's network suricata blocks the internal IPs on the LAN interface!
His network topology
LAN = 10.10.30.0/24 = em0 interface (WAN on igb0)
VLANs are:
192.168.69.0/24 = em0.69
10.10.100.0/24 = em0.100[In reply to Sam Sepiol]
/usr/local/etc/suricata/suricata_37556_em0/passlist10.10.10.1/32
10.10.30.0/24
10.10.30.254/32
10.10.100.0/24
WANIP
127.0.0.1/32
192.168.1.0/24
192.168.69.0/24
195.43.166.12/32
::1/128
fe80::b62e:99ff:fe62:28ea/128
fe80::b62e:99ff:fe62:28eb/128
suricata.log.lan
suricata.log.wanTomorrow i'll also share the snort2c log when suricata will be triggered and i'll also post a screenshot from the Alerts page with the exact time.
Thank you for the additional info. I will duplicate this IP setup in my VMware virtual machine and see if I can replicate the problem. I've made a change in my test system that "might" have an impact, but it would be much more reassuring to have a test case that reliably fails, and then works after my patch is applied.
There was a new Suricata package update posted this afternoon. It addresses another issue with VLANs and I don't expect it to fix your problem, but it would not hurt to test it out just to verify.
-
Sadly I have thus far been unable to replicate this problem yet again. So it must be something in my test environment that is preventing the problem from happening (blocking a Pass List IP address).
One problem I have in my current test setup is that I don't have enough, nor the right types, of hardware to configure real VLANs. I don't have a managed switch. Just have never needed one here at home. Probably need to get one eventually. I also don't have a bunch of extra base metal hardware either. I usually test with VMware Workstation and multiple VMs configured in there. But I can't get VLANs working in VMware Workstation. The instant I put a VLAN on one of the virtual interfaces of a VM, it stops passing traffic to any other VM.
I did configure an interface to use all of your IP address subnets, though, and still don't get blocks of addresses on the Pass List. I'm stumped about what the cause might be. At this point I wonder if it might be a multithread concurrent access issue. But if so, I really can't imagine what problem that would cause since the threads are only reading from the Pass List table. Once startup is complete, nothing writes to the Pass List again.
I will build a debug version of Suricata and step through the blocking module code line-by-line to see if something presents itself.
-
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files. -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.logfile for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.It may be that the Pass List table in memory is being updated by one thread at the same time another thread is trying to read from it. That could cause a problem where an IP that is actually on the Pass List is getting reported back as not being in the list. This would also explain why I have trouble duplicating the bug because my VM is stable after coming up, and no firewall interface IPs change nor do the interfaces themselves cycle up and down. And my traffic load is very light. So I would likely not have the conditions to trigger the bug.
I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???
-
@bmeeks said in Suricata is blocking LAN and WAN IPs:
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.logfile for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home. -
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks said in Suricata is blocking LAN and WAN IPs:
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks maybe we can try with another volunteer here on the forum that have a baremetal pfSense box. I really want to help you!
If you need anything else let me know
When I'll be home tonight I'll post the other files.I have an idea of what might be happening. If I am correct, it is a multiple thread concurrent access problem when checking and/or updating the Pass List.
Will you check your
suricata.logfile for the interfaces where LAN hosts are getting blocked and see if there are instances of IP addresses being added and removed for firewall interfaces? These will be tagged with some text similar to either "...added address xxxx to automatic firewall interface IP Pass List..." or "...deleted address xxxx from automatic firewall interface IP Pass List...". I'm curious if the timestamps logged for any of these messages correspond with the timestamps of any LAN host blocks.I've quickly check my first suricata.log and i see 10.10.10.1 and 10.10.20.254, both are VIP, one from dnsbl and the other is the haproxy VIP. But these IPs weren't blocked by suricata.
I'll check from my desktop when I'll be home.It's a bit more complicated than those particular IPs getting blocked. I've received your PM with your email address and will respond there with some more details when I send the test package.
-
With the new version I wanted to give it another try. But hell no, Suricata is still blocking me.
I also wouldn't mind some progress here.
-
@bob-dig yes, the _3 version doesn't have any fix for this problem. Later in the day I'll try the new binary for bmeeks and in a day I'll post back here the results
-
@bmeeks said in Suricata is blocking LAN and WAN IPs:
I noticed in your previously posted logs that some firewall interface IPs were logged as changing. It looked like maybe an interface cycled, or perhaps it's a VPN tunnel coming up and down ???
I looked at my log and there where some changes but concerning 192.168.1.* I can't see a problem.
2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> adding firewall interface hn1 IPv4 address 192.168.1.1 to automatic interface IP Pass List. 2/10/2021 -- 07:02:56 - <Info> -- alert-pf -> Added IPv4 address 192.168.1.0/24 from assigned Pass List. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> deleted address 192.168.1.1 from automatic firewall interface IP Pass List. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> Received notification of IP address change on interface hn1. 2/10/2021 -- 07:10:04 - <Info> -- alert-pf -> added address 192.168.1.1 to automatic firewall interface IP Pass List.I have to run those cron jobs on a daily bases:
2 7 * * * root /usr/bin/nice -n20 /etc/rc.reboot 4 7 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update 8 7 * * * root /usr/bin/nice -n20 /etc/rc.reload_allPS: Running it on WAN seems to work for now, tomorrow I will know more...
-
-
@xm4rcell0x said in Suricata is blocking LAN and WAN IPs:
@bmeeks bad news
Well, crap! I guess it's back to pondering what could be going on. I really thought that change might do it. I so wish I could duplicate this on my test systems. Then I could identify the root cause and fix it instead of having to guess what may be happening.
There just must be something weird going on in Suricata's Radix Tree code, or else I am using that utility code improperly. The documentation on it is sparse.
I think I will try abandoning the Radix Tree and work on copying code over from the Snort binary package. That will take me a bit to get done and tested as not all the same supporting routines that exist in the Snort binary are present in the Suricata binary.
