Suricata blocking IPs on passlist, legacy mode blocking both

sgnoc

@SteveITS No issues running on WAN. I like running the bad reputation lists on my WAN. I'm having issues with internal interfaces blocking internal IPs within ranges or specifically listed on the default IP Pass List.

btspce

@SteveITS Yes WAN plus two other internal interfaces.
But due to suricata blocking the wan vip within seconds on start we have disabled suricata on wan.
But today one of our internal dns servers was blocked by suricata running since yesterday on an internal interface.
This is how we found out that the passlist was not working and sure enough the carp wan vip was blocked by suricata in the logs on wan.
This dns server is the primary pfsense dns server and is on the passlist.
So the conclusion is that there is still issues with the passlist not working randomly and there is a high risk that any random internal ip could be blocked at any moment with this version 7.0.2_3.

We are stopping suricata on all interfaces until this is resolved.

SteveITS

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

I like running the bad reputation lists on my WAN.

OK. We put those in a pfBlocker feed or alias and just create a rule.

I’ll watch on ours, AFAIK no internal IPs blocked yet. But thanks for the heads up. Threads like these recent Suricata threads are helpful to know what’s going on.

AlternateShadow

Suricata is running only on internal interfaces here and DST addresses on the pass list are getting blocked.

SteveITS

Is everyone using the Default pass list or a custom list? We're using a custom list with all the "Auto-Generated IP Addresses" boxes checked, and our "trusted" alias added.

btspce

@SteveITS Threads like these is why we waited with the suricata upgrade but unfortunately there were still bugs left. Still grateful for @bmeeks great work.
Keep an eye on your lan ip so it doesn't get randomly blocked (!) . IP's from both the default passlist and the IP Pass List can be blocked as it is now.

We are using a custom passlist with all the "Auto-Generated IP Addresses" boxes checked and trusted alias added aswell.

SteveITS

@btspce What rule is blocking your internal IPs? I'm wondering if it's not something we have enabled.

Our LAN IP has actually shown up but not been blocked... I suppressed a "SURICATA SSH invalid banner" alert yesterday from an internal network scanner/probe IP and it didn't block either.

I'd upgraded Suricata and set it back to Auto the day before.

btspce

@SteveITS Our WAN VIP and our DNS internal IP were both found in suricatas block list and was very much blocked until removed.
Suricata works very well in that regard :)

btspce

@SteveITS

WAN VIP
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}

DNS
[Block Dst] [] [1:2035465:4] ET INFO Observed Discord Domain in DNS Lookup (discord .com) [] [Classification: Misc activity] [Priority: 3] {UDP}

SteveITS

@btspce FWIW we don't have either of those enabled...DShield is covered by the ET_Block feed in pfBlocker (so plain fw rule) and "info" is usually meant as informational/observation per Bill and we'd seen a lot of false positives so we don't have those enabled. So, small possibility it's rule related but I would think not.

"when I enable pass list debugging, everything starts working as normal"

Knowing absolutely nothing about the code, maybe thread/timing related?

btspce

@SteveITS Well @bmeeks already found and fixed two bugs related to the passlist randomly not working higher up in this thread which was included in the latest suricata version as I understands it so another one seems likely at this point. I'm waiting for Bill to chime in but it's weird you don't see any issues yet.

Anyway suricata should not be blocking whitelisted ip's.

sgnoc

@SteveITS I'm using the default pass list on all of my interfaces.

bmeeks

@btspce and @sgnoc:
I need some additional information from both of you to help narrow this down.

Post the full output of the suricata.log file for the impacted interface (or interfaces if several). You can easily view that file and copy its contents to the clipboard for pasting here on the forum under the LOGS VIEW tab in Suricata. To make reading the file easier, once you paste its contents into your post, highlight all the text you just pasted with your mouse and then click the "Code" icon at the top of the post submission dialog. That icon looks like this: </>.
Use the DIAGNOSTICS > EDIT FILE menu choice in pfSense and browse to the configuration directory for an impacted Suricata interface and paste the full contents of the pass_list file back here. You will find the file under /usr/local/etc/suricata/suricata_xxx_yyyyy on the firewall. Again, use the DIAGNOSTICS > EDIT FILE menu choice to browse to the file and open it. Paste the contents back here. To format the pasted text so it's easier to read, do the same thing as step #1 above: highlight all of the pasted in text and click the Code icon (</>) to format it.
Are you using VLANs on the impacted interfaces? If so, how many?

Turn on the pass list debugging option as described in this post of mine higher up in this thread: https://forum.netgate.com/topic/184858/suricata-blocking-ips-on-passlist-legacy-mode-blocking-both/8.

I examined the Pass List logic pretty much all day yesterday, but I am not finding anything obviously wrong. Whatever is happening is subtle because not all users are impacted.

btspce

@bmeeks Hello Is there any other way I can send these files to you so we don't have to show our internal/external ip adresses for the whole world ?

bmeeks

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Hello Is there any other way I can send these files to you so we don't have to show our internal/external ip adresses for the whole world ?

Yes, you can send them to my Gmail account. Here is first part of the address. The second part is of course gmail.com.

billmeeks8

btspce

@bmeeks Email sent

bmeeks

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Email sent

Confirmed receipt with a reply. Thank you for sending the data.

sgnoc

@bmeeks I'm trying to get this information for you. The trouble I seem to be having is it only happens when pass list debugging is off. When I turned on pass list debugging on the interface, the problem goes away, at least with one interface. I'm waiting to see if another interface with debugging on will alert, but it doesn't alert that often.

I'll continue to try and get you the above information as soon as possible.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks I'm trying to get this information for you. The trouble I seem to be having is it only happens when pass list debugging is off. When I turned on pass list debugging on the interface, the problem goes away, at least with one interface. I'm waiting to see if another interface with debugging on will alert, but it doesn't alert that often.

I'll continue to try and get you the above information as soon as possible.

I will take it either way (with and/or without the pass list debugging). I'm really struggling to understand what relationship the pass list debugging option has, though. I have gone through the code multiple times trying to see if anything different happens relative to blocking with that enabled versus disabled, and I am not finding it.

sgnoc

@bmeeks Well, I finally got back to my network. I attempted to start from a fresh suricata install and have had nothing but trouble since. I completely uninstalled Suricata, and then did a fresh install. Now I'm right back to my WAN interface blocking my WAN IP again, like it did in a previous post of mine on this topic.

I've tried uninstalling and reinstalling, restarting the Suricata service, and also restarting the pfSense router. Nothing so far has resolved it. I've had to disable blocking on the WAN interface so I can keep my network going. I have never had this or other interface internal IP blocking issues previous to this major version of Suricata, so I'm stumped. I've collected as much as possible from the logs, but without having the pass list debugging enabled. In this case it is easy, the WAN IP was not put in the default pass list. The interface was up and operational when suricata was installed, and the WAN Gateway is in the list, just not the WAN IP. I have tried disabled blocking, restarting the interface, then enabled and restarting again, but the default pass IP list is not updating with the WAN IP.

I am using VLANs on the internal interfaces, but not the WAN interface. On the internal switch of the XG-7100, I'm using 8 VLANs (not using default VLAN 1), ix0 is the WAN (no VLAN) and ix1 is going to the downstream switches using 6 VLANS (not using default VLAN 1).

As a note of what I saw in the log, it appears for whatever reason the WAN IP was added and removed from the IP Pass List multiple times, with an ending result of being deleted, causing the IP to be blocked on the next alert.

I still have blocking enabled on on other interfaces to do testing, but I have to keep my WAN up, so I don't want to do too much testing unless it is specific with it.

Here is what I've collected.

*** EDIT *** I have to put this log in its own post, because if individual post length limitations

Default Pass List IPs:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64