Suricata not respecting pass list

sef1414

I've had this issue for a couple months now, wanted to see if anyone had any ideas. There are a few posts out there that suggest other users having similar issues, but I'm not sure they are exactly the same.

I am running Suricata in legacy mode. I have a pass list configured. It has all the boxes checked, and then points to one firewall alias. This alias is a list of about 800 IPs. On the Suricata interface, I have the custom pass list selected. I've restarted the interface, service, as well as pfsense, many times. However, it still blocks IPs that are on my pass list. I thought perhaps there was an issue with nested aliases, (using aliases within the alias selected on my pass list) so I removed those, and still no dice.

Thanks for any help.

bmeeks

Two questions for you.

What version of the Suricata package are you running currently? The latest version is 6.0.4.
Are the IP addresses getting blocked coming from a FQDN (fully qualified domain name alias)? Sometimes those can be very tricky to use because of short TTL values used on many CDN and similar networks.

This has been a sporadic and seemingly random issue for several years. I have never been able to duplicate in my test environment, and I have tried and tried and tried again. I have had users send me their exact Pass List and I have copied straight into my test virtual machines, and still have not been able to replicate this issue.

Because I can't replicate it, troubleshooting it consists basically of trying various coding "shots in the dark". That's one reason I'm asking for your current Suricata package version. There were a few new changes introduced in 6.0.4 in a further attempt to maybe fix this.

The code uses built-in functions from the Suricata binary to store Pass List IP addresses and networks, and then calls other functions in the binary to check if the IP address from an alerting packet is present in the Radix Tree structure the Suricata binary maintains.

sef1414

@bmeeks Ok, sounds similar to what you were saying on a thread I read from around this time last year. I actually didn't want to cause confusion, but I believe (I'm not certain) that it was working "mostly", and then stopped a few months ago, after a pfsense update. I was waiting to see if another update would resolve it and have just been busy. I say mostly, as I think its possible it was only FQDN that were getting blocked. At present moment, I'm seeing blocks on simple static IP hosts (not FQDN). I am on package 6.0.3_3. I will update now and see if it persists. Thanks for your work on this package.

bmeeks

@sef1414 said in Suricata not respecting pass list:

@bmeeks Ok, sounds similar to what you were saying on a thread I read from around this time last year. I actually didn't want to cause confusion, but I believe (I'm not certain) that it was working "mostly", and then stopped a few months ago, after a pfsense update. I was waiting to see if another update would resolve it and have just been busy. I say mostly, as I think its possible it was only FQDN that were getting blocked. At present moment, I'm seeing blocks on simple static IP hosts (not FQDN). I am on package 6.0.3_3. I will update now and see if it persists. Thanks for your work on this package.

I really sort of suspect some kind of bug in the Radix Tree code within the Suricata binary itself, but I am not familiar enough with the algorithm or the code to effectively debug it. Suricata uses a Radix Tree to store some IP info related to rule signatures, and I just borrowed the feature to store IP addresses input from a Pass List and then later search for matches. A Radix Tree lets you search by specific IP, but also can determine if a given IP address is contained within a subnet stored in the Radix Tree. That's what makes it versatile.

Because I have never been able to reliably replicate the bug, I suspect the problem in the Radix Tree might actually be something with multiple threads accessing the structure concurrently. If that is the case, it would explain why the bug is somewhat random, and also would explain why I have trouble replicating it. The latest 6.0.4 version of the Suricata package on pfSense contains some additional changes that read-lock the Radix Tree when the custom blocking module is searching it for matches (and also write-lock it when updating it with IP addresses). Those fixes are part of the "shot in the dark" programming efforts I mentioned earlier.

sef1414

@bmeeks Awesome. I will report back here if that resolves it.

sef1414

@bmeeks As mentioned this has been going on for a few months.. I think I've updated once, maybe twice. But.. that one may have done the trick. So far, it looks good. Will keep an eye on it for the next week or so and post back here either way. Thanks for your help.

sef1414

@bmeeks Just a follow up here. No issues for a few weeks. Then, all of a sudden last week, whitelisted IP's started getting banned. Haven't updated anything since, going to do that. One observation, I wonder if it could be related... my swap is full. I haven't got around to increasing swap size, as it seems to be slightly less than straightforward. I've wondered why that happens, when I have 16GB memory and ample memory available, but from some reading maybe that is just how FreeBSD operates. Anyways, not looking to trouble shoot that, just thought it was worth mentioning in case that could be impacting it.

bmeeks

@sef1414 said in Suricata not respecting pass list:

@bmeeks Just a follow up here. No issues for a few weeks. Then, all of a sudden last week, whitelisted IP's started getting banned. Haven't updated anything since, going to do that. One observation, I wonder if it could be related... my swap is full. I haven't got around to increasing swap size, as it seems to be slightly less than straightforward. I've wondered why that happens, when I have 16GB memory and ample memory available, but from some reading maybe that is just how FreeBSD operates. Anyways, not looking to trouble shoot that, just thought it was worth mentioning in case that could be impacting it.

I seriously doubt the swap usage is related to Suricata. What version of pfSense are you running? If still on 2.5.2 CE (or Plus 21.05.2), there is a bug in the pcscd daemon that chews up memory and causes swap usage and eventual exhaustion if the box is not restarted. That bug is fixed in pfSense 2.6.0 CE and pfSense Plus 22.01. You may be getting hit by that bug.

I don't know if it will have an impact or not, but upstream Suricata has made some fixes in the Radix Tree code within the binary. I am hopeful they help with the longstanding random Pass List issue. The fixes are related to netmask issues within Suricata itself. They are slated to come out with the next Suricata binary updates.

sef1414

@bmeeks Ok, wasn't suggesting Suricata was causing the swap usage issue, rather just including as potentially relevant info. After all, I am a layman when it comes to most of this stuff. I'm running 2.5.2, I will get that updated this week. I'm quite sure it is that bug, as it is a recent problem.

As for the pass list issue, thanks for the update. Sorry I didn't have better news to report.

xpxp2002

@bmeeks said in Suricata not respecting pass list:

I don't know if it will have an impact or not, but upstream Suricata has made some fixes in the Radix Tree code within the binary. I am hopeful they help with the longstanding random Pass List issue. The fixes are related to netmask issues within Suricata itself. They are slated to come out with the next Suricata binary updates.

I'm running into the same issue as sef1414. Looking in suricata.log, I think I found the issue.

It appears to be the mishandling of CIDRs in the pass list. For example, this is logged for one of my interfaces:

1/3/2022 -- 04:01:13 - <Info> -- alert-pf -> Received notification of IP address change on firewall interface hn4.
1/3/2022 -- 04:01:13 - <Info> -- alert-pf -> added address 192.168.10.1 to automatic firewall interface IP Pass List.

However, this interface is configured as 192.168.10.1/24. The pass list also shows 192.168.10.0/24. But according to this log, Suricata is only internally adding the interface address 192.168.10.1 to the pass list and ignoring the rest of the address space.

Hopefully the Suricata binary updates will be published to the pfSense package repo soon, as this bug has been breaking connectivity daily for hosts that should only have the non-local IP blocked, and required manual intervention each time to fix.

bmeeks

@xpxp2002 said in Suricata not respecting pass list:

@bmeeks said in Suricata not respecting pass list:

I don't know if it will have an impact or not, but upstream Suricata has made some fixes in the Radix Tree code within the binary. I am hopeful they help with the longstanding random Pass List issue. The fixes are related to netmask issues within Suricata itself. They are slated to come out with the next Suricata binary updates.

I'm running into the same issue as sef1414. Looking in suricata.log, I think I found the issue.

It appears to be the mishandling of CIDRs in the pass list. For example, this is logged for one of my interfaces:
1/3/2022 -- 04:01:13 - <Info> -- alert-pf -> Received notification of IP address change on firewall interface hn4.
1/3/2022 -- 04:01:13 - <Info> -- alert-pf -> added address 192.168.10.1 to automatic firewall interface IP Pass List.
However, this interface is configured as 192.168.10.1/24. The pass list also shows 192.168.10.0/24. But according to this log, Suricata is only internally adding the interface address 192.168.10.1 to the pass list and ignoring the rest of the address space.

Hopefully the Suricata binary updates will be published to the pfSense package repo soon, as this bug has been breaking connectivity daily for hosts that should only have the non-local IP blocked, and required manual intervention each time to fix.

You are confusing two related but different things that happen during runtime. The custom blocking module has the Pass List provided by the user, but it also generates its own internal dynamic pass list containing ONLY the firewall interface addresses. Meaning the IP addresses defined for each active firewall interface. A thread is launched that hooks to the kernel's routing table and receives updates for any interface IP addresses on the firewall that change. So that might happen if your WAN, for example, was provisioned via DHCP or PPPoE. When the WAN bounces, the interface IP might change. Using the monitoring thread, the custom blocking will update its internal pass list.

So in your scenario, if you examine the suricata.log file you should see entries during startup where Suricata loads your supplied Pass List (or else the default one). That should include the 192.168.10.0/24 network.

What should happen is the Radix Tree code adds the 192.168.10.1 address, but since it is already covered by an existing subnet defintion (192.168.10.0/24), there should be no change in outcome. But that is something I can investigate. Maybe the internal Suricata Radix Tree code is not behaving properly in that circumstance. Typically I was expecting the WAN IP address to be the one changing. That was an old problem that was fixed by adding the interface monitoring thread code a long time ago. Under the old system, the WAN IP address might change during Suricata runtime and the blocking module would be unaware. That would frequently result in the new WAN IP getting blocked. But with the interface monitoring thread, Suricata keeps its internal interface IP pass list updated with current changes.

xpxp2002

@bmeeks I understand. I see that in the Suricata logs during startup. I rebooted the firewall this morning to see if it would help, but I'm still seeing the same behavior. Is there anything else I can/should check? Or is this almost certainly a bug?

bmeeks

@xpxp2002 said in Suricata not respecting pass list:

@bmeeks I understand. I see that in the Suricata logs during startup. I rebooted the firewall this morning to see if it would help, but I'm still seeing the same behavior. Is there anything else I can/should check? Or is this almost certainly a bug?

Nothing I am aware of on the user side assuming the Pass List contains all the necessary networks.

I really think the problem is buried within the logic of the Radix Tree utility code supplied with the Suricata binary. There is an active bug report on it being worked by the Suricata team. I'm hoping the fixes they put in for their own use of the Radix Tree will also percolate down to the tree code behaving better with my Suricata custom blocking module.