Suricata Pass Lists and Alias (URL/URL Table)

justme2

All,

In searching did not find this specific topic addressed - apologies if this is a duplicate.

Premise for use case: ability to digest IPs/Networks from 3rd party source(s) where transactions to/from those destinations must supersede IDS/IPS blocking (these would still be captured in the log for retroactive review). While for all other destinations IDS/IPS blocking must supersede transaction. This inherently avoids reducing overall security stance as various rules would not have to be disabled to guarantee passage for certain destinations. If its a choice between disabling a rule (due to false positive) or bypassing (via pass list) for certain destinations, seems that the bypass is the lesser of two evils as the rules would still apply for all other destinations to maximize protection potential.

In various cases it makes sense to be able to utilize an external IP Address list (maintained by the source entity): URL (IPs) or URL Table (IPs) within certain rules - including the use of that alias within Suricata. However, it appears that unless the list is local - it cannot be used by Suricata. Issues at play:

Creates significant overhead to duplicate and maintain that same list on a given install.
Any organization that would have multiple/numerous installs - becomes nightmarish to maintain all those Aliases across all those installs.

For clarity, even if there were a management console solution for fleets of pfSense installs - using the source URI would be the [strongly] preferred method as it enables a litany of use cases that still would not be plausible via a management console and it means that each install is able to obtain the list independently (also accounts for instances where the response data may be different by geo/region/et al). One other issue is that some sources are plain text and some are JSON formatted (some sources only provide a JSON formatted option) - the JSON formatted sources do not appear to be consumable for aliases, at present.

Has anyone else run into this? Any solution(s)? One quick [and ugly] means appears to be to use perl to grab the source [JSON] data and generate a text file [consumable by the built-in retrieval function] on a central pfSense install, then have the other pfSense installs pull (via URL/URL Table) from that central one over VPN. However, that still doesn't solve the problem of not being able to use that alias within Suricata.

FWIW - did not test with Snort as Suricata was required, but suspect it may be the same issue as various code segments seem to be shared between the two.

Thanks!

bmeeks

Both Suricata and Snort can use aliases in a Pass List with some restrictions. The alias must be of a type that creates its own pf table and populates that table with IP addresses. You can examine pf tables under DIAGNOSTICS > TABLES in the pfSense menu.

You can enter the aliases by constructing a custom Pass List and adding the aliases at the bottom under the Addresses section. Add as many lines there as you need.

One big caveat to this answer is that I have tested with FQDN (fully qualified domain name) aliases, but I have not tested URL Table aliases. I do know that the internal custom blocking module will look for alias names in the list of pf tables, and if it finds a match will use the IP addresses from the table in the pass list logic. So if you have a table name showing under DIAGNOSTICS > TABLES, then you can put that name down as an address in the Pass List.

Aliases on pfSense are typically resolved once per 5-minutes for FQDN aliases. I think some of the others have different update times. As I said, I've never experimented with URL Table aliases. The Pass List logic in the custom blocking plugin is simply looking for a legal IP address in any pf table it references.

justme2

Understood. Here is an actual example of what does not work:

Create an Alias as URL Table (IPs)
Name used was "IP_Zoom" (if that matters)
Use "https://assets.zoom.us/docs/ipranges/ZoomCDN.txt" as the URI (used 3 days as an arbitrary value)
The contents will appear under Diagnostics -> Tables
However, when editing a Pass List for Suricata, you cannot add this as an item within the Pass List (or at least as you start to type "IP_" - it does not appear in the list as a select-able item. (Inference being that its not viable for use within a Pass list - perhaps the lack of it appearing is the bug and it can still be utilized?)

Tried a couple different variations and none seem to work - despite their appearance under Diagnostics -> Table. It seems to be a common scenario is that if the contents of the list are populated from an external source, then the alias is not available for use within a Pass List. Aliases (including those with DNS resolution) that are "locally sourced" appear to be viable for use within a Pass List.

Thanks!

bmeeks

@justme2 said in Suricata Pass Lists and Alias (URL/URL Table):

Understood. Here is an actual example of what does not work:

Create an Alias as URL Table (IPs)

Name used was "IP_Zoom" (if that matters)

Use "https://assets.zoom.us/docs/ipranges/ZoomCDN.txt" as the URI (used 3 days as an arbitrary value)

The contents will appear under Diagnostics -> Tables

However, when editing a Pass List for Suricata, you cannot add this as an item within the Pass List (or at least as you start to type "IP_" - it does not appear in the list as a select-able item. (Inference being that its not viable for use within a Pass list - perhaps the lack of it appearing is the bug and it can still be utilized?)

Tried a couple different variations and none seem to work - despite their appearance under Diagnostics -> Table. It seems to be a common scenario is that if the contents of the list are populated from an external source, then the alias is not available for use within a Pass List. Aliases (including those with DNS resolution) that are "locally sourced" appear to be viable for use within a Pass List.

Thanks!
What version of the Suricata package are you running and are you using pfSense 2.6 or pfSense Plus 23.01?

I am pretty sure this will work, but some minor code changes are necessary.

Update:
Tested this in a Suricata package update I'm working on now, and it worked. Some code changes in a few places are required, but they are pretty minor. ~~This new ability will be in the upcoming Suricata-6.0.10_2 package update for pfSense Plus 23.01 and 2.7.0 CE snapshots (once the snapshots resume building).~~

For the moment pfSense 2.6.0 CE is not getting package updates, so if you unless you are running 23.01 in production you will not see this update.

Later Update:
Sorry, but I need to amend the date for this coming out. The Netgate team was fast and merged my other Suricata 6.0.10_2 package updates just a bit before I committed this URL Table Alias change. I will definitely get this in the next update, but it did not get into the just merged 6.0.10_2 update. Sorry.

justme2

So just tested and received "Custom Address entry '<NAME>' is not a Host or Network Alias!". That was using the correct value (case matched) for the Alias.

Test environment has: Suricata package v6.0.4_1

bmeeks

@justme2 said in Suricata Pass Lists and Alias (URL/URL Table):

So just tested and received "Custom Address entry '<NAME>' is not a Host or Network Alias!". That was using the correct value (case matched) for the Alias.

Test environment has: Suricata package v6.0.4_1

Yeah, in my initial rush to experiment I missed two other code changes. One is in the verification code in the Pass List edit source file, but there are some other changes needed in the main Pass List building function in the snort.inc file. The feature does work, I've verified that. But several changes are required to pull it all together.

justme2

Understood. Looking forward to the changes in an upcoming version.

Thanks!

bmeeks

@justme2 said in Suricata Pass Lists and Alias (URL/URL Table):

Understood. Looking forward to the changes in an upcoming version.

Thanks!

I'm working on a 6.0.10_3 update now for the 23.01 and 2.7 CE branches. The 6.0.10_3 update will have the URL Tables feature.

I'm not sure about backporting the feature to the 6.0.4 Suricata branch in pfSense 2.6 CE. There are lots of changes in the new Suricata PHP code base due to changing over to PHP 8.1 in pfSense. The PHP 8.1 changes make direct backporting impossible. You have to carefully cherry pick and then rewrite/refactor the code to adapt it back to PHP 7.4.

Ultimately what happens to packages for 2.6.0 CE depends on how long the 2.7 CE release schedule gets. If 2.7 CE winds up delayed for months, then maybe some targeted package updates can be done in the 2.6.0 CE branch.

justme2

Fair enough, sounds good.

Thanks!

bmeeks

@justme2 said in Suricata Pass Lists and Alias (URL/URL Table):

Fair enough, sounds good.

Thanks!

I have it working in a new Suricata package for 23.01 and 2.7 CE Devel.

This feature will be in the next update of Suricata that shows up for 23.01 pfSense Plus and 2.7 CE snapshot users. The package version will be 6.0.10_3.

Here are some screenshots showing the new feature in action.

Defined the URL Table alias under FIREWALL > ALIASES > URLs:

Assigned the URL Table alias to a custom Pass List on the PASS LISTS tab:

Assigned the custom Pass List to the LAN interface in Suricata under INTERFACE SETTINGS:

Here is the content of the custom Pass List when using View List on the INTERFACE SETTINGS tab for the LAN:

And here is the suricata.log startup info for the LAN interface showing the custom blocking plugin read and processed the new "IP_Zoom" table alias:

Looking under DIAGNOSTICS > TABLES shows there are 3525 entries in this URL Table: