Snort Pass Lists + pfBlockerNG ingestion
-
Hello! Was hoping someone could clue me in on whether a planned dynamic pass list implementation is possible using pfBlockerNG.
Essentially pfBlockerNG will ingest html IP lists that are updated regularly (think Azure Service Tags et al) and create alias lists that are ingestible by Snort.
Then these aliases will be nested into the Snort passlist assigned to various interfaces. In this way those passlists could be updated dynamically, in time with pfBlockerNGs own cron timing.
Here's the issue:
I've seen info about Snort not updating the contents of pass list changes until it is restarted on the interface. However, a lot of the use cases I've seen don't quite line up with my own. I've seen threads where people manually update their Snort pass lists with new IP entries (which requires interface restart). I've seen attempts at dynamic passing through FQDN's as well (which Snort will not ingest).But if a nested alias is changing, is Snort capable of checking against changes to the underlying alias without needing to restart on the interface?
Does Snort update it's tables for the pass list with any sort of regularity without the restart? Even having it update on a semi-daily basis would be beneficial.
Sorry if I'm barking up the wrong IDS solution with this.
-
I maintain the Snort package, so I know how it works internally.
Snort is incapable of handling any type of dynamic list. It also is incapable of handling FQDN aliases. The Snort binary reads its pass list at startup and stores the contents in a RAM construct. It cannot update that list dynamically. You must stop the binary, create an updated list, then restart the binary so it will read the updated list in RAM.
So what you describe in your wish list is not currently available. However, I am always open to new feature Pull Requests. You are welcome to submit code to implement this here (for the customized Snort binary used on pfSense): https://github.com/pfsense/FreeBSD-ports/tree/devel/security/snort, and here (for the GUI package): https://github.com/pfsense/FreeBSD-ports/tree/devel/security/pfSense-pkg-snort.
-
Thank you for the quick response. Maybe our approach is completely wrong, but it would seem that this would be a common issue. How does a company that relies on cloud resources (with changing IPs) always allow those resources, without disabling or suppressing so many snort rules that the product becomes ineffective?
-
@rtw915 said in Snort Pass Lists + pfBlockerNG ingestion:
Thank you for the quick response. Maybe our approach is completely wrong, but it would seem that this would be a common issue. How does a company that relies on cloud resources (with changing IPs) always allow those resources, without disabling or suppressing so many snort rules that the product becomes ineffective?
The problem with dynamic IP addresses and an IPS is the overhead of constantly looking up IP addresses for domain names. Do you want the IPS to hold the packet stream for possibly more than two minutes while a DNS lookup completes (in the extreme case where the first DNS server times out)? What would that do to throughput? Even a fast DNS lookup is a very long time when compared to a stream of packets going through a firewall.
And in the case of web sites behind load balancers and CDNs, what good is caching a domain lookup? Many sites have TTL values as short as two minutes or less (ridiculous in my view, but some do it). So the IDS would have to basically do a DNS lookup on every new TCP stream or UDP session. That would kill throughput. What should the IDS/IPS do with the traffic while waiting for the DNS lookup?
Even today, the FQDN capability for aliases in pfSense relies on doing DNS lookups every 5 minutes. For those 2-minute TTL sites, the value currently held in the FQDN alias could very well be incorrect. And many such load balancing and CDN sites return different IP addresses for each DNS query. So client #1 in your network gets IP address X, and when client #2 queries it gets IP address Y, but when the firewall itself queried to update the FQDN alias it got IP address Z. All three are correct, but none of them is "right" ... .
So there is no good answer to this problem. It can work for web sites that have somewhat static IP addresses (that is, they are not behind any load balancer or CDN), but if that is the case why would you need a FQDN alias in the first place? Just use the static IP.
In your case, if you use cloud resources with dynamnic IPs, then you will need to define some Pass List entries that contain all the possible IP addresses (or perhaps all of the possible subnets).
-
Thank you for the detailed response. I completely understand the FQDN limitations here and I honestly don't understand how inbound resolutions per request could even happen, let alone at a reasonable latency to not impact performance.
I do have a specific example of what we our goal is, but this use case applies in many instances for us. We want to allow Google bots to scrape our websites and for it to never get blocked. Google provides this: https://www.gstatic.com/ipranges/goog.json which contains the subnets that we want to never be blocked. These lists can be updated by the cloud resource, but not too frequently. Leveraging a list like this wouldn't require a DNS lookup, but obviously if Snort never updates it's whitelists except on reload the cached data would become stale over time.
-
@rtw915 said in Snort Pass Lists + pfBlockerNG ingestion:
Thank you for the detailed response. I completely understand the FQDN limitations here and I honestly don't understand how inbound resolutions per request could even happen, let alone at a reasonable latency to not impact performance.
I do have a specific example of what we our goal is, but this use case applies in many instances for us. We want to allow Google bots to scrape our websites and for it to never get blocked. Google provides this: https://www.gstatic.com/ipranges/goog.json which contains the subnets that we want to never be blocked. These lists can be updated by the cloud resource, but not too frequently. Leveraging a list like this wouldn't require a DNS lookup, but obviously if Snort never updates it's whitelists except on reload the cached data would become stale over time.
Well, a Snort restart happens pretty quickly, and it restarts on every rules update (the periodic update cron task). So maybe you simply create a shell script that restarts Snort when the list is updated. Snort's Pass List is a plaintext file located in the configuration subdirectory for the interface. If you look under
/usr/local/etc/snort/snort_xxxxxx
you will find the configuration information for the Snort instance on that interface. Each configured Snort interface has its own subdirectory under/usr/local/etc/snort
. The subdirectory name consists of the physical NIC device name and a random UUID number. So a shell script could update the text file in the subdirectory for the Pass List, and then call/usr/local/etc/rc.d/snort.sh restart
to restart Snort all all interfaces. This would cause each configured Snort instance to read the configuration files and restart.This would be a workaround, and the changes you make to the Pass List that way would not persist if you restart Snort from the GUI controls. This is because the GUI controls will read the configuration from the firewall's
config.xml
file and rebuild the Pass List file from that data. Obviously that would not contain any updates done directly to the text file in the Snort configuration subdirectory. Packages on pfSense store configuration data in the firewall'sconfig.xml
file and use that to recreate all the text-based config files for the binaries (snort, in this case). One key distinction between restarting from the GUI and restarting via that shell script I referenced is the shell script will NOT rebuild the config files. It will use what exists currently in the subdirectory. Only the GUI restart will rebuild the config files from the data residing inconfig.xml
on the firewall.There are a couple of other more advance methods you could use, but all of them are workarounds of one sort or the other.
-
Would you have proposed is very interesting, and has given me an idea. Can you please punch some holes in this? We currently pull that JSON file in with PFblockerNG using a pass alias. Then we nest the dynamic list into our Snort_master_passlist alias which is then used as the "custom IP address from configured alias" field in the Snort past list config. PFblockerNG Cron job runs on the hour. Would we be able to run this: /usr/local/etc/rc.d/snort.sh restart five past every hour as its own independent Cron job, and update Snort that way?
-
@rtw915 said in Snort Pass Lists + pfBlockerNG ingestion:
Would you have proposed is very interesting, and has given me an idea. Can you please punch some holes in this? We currently pull that JSON file in with PFblockerNG using a pass alias. Then we nest the dynamic list into our Snort_master_passlist alias which is then used as the "custom IP address from configured alias" field in the Snort past list config. PFblockerNG Cron job runs on the hour. Would we be able to run this: /usr/local/etc/rc.d/snort.sh restart five past every hour as its own independent Cron job, and update Snort that way?
Sure. Just make sure your restart cron job does not happen at the same time as say the daily rules update cron task (you set that interval on the GLOBAL SETTINGS tab). You don't want them both trying to restart Snort at the same time.
Your problem has also given me an idea for an improvement in the custom blocking module I wrote for Snort. While a true realtime FQDN-type alias is not feasible, I can have the blocking module accept pfSense aliases directly into the Pass List file. Currently, when the GUI is creating the file, it resolves the alias into IP addresses and writes those directly into the file. I am considering making the blocking module itself able to digest the alias name. Then it can do a quick search of the
pf
tables via a FreeBSD system call to see if an IP matches (that is, the IP is currently stored in apf
table by the firewall). So in this way, as you update aliases in pfSense, Snort could "see" those updates essentially in realtime. -
@bmeeks said in Snort Pass Lists + pfBlockerNG ingestion:
Your problem has also given me an idea for an improvement in the custom blocking module I wrote for Snort. While a true realtime FQDN-type alias is not feasible, I can have the blocking module accept pfSense aliases directly into the Pass List file. Currently, when the GUI is creating the file, it resolves the alias into IP addresses and writes those directly into the file. I am considering making the blocking module itself able to digest the alias name. Then it can do a quick search of the pf tables via a FreeBSD system call to see if an IP matches (that is, the IP is currently stored in a pf table by the firewall). So in this way, as you update aliases in pfSense, Snort could "see" those updates essentially in realtime.
Wow, that sounds awesome! I really hope you could implement this. Thank you for your help.
-
@rtw915 said in Snort Pass Lists + pfBlockerNG ingestion:
@bmeeks said in Snort Pass Lists + pfBlockerNG ingestion:
Your problem has also given me an idea for an improvement in the custom blocking module I wrote for Snort. While a true realtime FQDN-type alias is not feasible, I can have the blocking module accept pfSense aliases directly into the Pass List file. Currently, when the GUI is creating the file, it resolves the alias into IP addresses and writes those directly into the file. I am considering making the blocking module itself able to digest the alias name. Then it can do a quick search of the pf tables via a FreeBSD system call to see if an IP matches (that is, the IP is currently stored in a pf table by the firewall). So in this way, as you update aliases in pfSense, Snort could "see" those updates essentially in realtime.
Wow, that sounds awesome! I really hope you could implement this. Thank you for your help.
Something pretty much like you said you needed is coming soon. Look for the update in the pfSense DEVEL snapshots in the near future. Here is a post I made describing the new feature: https://forum.netgate.com/topic/160771/new-often-requested-snort-feature-coming-soon.
-
@bmeeks said in Snort Pass Lists + pfBlockerNG ingestion:
Something pretty much like you said you needed is coming soon. Look for the update in the pfSense DEVEL snapshots in the near future. Here is a post I made describing the new feature: https://forum.netgate.com/topic/160771/new-often-requested-snort-feature-coming-soon.
Thank you so much! That looks amazing.