Web filtering / blocking - pfBlockerNG DNSBL category memory inquiry - alternative options?

[mzaknoen]

Looking to block malicious sites on the network (gambling, adult, etc)

Read about SquidGuard, but it looks like it is being phased out and pfBlockerNG is the way to go.

Just installed pfBlockerNG DNSBL, did not install the "devel" version, initially I thought it may be development only, but upon reading it looks to be pretty stable, so I can certainly uninstall / re-install.

XG-2758 (I know, EOL - upgrading soon)

Intel(R) Atom(TM) CPU C2758 @ 2.40GHz
Current: 1200 MHz, Max: 2400 MHz
8 CPUs : 1 package(s) x 8 core(s)

16GB Ram - memory usage is typically around 8-15%

100GB Disk space (96% free)

Looking into setting up category filtering, when checking the "adult" section, I am prompted with the memory warning. Based on my config - thoughts on if I have enough memory to accommodate the download of this giant list? I have read threads/posts on it crashing 8GB systems resulting in the firewall being unusable.

Dumb question - is there a way for me to download that list of IP's on a computer and copy is over to my Pfsense device to avoid RAM usage of downloading that big file.

Any alternative options for blocking specific categories? I have read about Cloudflare but haven't done too much research on implementing. Is it a difficult process to setup in Pfsense?

[jrey]

@mzaknoen

Looking to block malicious sites on the network

generally the issue regardless of what you are trying to do will be based on the list effectiveness

installed pfBlockerNG DNSBL, did not install the "devel" version

There is currently no difference and the "non - devel" version is the way to go for most users.

Looking into setting up category filtering, when checking the "adult" section, I am prompted with the memory warning.

Memory warning specifically saying?

What list specifically ? Guessing UT1 -> Adult ? that is something like 4.5 million "domains" with a file size of 122mb but well less than 8gb RAM
That said the list is also full of bloat.

is there a way for me to download that list of IP's on a computer and copy is over to my Pfsense device to avoid RAM usage of downloading that big file.

yes, but downloading and using ram based on what was downloaded are different things.

Any alternative options for blocking specific categories?

yes,
everything from education of users
to block everything and move to allow certain sites only
and anything in between.

there is a balance and that will be different for every use case.

consider the following sample from the UT1 adult list as I suspect that is what you are running into problems with.

Ask yourself, do I have a need for anything on blogspot.com?
Yes, write it down (you will want to whitelist that website(s) if you do)
No, nice, just continue

UT1 - Adult domains (raw file) 4,511,799 122mb
remove all blogspot lines (raw file) is now 891,692 domains and 18mb
add 1 line containing blogspot.com to the TLD list
( a quick DNS scan to others for example blogspot.hr are all cname or redirects to the .com)

Okay, i have not even tried to load the list, it is not a list I would ever consider using. There are other ways with far less impact.

However, for the purpose of this example I added only blogspot.com to a TLD for testing. I grabbed one of the URLs from the list (bad me)

then over to a browser. Don't try this at home kids

immediately gets the redirect

and in the log we see the original request getting the cname reply
and the website being blocked by the 1 line added to the TLD

A second scan of the original domain list, shows that many of the names don't even resolve, so those are just old and could also be removed.

Size of list does not equate to "effectiveness" of the list -- also applies to any/all of the available lists.

If the math is correct I've removed 3,620,107 lines from the file, and effectively have the same blocking with the addition of 1 line. Of course I'm not going to try every single one, although it would be easy enough to script a test.
This is where the user education can come into play, why on earth would you be going to a website like

zxaswdserdwokgkmbjnhntbftherhbfokmlplfnvhrfdx.(some TLD)

certainly not by typing that address in.

Often it is better, to determine what needs to be blocked specifically by reviewing logs. Do I use lists, certainly do. But certainly no need to hit the finishing nail with a sledge hammer.

Memory is pretty flat lined here - holding at