PfBlockerNG w/ DNSBL > Squid(+Guard) for Content Filtering?

chrcoluk

squidgard can offer more refined filtering.

lets say e.g.

dropbox.com/someuserA/malware.zip
dropbox.com/safeuserB/picture.jpg

A DNSBL list might block the entire dropbox.com domain, which is not good, and is a big reason alexa is recommended to remove popular sites from the list.

Whilst squidguard could filter the first url whilst at the same time allowing the second url,

On the flipside I dont like sending all my http/https traffic through an intermediate proxy so I use pfblockerNG not squidguard.

What might be considered an ultimate solution is that only domains in the dnsbl are routed to squid, and all other domains are direct, this is how UK isp's filter IWF traffic.

In regards to blocking the resolved ip's of domains in dnsbl, you got 2 obvious issues.

1 - the processing time to resolve everything in the list to generate the ip table would be very significant, and could even get you blocked on dns resolvers for a flood of connections.
2 - ip addresses's can be shared by many domains so if you block ip's then you are probably also going to be blocking harmless sites as well sharing the same ip address.

guardian

@chrcoluk:

On the flipside I dont like sending all my http/https traffic through an intermediate proxy so I use pfblockerNG not squidguard.

What might be considered an ultimate solution is that only domains in the dnsbl are routed to squid, and all other domains are direct, this is how UK isp's filter IWF traffic.

Just curious what your concern is?

On the flipside I dont like sending all my http/https traffic through an intermediate proxy so I use pfblockerNG not squidguard.

Security? / Performance? / Something else?

Unless you give it your browser keys, SquidGuard isn't going to be able to examine https anyway.

chrcoluk

performance and possible compatibility breakage on sites.

javcasta

@pfBasic:

Originally I had tried squidguard with both transparent proxy or wpad. It was a pain to implement and never got great results, so I quit.

pfbng w/ dnsbl has been excellent. it just seems to work.

Recently I found out how to easily use shallalist by category with DNSBL and have it updated weekly with cron. Thanks to javcasta
https://forum.pfsense.org/index.php?topic=120072.0#msg664172

This combined with redirecting all DNS requests to pfsense (DNSBL) seems to me to be a very easy to implement and very effective method of content filtering your network.
https://doc.pfsense.org/index.php/Redirecting_all_DNS_Requests_to_pfSense

I'm posting this because it doesn't seem like many people are using pfbng & dnsbl for this type of content filtering and I've seen it mentioned that people should use squidguard for that.

However, this method seems to filter http & https without any client level modification or complex pfsense configuration.

Is there something obvious that I'm missing as to why this doesn't seem to be heavily utilized for this type (content based lists) of content filtering? Or it it and I just now figured it out?

Also, how could someone on my network bypass/defeat this filtering setup?

Hi. Thanks for mentions.
I have a new version for this script: https://forum.pfsense.org/index.php?topic=120072.msg685621#msg685621

Regards.

walkingman

Hi All
I have been looking at this and have hit a problem i hope you can help me with. If i enter in a browser for example youporn.com it will be blocked as expected but if i enter www.youporn.com it is not blocked. I have tested this on multiple sites with the same result.
Is it posible to include the www without editing the block list by hand.

Thanks

BBcan177

@walkingman:

Hi All
I have been looking at this and have hit a problem i hope you can help me with. If i enter in a browser for example youporn.com it will be blocked as expected but if i enter www.youporn.com it is not blocked. I have tested this on multiple sites with the same result.
Is it posible to include the www without editing the block list by hand.

Thanks

You will need to enable the TLD option. Otherwise only the listed domains are blocked. Will need to run a Force Reload-DNSBL for it to take effect.

pfBasic

Keep in mind that TLD needs a lot of RAM. It's dependent on the number of hosts you have blocked in your lists. I have 8GB RAM and TLD used it all then my system crashed, maybe someday I'll throw some more RAM in there.

BBcan177

The pkg has limits on how many domains are processed via TLD. So it should have prevented the unbound memory exhaustion issue. If it's reproducible let me know and I may tweak the limits. The more details the better.

walkingman

Thanks for the help, but enabling TLD on my meager system has limited results because of the table size. Looks like i will
have to return to squid/squidgaurd.

BBcan177

@walkingman:

Thanks for the help, but enabling TLD on my meager system has limited results because of the table size. Looks like i will
have to return to squid/squidgaurd.

Sorry nothing I can control as Unbound creates pointers and uses more memory. Maybe it's time to upgrade. :)

MrGlasspoole

Just reading this and i wonder about the RAM.
I have 2GB and TLD enabled in DNSBL and pfSense says "Memory usage 26% of 2013 MiB".

I have 10 host sources in DNSBL Feeds and EasyList enabled. So how can 8GB be not enough?

Is there a place I didn't see yet where i see how many domains are blocked?
Something like what you see if you update?

pfBasic

I don't think that TLD alone was the reason my system crashed. I also use ZFS and RAM disk. I have probably about ten lists as well but a couple of them have a LOT of domains.
I think TLD worked as it was supposed to but since I'm using RAM disk and ZFS I don't think the system appreciated such a small amount of RAM available and gave it up.

I might try enabling it again to see what happens later and report back. But at this point I don't really need TLD. When I do I'll just add more RAM so that I have enough for all of my domains.

BBcan177

The blue infoblock for TLD will indicate recommended RAM requirements. You can review the pfblockerng.log in the Update tab window during any manual force runs. Or review the log in the Log browser tab. The that log will show stats per download and also the TLD stats after all DNSBL feeds have been downloaded and processed.

mod

@mrglasspoole hi;
I had to turn off ipv6 which i don't use(ignored in linux) and this is on a dual core AMD APU with 4 gigs.
I also have issues with abuse list at first before killing ipv6 in pf latest and running out of ram issue.
but at least it tells you and shuts down.
but would like the caching squid had and av.