PfBlockerNG - query on application of DNSBL - subdomain vs domain blocking

[Andrew453]

Hi

I've read through the posts on the DNSBL (domain level) blocking offered by pfBlockerNG and the various easy lists that can be used.

I can see that it seems to be deliberate that pfBlockerNG will only explicitly block the specific domains that are listed.

So, for example, an entry for abcdef.com, won't block ads.abcdef.com, seemingly deliberately.

Is that indeed the intended behaviour and if so, is it consistent with what's intended by the authors of the block lists?  Most of the block lists only include entries like abcdef.com … leaving ads.abcdef.com or similar to go straight through the filter (meaning it doesn't block ads in most cases!).

Is there any way to make it also block subdomains please?  It's possible in the dns resolver (you use local-zone in addition to local-data), but pfBlockerNG doesn't seem to be set up that way.

Thanks

Andrew

[BBcan177]

See the following:
https://forum.pfsense.org/index.php?topic=102470.msg593514#msg593514

[Andrew453]

Thanks, yes, I'd read that.

It's perfectly right that if a DNSBL includes ads.yahoo.com, you wouldn't want to block *.yahoo.com

My point is many of the DNSBLs appear to assume that if it had yahoo.com listed, that the filter would automatically block ads.yahoo.com (i.e. the other way around).  That's not happening in pfBlockerNG.

(Yahoo's a bad example, as it clearly has legit domains.  cf something like xxxads.com vs ads.xxxads.com.  At the moment, many clearly dodgy domains get straight through)

So the point is that there appears to be a mismatch of conventions between what the authors of the DNSBLs assume will be blocked, vs what pfBlockerNG is actually blocking, with the result that many DNSBLs are actually ineffective when applied through pfBlockerNG.

[BBcan177]

I have to think about how I would do that in the package :) Not just so straight forward…

It will wreak havoc with de-duplication trying to add the redirect lines to single domains, and filtering out duplicate domains that are in other feeds which are listing a sub-domain for a previously added domain with a redirect....  Unbound will crash if there are two lines with the same domain when using a "redirect" rule...

I will work away at it and see what I can come up with...

[Andrew453]

Thank you.  My suggestion would be:

1)  add a global option in pfBlockerNG for blocking entire domains where a DNSBL includes say abcdef.com (I think it has to be global option rather than per list, otherwise you run into trouble on de-duping as you say)
2)  do the combination/de-duping/suppression etc as per normal (so people can override the global behaviour on a per domain basis by suppressing e.g. abcdef.com, but adding ads.abcdef.com)
3)  at the point where pfBlockerNG translates the de-duped list into a conf file to pass to Unbound, check each domain being added:
  - if it is a domain: then add it into the conf file with a local zone too - i.e. block the lot
  - if it is a subdomain: then add it into the conf file with just a local data entry - i.e. just block the particular subdomain mentioned

I suspect you may need to build a list of top level domains into pfBlockerNG to do the last part.  You can't just count dots unfortunately, as abc.co.uk has two (but is still a domain), whereas abc.com only has one.

On the crashing point, one way to deal with that is to sort the de-duped list by domain (e.g. invert the character order of each domain string, then sort, then invert back).  Then when you apply the logic in (3), what you'll get is one zone redirect per domain, followed by all the data entries pertaining to that domain - hence no crash.

Just a thought as to how you might do it - I suspect it might be a bit more complicated!

Good luck, and thank you!

Andrew