PfBlockerNG v2.1 w/TLD

BBcan177

Is it possible to use some of the paid databases from Maxmind with pfBlockerNG ? (other than GeoLite2)

And if possible, how ?

Hi, I have not tested it, but if you do have a paid subscription, I would think that they would give you an API key that could be added to the MaxMind URL…

You could manually edit the URL in /usr/local/www/pfblockerng/pfblockerng.php

https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/www/pfblockerng/pfblockerng.php#L52

And adjust the MaxMind cron task to run once a week (I think thats the update freq. of the paid subscription database).

https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L5303

LIGISTX

@RonpfS:

http://10.10.10.1:8443 return a gif

It should be https://10.10.10.1:443 but that doesn't return and doesn't it log to dnsbl.log either.

Sorry guys if this is dumb, or I missed something obvious. I am power reading as much as I can possibly stand (eyes starting to hurt) about PFBNG, and I came across this and from my understanding from previous reading, this site should be blocked via all of the settings I have… I think?

I navigated to https://10.10.10.1:443 in chrome and got the "Your connection is not private" warning, not a Gif. This tells me something isn't working correct. I am not even sure where to start with my question, so I will start at the obvious:

This does mean something isn't working, correct? If so, what exactly may that be?

Once again, just trying to get my feet wet and start actually using pfsense for what its worth, I have had my box deployed for about a month now with no packages running, and for better or worse this was my first package to try!

Edit

Also, is there a list somewhere of all of the available DNSBL Feeds? I have read through lots of pages and have input all of BBCan's lists that I have found, I didn't document them.... and I am not sure how to printout a list of all that are currently running on my pfsense box. Like I said, pretty noob, but very interested in learning :)

I guess to this same question, like I said, I haven't kept a list, would it be an issue to have the same feed in multiple groups? I don't think this has happened, but without keeping valiant records of what I have added, it would be hard to even figure this out unless there is a way to show what all has been loaded and running (I assume there is a way to do this).

And what I have seen BBCan117 write SO MANY times is, its better not to block the world, still trying to figure out exactly how to do that...

Also, what I can say, I am somewhat happy I started using this, finally using the 8 GBs of RAM I put in this box! Up to almost 70% utalization, before PFBNG I was at ~2% LOL!

BBcan177

See the following:
https://forum.pfsense.org/index.php?topic=119954.msg663601#msg663601

LIGISTX

@BBcan177:

See the following:
https://forum.pfsense.org/index.php?topic=119954.msg663601#msg663601

Ok. Well, maybe it is ok. I pulled up 10.10.10.1:443 on my iPad s and just allowed it to show the website. The iPad tab in chrome did say "1x1 pixel". So I am thinking this means it actually is working, I doubt the chrome tab would say 1x1 pixel on its own lol.

LIGISTX

Edited, I think I have figured out my issue, it was a dumb question in the first place :-[

BluBoy

I just wanted to ask if I can (ab)use the GeoIP data in PFBlocker.

I'm currently using it to drop outbound traffic to certain countries.
I'd also like to set up multiple aliases, so I can route traffic to certain other countries via a designated interface.

Is this possible?

BBcan177

@BluBoy:

I just wanted to ask if I can (ab)use the GeoIP data in PFBlocker.

I'm currently using it to drop outbound traffic to certain countries.
I'd also like to set up multiple aliases, so I can route traffic to certain other countries via a designated interface.

Is this possible?

You can goto the IPv4/6 Tab and create any combination of GeoIP aliases… Click the blue infoblock icons for further details... You can also use "Alias type" settings which will allow you to manually create your firewall rules and add the GeoIP Aliases as required.

pftdm007

How do you guys block these annoyings youtube ads popping up before the video playback starts or worst are showing up randomly during playback?

This is weird, I am not seeing any form or kind of advertisement on my computers, but on our android devides we see those. Two examples:

https://youtu.be/weyiTnwtGrk
https://youtu.be/d-AH_IEnLIg

Damn I hate those!

I currently maintain a decent list of DNSBL ads feeds but for whatever reasons the ads are getting by.

http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://hosts-file.net/ad_servers.txt
https://adaway.org/hosts.txt
http://sysctl.org/cameleon/hosts
http://winhelp2002.mvps.org/hosts.txt

shonjir

Hi, BBcan -

First, I'd like to thank you for this excellent tool!

I've been testing out the TLD feature with version 2.1.1_6 with a set of DNSBL lists totaling more than 600k domains. It's working really well, but as you explained it uses a lot of memory.

I think the memory situation could be improved significantly if deduplication could be improved to identify redundant subdomains that don't have a corresponding higher level domain in the list.

I've noticed that some of the larger lists I'm using (for example, hosts-file.net) contains large numbers of domains with randomly generated third, fourth, or fifth level subdomains. Other than Bambanek's DGA list (all 2nd level) most of the generated names are third level or below.

There are a few methods I can think of that this could be implemented.

The easiest method would be to simply truncate subdomains lower than a certain level (say, second, third or fourth) before deduplication. This would be relatively fast and should work relatively well. A configuration option could allow the user to determine how aggressively to apply this. More aggressive truncation would result in substantially shorter lists.

Another way is to dynamically by count the number of subdomains for each higher level domain, and if the count is greater than some configurable threshold replacing subdomains with the higher level domain. This would take more processing but could be a way to identify domains that could be replaced by a wildcard.

BBcan177

@shonjir:

Hi, BBcan -

First, I'd like to thank you for this excellent tool!

Thanks!

I think the memory situation could be improved significantly if deduplication could be improved to identify redundant subdomains that don't have a corresponding higher level domain in the list.

I have intentions to add "Reputation" options to DNSBL… Similar to "Reputation" in the IPv4 section. Maybe after the next release of the package.

TLD will block the whole domain/sub-domains if the root domain is listed... Its really strange that some of these feeds will list >50 Sub-domains which are obviously DGA but not list the root domain name?

You can run the following command to find which domains are listed over a specific count.

You can modify this command:

find the first two TLD levels ( cut -d '.' -f 1-2)
find the first three TLD levels ( cut -d '.' -f 1-3)

find count of 30 ( max=30 )

sed 's/local-data: \"\(.*\) 60.*/\1/' /var/unbound/pfb_dnsbl.conf | rev | cut -d '.' -f 1-2 | rev | awk -v max=50 '{a[$0]++}END{for(i in a){if(a[i] > max){print i}}}'

Then you can choose which of these repeated domains to add to a [u][b]DNSBL Customlist[/b][/u], which will block the domain/sub-domains and remove all the duplicated entries... After adding them, run a "Force reload - DNSBL"

Keep in mind that (msn.com and amazonaws.com) also have a lot of sub-domains listed...

[code]grep "msn.com" /var/unbound/pfb_dnsbl.conf
grep "amazonaws.com" /var/unbound/pfb_dnsbl.conf[/code][/i]

shonjir

Glad to hear you're planning a reputation system - that is basically what I was thinking when I'd suggested counting subdomains.

@BBcan177:

sed 's/local-data: \"\(.*\) 60.*/\1/' /var/unbound/pfb_dnsbl.conf | rev | cut -d '.' -f 1-2 | rev | awk -v max=50 '{a[$0]++}END{for(i in a){if(a[i] > max){print i}}}'

Then you can choose which of these repeated domains to add to a [u][b]DNSBL Customlist[/b][/u], which will block the domain/sub-domains and remove all the duplicated entries... After adding them, run a "Force reload - DNSBL"

This is an awesome suggestion!  I hadn't thought of periodically searching for DGA type domains as a secondary process to build a filter.  That would allow experimenting with various algorithms.

For example, I was thinking today that maybe the TLD filter list could be used to moderate the clipping process so that it doesn't disproportionately affect country-code sites.
[/i]

BBcan177

@shonjir:

For example, I was thinking today that maybe the TLD filter list could be used to moderate the clipping process so that it doesn't disproportionately affect country-code sites.

There is an option in Alexa TOP1M, that allows for whitelisting specific TLDs… So I guess it could work similar to that where selected ccTLDs are excluded from the Reputation process... Will keep this in mind.... Thanks!

wallestack

BBcan177,

Thank you for all the work you’ve done in developing pfBlockerNG. I stumbled across pfBlockerNG while looking for an adblock solution on pfSense and have been pleased at how well it works.

I would like to share my experience with pfBlockerNG DNSBL on an iOS device and suggest a new feature. As some other people have pointed out, Safari on iOS presents a certificate warning for the HTTPS sites DNSBL redirects. (https://www.reddit.com/r/PFSENSE/comments/4f56zk/blocking_ads_using_pfblockerng_dnsbl/) As you point out, other browsers like Chrome on iOS behave better and silently drop the redirected requests. While the iOS device is connected to a network that is protected by pfBlockerNG, Chrome presents a much better user experience since it does not constantly show certificate warnings.

However, Chrome on iOS does not support adblock extensions like Safari does. When the iOS device moves to a network that isn’t protected by pfBlockerNG, sites loaded by Chrome now contain ads. Ironically, Safari presents a better user experience since it supports adblock extensions and does not present certificate warnings for ad domains.

A frustrating trivia fact: on the latest version of iOS, the only way to remove a certificate that was trusted in Safari is to use the “Reset All Settings” option. (https://apple.stackexchange.com/questions/130612/how-do-you-remove-ssl-certificate-exceptions-on-ios/130626#130626) Certificates trusted in Safari are not part of a “Profile” that can be deleted. If a user accidentally clicks on “Continue” or “Trust” for any of the numerous certificate warnings presented by Safari for DNSBL redirects, the only way to revoke the self-signed certificate is to reset the iOS device.

In one of your responses (https://www.reddit.com/r/PFSENSE/comments/4f56zk/blocking_ads_using_pfblockerng_dnsbl/d26da1c/), you mention that:

“Solutions like installing AD Blocking with a pi-hole device work by basically sending an NXDOMAIN for these DNS queries. This works, however, you don't get a log of what was blocked.”

Would it be possible to offer two options for DNSBL – redirect and serve pixel as it is currently doing or sending NXDOMAIN, at the cost of some logging? Perhaps even offering the ability to select different options for HTTP and HTTP requests.

Eg:
(current behavior)
HTTP: pixel
HTTPS: pixel

(configuration that would help out some iOS users)
HTTP: pixel
HTTPS: NXDOMAIN

(other possible configuration)
HTTP: NXDOMAIN
HTTPS: NXDOMAIN

–--

Edit: I just found your post https://forum.pfsense.org/index.php?topic=124945.msg691099#msg691099 that says "Upcoming version will allow mixing of the DNSBL VIP and 0.0.0.0 so that the domains that are causing these erorrs, can utilize 0.0.0.0 whilst all others use the DNSBL VIP and the logging remains intact... or if you didn't want to log the blocked alerts...."

It seems like you're already working on this feature. :) Thanks!

BBcan177

@wallestack:

BBcan177,

Thank you for all the work you’ve done in developing pfBlockerNG. I stumbled across pfBlockerNG while looking for an adblock solution on pfSense and have been pleased at how well it works.

Thanks!

Edit: I just found your post https://forum.pfsense.org/index.php?topic=124945.msg691099#msg691099 that says "Upcoming version will allow mixing of the DNSBL VIP and 0.0.0.0 so that the domains that are causing these erorrs, can utilize 0.0.0.0 whilst all others use the DNSBL VIP and the logging remains intact… or if you didn't want to log the blocked alerts...."

It seems like you're already working on this feature. :) Thanks!

Yes the next release will have this feature…

shonjir

@BBcan177:

Then you can choose which of these repeated domains to add to a DNSBL Customlist, which will block the domain/sub-domains and remove all the duplicated entries… After adding them, run a "Force reload - DNSBL"

@shonjir:

For example, I was thinking today that maybe the TLD filter list could be used to moderate the clipping process so that it doesn't disproportionately affect country-code sites.

I thought I'd contribute my implementation of this idea. It's a simple perl script that takes the current unbound configuration, analyzes it, and writes an aggregate TLD file that can be used as a pfBlockerNG list. I run this via cron on the half hour and have pfBlockerNG check the list once an hour.

#!/usr/bin/env perl

# Search pfblocker domain list for DGA sites and extract TLD.
# Credit to BBcan177 for the following:
# sed 's/local-data: \"\(.*\) 60.*/\1/' /var/unbound/pfb_dnsbl.conf | rev | cut -d '.' -f 1-2 | rev | awk -v max=50 '{a[$0]++}END{for(i in a){if(a[i] > max){print i}}}'

use strict;
use warnings;

use File::stat qw(stat);
use List::Util qw(min max);
use Data::Dumper qw(Dumper);

# Debugging
my $DEBUG = 0;

# Minimum subdomains to consider
my $SUBDOMAIN_MIN = 2;
# Ignore more than this many subdomains
my $SUBDOMAIN_MAX = 5;
# Ignore domains with fewer than this many hits
my $SENSITIVITY = 10;

# Path to TLD
my $PFB_TLD = "/usr/local/pkg/pfblockerng/dnsbl_tld";

# Path to blacklist
my $PFB_DNSBL = "/var/unbound/pfb_dnsbl.conf";

# Path to output
my $PFB_AGGREGATE = "/var/db/pfblockerng/pfb_aggregate.txt";

# debugging
if ($DEBUG) {
  $PFB_TLD = "dnsbl_tld";
  $PFB_DNSBL = "pfb_dnsbl.conf";
}

sub DEBUG($;@) {
  my ($format) = shift;
  printf "DEBUG: $format\n", @_ if $DEBUG;
}

# Exit unless files exist
if (! -e ${PFB_DNSBL}) {
  print "$PFB_DNSBL: not found\n";
  exit 1;
}
if (! -e ${PFB_TLD}) {
  print "$PFB_TLD: not found\n";
  exit 1;
}

# Exit unless aggregate file is updated
if (-e $PFB_AGGREGATE) {
  my $bl = stat($PFB_DNSBL);
  DEBUG("DNSBL mtime: $bl->mtime");
  my $agg = stat($PFB_AGGREGATE);
  DEBUG("AGGREGATE mtime: $agg->mtime");
  if ($agg->mtime > $bl->mtime) {
    print "Blacklist unchanged.  Nothing to do.\n";
    exit 1;
  }
}

# Read the TLD list in and reverse element and character order to obtain longest match first.
my %tldlist = ();
open(TLD, ${PFB_TLD}) or die;
while (<tld>)
{
  chomp;
  $tldlist{$_} = 0;
};
close(TLD);
#DEBUG Dumper \%tldlist;

# import DNSBL
my $total = 0;
my $variants = 0;
my $ignored = 0;
my %domains = ();
open(DNSBL, ${PFB_DNSBL}) or die;
while (<dnsbl>) {
  chomp;
  next unless /^local-data:/;
  s/^local-data: \"//;
  s/ .*//;
  # Find longest TLD match
  my @domain = split /\./;
  TLD: for my $i (0 .. $#domain) {
    my $key = join(".", @domain[$i .. $#domain]);
    if ( exists($tldlist{$key}) ) {
      # Remove discrete TLD elements and append compound TLD
      if ($i > 0) {
        @domain = @domain[0 .. $i - 1];
        push @domain, $key;
      }
      $tldlist{$key}++;
      last TLD;
    }
  }
  #
  # Add domain to domains hash
  $total++;
  my $start = max(0, $#domain - ($SUBDOMAIN_MAX - 1));
  my $end = max($SUBDOMAIN_MIN - 1, $#domain - ($SUBDOMAIN_MIN - 1));
  if ( $end < $#domain ) {
    for my $i ($start .. $end) {
      my $key = join(".", @domain[$i .. $#domain]);
      $domains{$key}++;
      DEBUG("ADDED: $key : $domains{$key}");
      $variants++;
    }
  } else {
    DEBUG("IGNORED: @domain");
    $ignored++;
  }
}
close(DNSBL);
print "$total domains ingested\n";
print "$variants possible variants (ignored $ignored)\n";

# Filter domains in reverse order
my @keys = sort {
my $aa = join(".", reverse split(/\./, $a));
my $bb = join(".", reverse split(/\./, $b));
$bb cmp $aa
} keys %domains;
my $clipped = @keys;
print "$clipped unique variants\n";
#
# Filter domain list
my $filtered = 0;
my $aggregated = 0;
foreach (@keys) {
  # Subtract aggregate count from higher order domains
  my $count = $domains{$_};
  if ($count > 1) {
    my @domain = split(/\./, $_);
    for my $i (1 .. $#domain) {
      my $key = join(".", @domain[$i .. $#domain]);
      $domains{$key} -= $count if (exists($domains{$key}));
    }
  }

  if ($domains{$_} < $SENSITIVITY) {
    DEBUG ("FILTERED: $_ : $domains{$_}");
    $filtered++;
    delete $domains{$_};
  } else {
    $aggregated++;
    DEBUG ("PRESERVED: $_ : $domains{$_}");
  }
}
print "$filtered domains filtered\n";
print "$aggregated domains aggregated\n";

DEBUG Dumper \%domains;

# Finally output the list
if (not $DEBUG) {
  my $output = 0;
  open(OUT, ">", ${PFB_AGGREGATE}) or die;
  foreach (sort { $domains{$b} <=> $domains{$a} } keys %domains) {
    print OUT "$_\n";
    $output++;
  }
  close(OUT);
  print "$output domains written\n";
}
[/i]</dnsbl></tld>

Fesoj

Hi!

After updating to pfSense 2.3.3 I am experiencing a small problem with pfBlockNG 2.2.2_6, though it might be that my problem is not related to the update. I setup a personal IPv4 list that shows up on the widget as pfB_myIPv4:

The list is on a local server and nothing special is involved:

When I check the "Original IP Files", the correct list is actually on the box:

But when I check the "Deny Files", there's only a single bogus entry that is also reported by the widget:

The list works correctly on other machines (with the older firmware), where I use the same pfBlockerNG version. I also went a couple of times through the update options and checked the original file for funny characters, etc.

Any idea what else I can look at?

doktornotor

@Fesoj:

I setup a personal IPv4 list that shows up on the widget as pfB_myIPv4:
When I check the "Original IP Files", the correct list is actually on the box:
But when I check the "Deny Files", there's only a single bogus entry that is also reported by the widget:

Sounds like you have a personal collection of duplicates covered by other lists. ::)

Fesoj

I "grepped" some patterns inside the /var/db/pfblockerng/original directory, but didn't find any duplicates.

BBcan177

@Fesoj:

I "grepped" some patterns inside the /var/db/pfblockerng/original directory, but didn't find any duplicates.

It could be in a larger CIDR:

grep "1.2.3.4" /var/db/pfblockerng/deny/*
grep "^1\.2\.3\." /var/db/pfblockerng/deny/*
grep "^1\.2\." /var/db/pfblockerng/deny/*
grep "^1\." /var/db/pfblockerng/deny/*

Fesoj

Yes, I know. I checked 16 and 24 bit patterns, but didn't study it thoroughly. I have another machine with pretty much the same lists and my own IP list loads completetely. When I have enough time, I'll check whether there are any differences. It can't be larger CIDRs, because my list contains IPs from normal VM hosters, where some customers think it is funny to attack PBXs. Usually the neigbhoring IPs are unobstrusive.