PfBlockerNG v2.1 w/TLD

shonjir

Then you can choose which of these repeated domains to add to a DNSBL Customlist, which will block the domain/sub-domains and remove all the duplicated entries… After adding them, run a "Force reload - DNSBL"

@shonjir:

For example, I was thinking today that maybe the TLD filter list could be used to moderate the clipping process so that it doesn't disproportionately affect country-code sites.

I thought I'd contribute my implementation of this idea. It's a simple perl script that takes the current unbound configuration, analyzes it, and writes an aggregate TLD file that can be used as a pfBlockerNG list. I run this via cron on the half hour and have pfBlockerNG check the list once an hour.

#!/usr/bin/env perl

# Search pfblocker domain list for DGA sites and extract TLD.
# Credit to BBcan177 for the following:
# sed 's/local-data: \"\(.*\) 60.*/\1/' /var/unbound/pfb_dnsbl.conf | rev | cut -d '.' -f 1-2 | rev | awk -v max=50 '{a[$0]++}END{for(i in a){if(a[i] > max){print i}}}'

use strict;
use warnings;

use File::stat qw(stat);
use List::Util qw(min max);
use Data::Dumper qw(Dumper);

# Debugging
my $DEBUG = 0;

# Minimum subdomains to consider
my $SUBDOMAIN_MIN = 2;
# Ignore more than this many subdomains
my $SUBDOMAIN_MAX = 5;
# Ignore domains with fewer than this many hits
my $SENSITIVITY = 10;

# Path to TLD
my $PFB_TLD = "/usr/local/pkg/pfblockerng/dnsbl_tld";

# Path to blacklist
my $PFB_DNSBL = "/var/unbound/pfb_dnsbl.conf";

# Path to output
my $PFB_AGGREGATE = "/var/db/pfblockerng/pfb_aggregate.txt";

# debugging
if ($DEBUG) {
  $PFB_TLD = "dnsbl_tld";
  $PFB_DNSBL = "pfb_dnsbl.conf";
}

sub DEBUG($;@) {
  my ($format) = shift;
  printf "DEBUG: $format\n", @_ if $DEBUG;
}

# Exit unless files exist
if (! -e ${PFB_DNSBL}) {
  print "$PFB_DNSBL: not found\n";
  exit 1;
}
if (! -e ${PFB_TLD}) {
  print "$PFB_TLD: not found\n";
  exit 1;
}

# Exit unless aggregate file is updated
if (-e $PFB_AGGREGATE) {
  my $bl = stat($PFB_DNSBL);
  DEBUG("DNSBL mtime: $bl->mtime");
  my $agg = stat($PFB_AGGREGATE);
  DEBUG("AGGREGATE mtime: $agg->mtime");
  if ($agg->mtime > $bl->mtime) {
    print "Blacklist unchanged.  Nothing to do.\n";
    exit 1;
  }
}

# Read the TLD list in and reverse element and character order to obtain longest match first.
my %tldlist = ();
open(TLD, ${PFB_TLD}) or die;
while (<tld>)
{
  chomp;
  $tldlist{$_} = 0;
};
close(TLD);
#DEBUG Dumper \%tldlist;

# import DNSBL
my $total = 0;
my $variants = 0;
my $ignored = 0;
my %domains = ();
open(DNSBL, ${PFB_DNSBL}) or die;
while (<dnsbl>) {
  chomp;
  next unless /^local-data:/;
  s/^local-data: \"//;
  s/ .*//;
  # Find longest TLD match
  my @domain = split /\./;
  TLD: for my $i (0 .. $#domain) {
    my $key = join(".", @domain[$i .. $#domain]);
    if ( exists($tldlist{$key}) ) {
      # Remove discrete TLD elements and append compound TLD
      if ($i > 0) {
        @domain = @domain[0 .. $i - 1];
        push @domain, $key;
      }
      $tldlist{$key}++;
      last TLD;
    }
  }
  #
  # Add domain to domains hash
  $total++;
  my $start = max(0, $#domain - ($SUBDOMAIN_MAX - 1));
  my $end = max($SUBDOMAIN_MIN - 1, $#domain - ($SUBDOMAIN_MIN - 1));
  if ( $end < $#domain ) {
    for my $i ($start .. $end) {
      my $key = join(".", @domain[$i .. $#domain]);
      $domains{$key}++;
      DEBUG("ADDED: $key : $domains{$key}");
      $variants++;
    }
  } else {
    DEBUG("IGNORED: @domain");
    $ignored++;
  }
}
close(DNSBL);
print "$total domains ingested\n";
print "$variants possible variants (ignored $ignored)\n";

# Filter domains in reverse order
my @keys = sort {
my $aa = join(".", reverse split(/\./, $a));
my $bb = join(".", reverse split(/\./, $b));
$bb cmp $aa
} keys %domains;
my $clipped = @keys;
print "$clipped unique variants\n";
#
# Filter domain list
my $filtered = 0;
my $aggregated = 0;
foreach (@keys) {
  # Subtract aggregate count from higher order domains
  my $count = $domains{$_};
  if ($count > 1) {
    my @domain = split(/\./, $_);
    for my $i (1 .. $#domain) {
      my $key = join(".", @domain[$i .. $#domain]);
      $domains{$key} -= $count if (exists($domains{$key}));
    }
  }

  if ($domains{$_} < $SENSITIVITY) {
    DEBUG ("FILTERED: $_ : $domains{$_}");
    $filtered++;
    delete $domains{$_};
  } else {
    $aggregated++;
    DEBUG ("PRESERVED: $_ : $domains{$_}");
  }
}
print "$filtered domains filtered\n";
print "$aggregated domains aggregated\n";

DEBUG Dumper \%domains;

# Finally output the list
if (not $DEBUG) {
  my $output = 0;
  open(OUT, ">", ${PFB_AGGREGATE}) or die;
  foreach (sort { $domains{$b} <=> $domains{$a} } keys %domains) {
    print OUT "$_\n";
    $output++;
  }
  close(OUT);
  print "$output domains written\n";
}
[/i]</dnsbl></tld>

Fesoj

Hi!

After updating to pfSense 2.3.3 I am experiencing a small problem with pfBlockNG 2.2.2_6, though it might be that my problem is not related to the update. I setup a personal IPv4 list that shows up on the widget as pfB_myIPv4:

The list is on a local server and nothing special is involved:

When I check the "Original IP Files", the correct list is actually on the box:

But when I check the "Deny Files", there's only a single bogus entry that is also reported by the widget:

The list works correctly on other machines (with the older firmware), where I use the same pfBlockerNG version. I also went a couple of times through the update options and checked the original file for funny characters, etc.

Any idea what else I can look at?

doktornotor

@Fesoj:

I setup a personal IPv4 list that shows up on the widget as pfB_myIPv4:
When I check the "Original IP Files", the correct list is actually on the box:
But when I check the "Deny Files", there's only a single bogus entry that is also reported by the widget:

Sounds like you have a personal collection of duplicates covered by other lists. ::)

Fesoj

I "grepped" some patterns inside the /var/db/pfblockerng/original directory, but didn't find any duplicates.

BBcan177

@Fesoj:

I "grepped" some patterns inside the /var/db/pfblockerng/original directory, but didn't find any duplicates.

It could be in a larger CIDR:

grep "1.2.3.4" /var/db/pfblockerng/deny/*
grep "^1\.2\.3\." /var/db/pfblockerng/deny/*
grep "^1\.2\." /var/db/pfblockerng/deny/*
grep "^1\." /var/db/pfblockerng/deny/*

Fesoj

Yes, I know. I checked 16 and 24 bit patterns, but didn't study it thoroughly. I have another machine with pretty much the same lists and my own IP list loads completetely. When I have enough time, I'll check whether there are any differences. It can't be larger CIDRs, because my list contains IPs from normal VM hosters, where some customers think it is funny to attack PBXs. Usually the neigbhoring IPs are unobstrusive.

BBcan177

If you have "Aggregation" enabled, that can also optimize the lists along with deduplication.

Fesoj

Yes, that was it. By accident all countries were blocked, after deselecting a certain country, everything is fine again.

Thanks.

Fesoj

This is just a question. Would it make any sense to add fail2ban functionality to pfBlockerNG?

This could be ssh-logins or other stuff. I like to add Asterisk to my pfSense boxes, so unsolicited registration attempts are a pain in the neck, though I get rid most of them by blocking most countries. Compared to the more static GeoIP lists, here one would release the IPs typically after a couple of hours, but block them immediately once a certain threshold gets exceeded.

An alternative would be to call pftcl from within Asterisk as needed.

dcol

In regards to TLD blocking, I have it setup and although it appears in the alerts, the packets are still delivered. Do I need to create a rule to actually block a TLD from arriving at its destination, which in my case is an email server.

virusbcn

Hello friends, i love this package but i have one question or one suggestion to add

I see in the queues of my mail server, mails in queue because can not connect to ip x.x.x.x , i not view this ip in country list ipv4 denegated, but maybe is in my personal ipv4 lists (dshield, etc..) and the question is:

How can i check if one ip is denegated in pfblockerng ??? there's no search ip engine ??? i want know what list ip is denegated this ip

Hugovsky

I'm having problems with iOS 11.3 BETA. It seems that safari can't go to 0.0.0.0 and pages can't load. Solution was to reinstall package and use the VIP address, as I was using the certificate hack and 0.0.0.0.

jeaman16

Hi
Thanks for this wonderful package and it helps me a lot.
Do you have an updated list of torrent sites or movie downloading sites like yts? I can't block the whole YTS sites after enabling TLD and adding this to Blacklist. Anyone please.. Thank you

ex1580

It would be really cool if it could automatically update the blocked TLDs based on the spamhaus statistics (https://www.spamhaus.org/statistics/tlds/) on a regular schedule. I realize that this may be more difficult than it sounds as I cant seem to find a spamhaus TLD feed, just a website. But if we dont dream then it will never happen!