pfBlockerNG-devel v2.2.5_20 PR #610


  • Moderator

    Link: https://www.patreon.com/posts/23743067

    MaxMind has deprecated GeoIP in favor of their new MMDB database format. This pull request is for the pfBlockerNG-devel version.

    Changelog:
    Update Maxmind library from GeoIP -> libmaxminddb Port
    Relocate EasyList to the Feeds tab to be added like all other Feeds.
    Modify EasyList parser
    Allow other Easylist/ADBlock/uBlock/ADGuard feed syntax to the DNSBL parser
    Add more EasyList Language specific feeds
    Add "Server.max-request-size = 1" to Lighttpd config
    Alerts Tab/Logs: Collect NAT IP addresses by Target:Port
    Improve SQLite3 DB validations
    Fix issue causing DNSBL/Unbound Counters to report over 100%
    The next version of pfBlockerNG-devel is planned for February 2019 which will include DNSBL IPv6 Blocking, and the new Python Unbound Integration that will be sure to blow your socks off ! !



  • @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    The next version of pfBlockerNG-devel is planned for February 2019 which will include DNSBL IPv6 Blocking, and the new Python Unbound Integration that will be sure to blow your socks off ! !

    How is pfBlockerNG going to activate the Python support in Unbound?

    I have already added Python support on my install by using a System Patch (https://github.com/twitched/pfsense/commit/1ff1605e8d2e2c9f87aac489fd7af7a407b3440c.patch) and an early shell command to nullfs mount the python libraries into the unbound chroot (/sbin/mount -t nullfs /usr/local/lib/python2.7 /var/unbound/usr/local/lib/python2.7).

    Are you going to do it in a similar way? I just want to make sure there isn't going to be conflict when this gets released.


  • Moderator

    @grimson

    I pushed a PR to add Python options to the pfSense Resolver GUI.

    https://github.com/pfsense/pfsense/pull/4029

    But only one script can run at a time.



  • @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    @grimson

    I pushed a PR to add Python options to the pfSense Resolver GUI.

    https://github.com/pfsense/pfsense/pull/4029

    So for most users this will also require a pfSense update before it can be used, as this isn't in 2.4.4p2 as of yet.

    But only one script can run at a time.

    No problem, I'll just merge my script with yours using the System Patches package.


  • Moderator

    @grimson said in pfBlockerNG-devel v2.2.5_20 PR #610:

    @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    @grimson
    I pushed a PR to add Python options to the pfSense Resolver GUI.
    https://github.com/pfsense/pfsense/pull/4029

    So for most users this will also require a pfSense update before it can be used, as this isn't in 2.4.4p2 as of yet.

    But only one script can run at a time.

    No problem, I'll just merge my script with yours using the System Patches package.

    Yes it will require an update or 2.4.5 I believe.
    Depending on what your script does, it might have to be integrated differently in the pfB python script.



  • @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    Depending on what your script does, it might have to be integrated differently in the pfB python script.

    It's mostly an extension of this one: https://gist.github.com/FiloSottile/e2cffde2bae1ea0c14eada229543aebd/ to prevent IPv6 resolution for services that don't like the He.net tunnel prefixes.

    If you want to add that functionality into pfBlockerNG I wouldn't mind. But I'm not sure how much sense this would make, as I doubt there are that many user interested in it. Also I have no problem doing it on my own.



  • Another question, is it intentional that version 2.2.5_20 no longer uses a lot of the domains listed under

    ---------------------------Third-party advertisers---------------------------!
    ! *** easylist:easylist/easylist_adservers.txt ***

    in the Easylist feeds. Basicly all domains ending with "$third-party" are no longer filtered.

    Edit:
    Correction, all domain entries containing a "$" are no longer filtered, that also includes things like "$popup,third-party".

    Edit2:
    As an interim solution I created a little patch that removes some of the filter options (https://adblockplus.org/filter-cheatsheet#filter-options) before the lists are parsed:

    --- pfblockerng.inc	2019-01-12 22:07:42.021169200 +0100
    +++ pfblockerng.inc	2019-01-12 21:59:19.000000000 +0100
    @@ -5655,6 +5655,7 @@
     
     								// Variables for Easylists
     								$easylist = $validate_header = FALSE;
    +								$e_pre_replace = array( '$popup,third-party', '$popup', '$script,third-party', '$script', '$image,third-party', '$image', '$third-party' );
     								$e_replace = array( '||', '.^', '^' );
     
     								$run_once = $csv_parser = FALSE;
    @@ -5694,6 +5695,9 @@
     											$line = trim($line, " \t\n\r\0\x0B\xC2\xA0");
     
     											if ($easylist) {
    +												//Remove Easylist filter options
    +												$line = str_replace($e_pre_replace, '', $line);
    +
     												if (substr($line, 0, 2) !== '||' ||
     												    substr($line, -1) !== '^' ||
     												    strpos($line, '$') !== FALSE ||
    
    

  • Moderator

    @grimson said in pfBlockerNG-devel v2.2.5_20 PR #610:

    How is pfBlockerNG going to activate the Python support in Unbound?
    I have already added Python support on my install by using a System Patch (https://github.com/twitched/pfsense/commit/1ff1605e8d2e2c9f87aac489fd7af7a407b3440c.patch) and an early shell command to nullfs mount the python libraries into the unbound chroot (/sbin/mount -t nullfs /usr/local/lib/python2.7 /var/unbound/usr/local/lib/python2.7).
    Are you going to do it in a similar way? I just want to make sure there isn't going to be conflict when this gets released.

    This might be of interest for you instead of the python script:

    https://nlnetlabs.nl/documentation/unbound/unbound.conf/

    DNS64 Module Options
           The  dns64  module must be configured in the module-config: "dns64 val-
           idator iterator" directive and  be  compiled  into  the  daemon  to  be
           enabled.  These settings go in the server: section.
    
           dns64-prefix: <IPv6 prefix>
                  This  sets  the  DNS64  prefix to use to synthesize AAAA records
                  with.  It must  be  /96  or  shorter.   The  default  prefix  is
                  64:ff9b::/96.
    
           dns64-synthall: <yes or no>
                  Debug  option,  default  no.   If  enabled,  synthesize all AAAA
                  records despite the presence of actual AAAA records.
    
           dns64-ignore-aaaa: <name>
                  List domain for which the AAAA records are  ignored  and  the  A
                  record is used by dns64 processing instead.  Can be entered mul-
                  tiple times, list a new domain for which  it  applies,  one  per
                  line.  Applies also to names underneath the name given.
    

  • Moderator

    @grimson said in pfBlockerNG-devel v2.2.5_20 PR #610:

    Another question, is it intentional that version 2.2.5_20 no longer uses a lot of the domains listed under

    ---------------------------Third-party advertisers---------------------------!
    ! *** easylist:easylist/easylist_adservers.txt ***

    in the Easylist feeds. Basicly all domains ending with "$third-party" are no longer filtered.

    The previous DNSBL parser was a bit aggressive in the domains that it would parse from the EasyList feeds. I intentionally reduced the parser to lines that start with "||" and end with "^"... The other variations can lead to FPs... other DNSBL Feeds will most likely add any missing domains.

    If I am missing something, please let me know...



  • This post is deleted!


  • @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    This might be of interest for you instead of the python script:

    https://nlnetlabs.nl/documentation/unbound/unbound.conf/

    DNS64 Module Options
    

    Not really. This could probably be used to generate false AAAA records, but that is not the intention. The intention is to return no AAAA records at all, so that the (for example Netflix) client isn't even trying to use IPv6 at all. False records would be the same as blocking IPv6, it will cause disruptions and possibly error messages.

    But you don't have to worry about that, I have no issues with adding that functionality by myself.

    @bbcan177 said in pfBlockerNG-devel v2.2.5_20 PR #610:

    The previous DNSBL parser was a bit aggressive in the domains that it would parse from the EasyList feeds. I intentionally reduced the parser to lines that start with "||" and end with "^"... The other variations can lead to FPs... other DNSBL Feeds will most likely add any missing domains.

    If I am missing something, please let me know...

    Well the EasyList feeds are IMHO among the best, but with the current parser they are mostly useless as they classify most ad networks/sources as third-party given that you rarely visit these domains directly but only get the ads from them. Prime example is googleadservices*com, with the current parser (TLD is still enabled) it is no longer blocked using EasyList.

    Adding other (probably less maintained) lists to fill those holes isn't a solution I'm too happy with, but again that is IMHO and I have no problem patching the parser for my needs.