PfBlockerNG v2.1 w/TLD



  • Thank you so much for this fantastic work!!!


  • Moderator

    @DownloadDeviant:

    THANKS! Can't wait! Good stuff….great work...and thanks for helping us dumb dumbs  :P here and over at Reddit!

    PS - is there a quick n dirty way to test PFBNG to be sure you've generally set it up correctly? Like going to a website and not seeing ads, etc.?

    Thanks… Are you on the latest 2.1.1_1 version?  Haven't heard much feedback yet, so not sure if many have installed it yet...

    Not sure what sites are the worst for ADs... but yahoo is probably up there....

    @mauroman33:

    Thank you so much for this fantastic work!!!

    Thanks!



  • @BBcan177:

    Thanks… Are you on the latest 2.1.1_1 version?  Haven't heard much feedback yet, so not sure if many have installed it yet...

    Not sure what sites are the worst for ADs... but yahoo is probably up there....

    I'm still on 2.0.17. I've slowed down my updating a bit since I've had some snags and had to rebuild 3 times in the past 7 weeks. Two were my fault…lol I thought I had router plugged into the battery port on the UPS but didn't...storm hit...lost power...pf went corrupt. Sooooooooo, I'm a bit worn out on tampering right now. lol That said, I'll probably upgrade it this weekend.

    Yahoo it is then. I'm very new to pfBNG so I need to learn it and get comfortable. I don't want to get  too aggressive. I just want it to serve as a companion for my Firefox plugins and to help keep my girlfriend protected.



  • pfBlockerNG-2.1.1_1 is working like charm, On 2.3.3-DEVELOPMENT (amd64) no issues.

    TLD Blacklist is really handy, Thanks BBcan177


  • Moderator

    Here are the links for Malware Corpus Tracker which can be used w/ pfBlockerNG DNSBL:

    Site:
    http://track.h3x.eu/about/400

    Available Feeds:
    https://tracker.h3x.eu/api/sites_1month.php
    https://tracker.h3x.eu/api/sites_1week.php
    https://tracker.h3x.eu/api/sites_1day.php
    https://tracker.h3x.eu/api/sites_1hour.php

    DO NOT Select all of these Feeds. You should pick only one Feed. For example: the "1Month" will include the "1Week/1Day/1Hour".

    [ Edit - change to https ]

    Twitter:
    https://twitter.com/h3x2b



  • Hi BBcan177,

    I can't update h3x feed from available feeds list in pfBlockerNG v2.1.

    It show below.

    [ h3x ]			 Downloading update .. 200 OK
     Remote timestamp missing 
     No Domains Found
    

    And I can't let TLD Exclusion List working. Can you give a example or check it works?



  • @ntct:

    Hi BBcan177,

    I can't update h3x feed from available feeds list in pfBlockerNG v2.1.

    It show below.

    [ h3x ]			 Downloading update .. 200 OK
     Remote timestamp missing 
     No Domains Found
    

    Same here

    @ntct:

    And I can't let TLD Exclusion List working. Can you give a example or check it works?

    Did you do a Force Reload after changing the list ?



  • [ 1month ]		 Downloading update .. 200 OK
      Remote timestamp missing 
     No Domains Found
    
    [ 1week ]		 Downloading update [ 07/30/16 12:31:20 ] .. 200 OK
      Remote timestamp missing 
     No Domains Found
    
    [ 1day ]		 Downloading update .. 200 OK
      Remote timestamp missing 
     No Domains Found
    
    [ 1hour ]		 Downloading update .. 200 OK
      Remote timestamp missing 
     No Domains Found
    

    Me three, anyone post how exactly you get these list working?


  • Moderator

    Here is a patch to fix the H3X Feed…  Sorry about that  ...

    @BBcan177:

    Here are the links for Malware Corpus Tracker which can be used w/ pfBlockerNG DNSBL:

    UPDATE:

    Guess the internal QA testing didn't work too well when I tested this Feed.
    Please follow these instructions below to patch the code to get the following feed to parse:

    Edit     /usr/local/pkg/pfblockerng/pfblockerng.inc

    Goto Line 3368 which contains the following:

    $h3x_feed = TRUE;
    

    Reference:
    https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L3368

    and add the following line after line 3368:

    $liteparser = TRUE;
    

    Then follow that with a    "Force Update"


  • Moderator

    @ntct:

    And I can't let TLD Exclusion List working. Can you give a example or check it works?

    Can you provide more detail about what you're trying to accomplish?



  • Works for me.

    For the lazy:

    vi +3368 /usr/local/pkg/pfblockerng/pfblockerng.inc
    

    to go straight to the line.

    Thanks BBcan


  • Moderator

    I have posted a PR #164 to fix the H3x parser issue noted above.
    ‎https://github.com/pfsense/FreeBSD-ports/pull/164‎

    Once this is merged the pkg will be at version 2.1.1_2.

    If you manually edited the file noted above, or not, you do not need to make any further changes with this version.



  • …...and I'm buying another 8 gigs RAM tonight (from 8G to 16G) now that unbound is VIRT 12.3G and I've swapped 6G.

    Worth it though.



  • Hi BBcan177

    Thanks for implementing this.  Would you be able to explain a bit more what the role of the /usr/local/pkg/pfblockerng/dnsbl_tld file is please?

    I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher.  But it seems itself to contain some second level domains?

    That said, when I've looked that the /var/unbound/pfb_dnsbl.conf on my set up that pfblockerng has created, it does contain exactly what I would expect to see (i.e. full blocking of the entire domain for second level domains, but only specific blocking for higher level domains).  So it does seem to be doing exactly what I'd like it to, but I'm not sure how the dnsbl_tld file is working to do that.

    Thanks.



  • Hi BBcan177,

    Is there any good install/setup/configure instruction (video or guide) for the last version op pfblockerNG, that you could/would recommend?

    Thanks for your advice, cheers Qinn


  • Moderator

    @Andrew453:

    I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher.  But it seems itself to contain some second level domains?

    Hi Andrew453,

    If I only used the TLD, it would be a simple process of looking at any listed Domain and seeing if it had only a second-level Domain (SLD) then block the entire Domain. However, there are suffixes like "uk.com" which is what I would call the TLD that is used to determine if there is one more level. So all of the TLDs (suffixes) in that file are known TLDs which is used in the determination process. Most of the file was taken from the "Public Suffix Registry".


  • Moderator

    @Qinn:

    Hi BBcan177,

    Is there any good install/setup/configure instruction (video or guide) for the last version op pfblockerNG, that you could/would recommend?

    Thanks for your advice, cheers Qinn

    There is a pfSense Hangout that I did which can be used for an overview of the pkg functionality. However, apart from the three main pfBlockerNG threads in this forum, there isn't any other documentation.



  • Thanks for the quick reply. Darn  :( I found this one can you can agree to this one?

    Youtube Video



  • @BBcan177:

    @Andrew453:

    I was expecting it to contain a pure list of TLDs which pfblockerng can then use to work out whether any given domain is a second level domain or higher.  But it seems itself to contain some second level domains?

    Hi Andrew453,

    If I only used the TLD, it would be a simple process of looking at any listed Domain and seeing if it had only a second-level Domain (SLD) then block the entire Domain. However, there are suffixes like "uk.com" which is what I would call the TLD that is used to determine if there is one more level. So all of the TLDs (suffixes) in that file are known TLDs which is used in the determination process. Most of the file was taken from the "Public Suffix Registry".

    Yes ok.  That's exactly what I thought the file was for.  (i.e. some eTLDs are longer than others, so you need a list e.g. .com vs .co.uk to work out what to treat as an eTLD)

    The thing that was confusing me was there were some domains in the list that looked a bit odd, e.g.

    myactivedirectory.com
    mydrobo.com
    mysecuritycamera.com
    myshopblocks.com
    myvnc.com

    I think all you're saying is that pfblockerng will treat those as eTLDs even though, strictly speaking, they aren't … which is fine.

    p.s. a big thank you for implementing this.  It was on my wish-list as I recall - https://forum.pfsense.org/index.php?topic=106534



  • @RonpfS:

    @ntct:

    Hi BBcan177,

    I can't update h3x feed from available feeds list in pfBlockerNG v2.1.

    It show below.

    [ h3x ]			 Downloading update .. 200 OK
     Remote timestamp missing 
     No Domains Found
    

    Same here

    @ntct:

    And I can't let TLD Exclusion List working. Can you give a example or check it works?

    Did you do a Force Reload after changing the list ?

    I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.

    I tried with Update, Cron , Reload.

    [ malw_corpus ] Downloading update .. 200 OK
      Remote timestamp missing
    No Domains Found



  • Hi,
    Thank you for your hard work on this package :)

    After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
    The DNSBL_EasyList won't delete the packets

    ![Desktop 31-07-2016 17.00.06-358.png_thumb](/public/imported_attachments/1/Desktop 31-07-2016 17.00.06-358.png_thumb)
    ![Desktop 31-07-2016 17.00.06-358.png](/public/imported_attachments/1/Desktop 31-07-2016 17.00.06-358.png)
    [Video 31-07-2016 16.54.57.zip](/public/imported_attachments/1/Video 31-07-2016 16.54.57.zip)



  • @Redyr:

    I'm on 2.1.1_2, the h3x fix is included, but I get the same error as above.

    I tried with Update, Cron , Reload.

    [ malw_corpus ] Downloading update .. 200 OK
      Remote timestamp missing
    No Domains Found

    Each URL contains sites that were active in the last period (month, week, day or hour).

    If you look at the 1hour or the 1day csv file, they only have one comment. The 1week and 1month have entries.

    You should only choose one of the feeds according to your need. I guess most will pick the 1month URL.



  • Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package.  It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information".  This happens for the "Installed Packages as well as "Available Packages" tabs!

    On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".

    Tried disabling both pfblockerNG and DNSBL to no avail.  Snort is disabled and the blocked hosts list is empty.

    Now I cannot update, install or uninstall packages….  How do I remedy to this?



  • I've had that trouble before too.  It happened when I was trying to change from the development thread for updates to the stable thread.  I couldn't update anything.  I eventually found some instructions to reinstall the main pfsense components from the command line.  I ended up still on the development thread and didn't venture to try to change it back after that.



  • @Andrew453:

    I've had that trouble before too.  It happened when I was trying to change from the development thread for updates to the stable thread.  I couldn't update anything.  I eventually found some instructions to reinstall the main pfsense components from the command line.  I ended up still on the development thread and didn't venture to try to change it back after that.

    Not sure I understand that.  I am not playing with development stuff, nor that I am configured to retrieve packages from development repos..  Just a vanilla pfsense install with pfblockerNG, snort and thats it.  Not normal all of a sudden I lose connection to repos..

    Also after a reboot I see these warnings in the main page:

    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00
    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11
    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20 
    

  • Moderator

    @lpallard:

    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00
    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11
    There were error(s) loading the rules: /tap/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20 
    

    These all seem to be related to the MaxMind IPv6 database. Looks like you will need to bump the pfSense max aliastable entries limit from 2M to 4M. If you enable aggregation in the general tab, it should condense the CIDRs and reduce the overall IP count. This changed due to using the new MaxMind Geolite2 database which seems to have smaller subsets of the data listed causing more IP entries to be added.


  • Moderator

    @CiscoX:

    After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
    The DNSBL_EasyList won't delete the packets

    I am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.



  • @lpallard:

    Not sure if this is related to pfblockerNG (2.1 w/ TLD) but I went to the package manager to install a package, and saw that my copy of pfblockerNG was outdated, so I clicked the yellow round arrow to update the package.  It went well, but immediately after I returned to the package manager I was greeted with a red ribbon saying "Unable to retrieve package information".  This happens for the "Installed Packages as well as "Available Packages" tabs!

    On the main page, I see "Obtaining update status ", then it turns to "Unable to check for updates".

    Tried disabling both pfblockerNG and DNSBL to no avail.  Snort is disabled and the blocked hosts list is empty.

    Now I cannot update, install or uninstall packages….  How do I remedy to this?

    From the following thread:

    https://forum.pfsense.org/index.php?topic=116019.0

    I followed the ssh command line execution steps:

    pkg update -f
    pkg upgrade -f

    and the same problem was resolved.



  • @BBcan177:

    @CiscoX:

    After updating to 2.1.1_2 i can't "clear DNSBL Packets" from the pfBlockerNG widge
    The DNSBL_EasyList won't delete the packets

    I am away for a few weeks but will check that out. Seems like some regression somewhere. Thanks for reporting.

    Hi, No problem. Have a nice Holiday :)



  • Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.



  • @Qinn:

    Hi there I followed this guide, http://fredmerc.com/2016/07/15/pfsense-adblock-using-pfblockerng-guide/ a rather short setup, there is only DNSBL and no IP4 is that new or is this guide missing it? Thanks for any help.

    Here are the original pfBlockerNG thread https://forum.pfsense.org/index.php?topic=86212.0
    and the pfBlockerNG v2.0 w/DNSBL thread https://forum.pfsense.org/index.php?topic=102470



  • I am getting this error when I try to use the Spamhaus list in this tread.

    ===[  DNSBL Process  ]================================================

    [ EasywoElements ] exists.
    [ SpamHouse_TLDS ] Downloading update .. 200 OK
      Remote timestamp missing .
      –--------------------------------------------------------------------
      Orig.    Unique    # Dups    # White    # Alexa    Final               
      ----------------------------------------------------------------------
      3        3          0          0          0          3                   
      ----------------------------------------------------------------------

    [ DNSBL FAIL ] [ Skipping : SpamHouse_TLDS ]

    [1470071701] unbound-checkconf[87654:0] error: error parsing local-data at 38 '(xmlhttp.readystate 60 IN A 10.10.10.1': Syntax error, could not parse the RR
    [1470071701] unbound-checkconf[87654:0] error: Bad local-data RR (xmlhttp.readystate 60 IN A 10.10.10.1
    [1470071701] unbound-checkconf[87654:0] fatal error: failed local-zone, local-data configuration
    [ Malware_1month ] Downloading update [ 08/01/16 12:15:01 ] .. 200 OK
      Remote timestamp missing .
      –--------------------------------------------------------------------
      Orig.    Unique    # Dups    # White    # Alexa    Final               
      ----------------------------------------------------------------------
      1221    956        0          0          0          956                 
      ----------------------------------------------------------------------

    [ Malware_1week ] Downloading update [ 08/01/16 12:15:04 ] .. 200 OK
      Remote timestamp missing .
      –--------------------------------------------------------------------
      Orig.    Unique    # Dups    # White    # Alexa    Final               
      ----------------------------------------------------------------------
      526      487        487        0          0          0                   
      ----------------------------------------------------------------------

    [ Malware_1day ] Downloading update [ 08/01/16 12:15:05 ] .. 200 OK
      Remote timestamp missing .
      –--------------------------------------------------------------------
      Orig.    Unique    # Dups    # White    # Alexa    Final               
      ----------------------------------------------------------------------
      48      47        47        0          0          0                   
      ----------------------------------------------------------------------

    [ Malware_1hour ] Downloading update .. 200 OK
      Remote timestamp missing
    No Domains Found

    –----------------------------------------
    Assembling database... completed
    Executing TLD
    TLD analysis. completed
    Finalizing TLD...  completed

    Original    Matches    Removed    Final

    6062        5530      1          6061

    Validating database... completed [ 08/01/16 12:15:08 ]
    Reloading Unbound…. completed
    DNSBL update [ 6061 | PASSED  ]… completed



  • Which Spamhaus URL are you using ?
    this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.

    as for the H3X, only one is needed
    https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896

    And do a Force Reload after making the modifications.



  • @RonpfS:

    Which Spamhouse URL are you using ?
    this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.

    as for the H3X, only one is needed
    https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896

    And do a Force Reload after making the modifications.

    Thank you.
    i see my mistake now.
    I was certain I had 2 feeds that contained data but I must have misplaced it?



  • Read the first posts (or more  ;)) of each of these threads:
    pfBlockerNG
    pfBlockerNG v2.0 w/DNSBL
    pfBlockerNG v2.1 w/TLD

    You will find some posts about IP and DNSBL Feed.



  • First of all thank you very much for your hard work and this awesome package!

    I was just wondering is it possible to somehow change the Rule Order setting to something like:
    pfB_Pass/Match | pfB_Block/Reject | All other Rules | (original format)
    so the first IP-list would be the whitelist?

    Right now I can't seem to figure out how to make custom LAN IPv4 whitelist (Permit_Outbound) rule to be the first in the rule list of the LAN interface. If I manually move it first. Next list update puts it bellow the blocklists (Deny_Outbound) again. Right now only the default setting | pfB_Block/Reject | All other Rules | (Original format) is partly usable for me (whitelist won't work) and all other rule order settings just mess my original LAN rules.

    I use Traffic Shaper queues in the floating rules so prefer not to move pfBlockerNG's rules in there too.

    Is this somehow possible or what am I missing, thanks?



  • Which version are you using ?

    with pfBlockerNG 2.1.1_2 I have these choices.

    And you can still use the Floating Rules, it won't affect the Traffic Shaper rules.




  • @Heimire:

    @RonpfS:

    Which Spamhouse URL are you using ?
    this https://www.spamhaus.org/statistics/tlds/ is just a web page, not a feed DNSBL can use.

    as for the H3X, only one is needed
    https://forum.pfsense.org/index.php?topic=115357.msg643896#msg643896

    And do a Force Reload after making the modifications.

    Thank you.
    i see my mistake now.
    I was certain I had 2 feeds that contained data but I must have misplaced it?

    The https://www.spamhaus.org/statistics/tlds/ page can be useful to find TLD to put in the TLD Blacklist.



  • There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:00
    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:11
    There were error(s) loading the rules: /tmp/rules.debug:37: cannot define table pfB_Europe_v6: Cannot allocate memory - The line in question reads [37]: table <pfB_Europe_v6> persist file "/var/db/aliastables/pfB_Europe_v6.txt" @ 2016-07-31 14:55:20 
    

    Check what is selected in this tab, as i had a similar problem and found since the update that the inverse of what i had previously selected had been selected causing over 1.5M IP's for this section and using up all the available memory.

    Rob




Log in to reply