PfBlockerNG v2.0 w/DNSBL



  • @BBcan177:

    There is no option in The Unbound Resolver to "Filter by host"…. So if you want to bypass DNSBL for some hosts... You need to define a different DNS server for those LAN devices....

    thanks for the info, is there any such feature request at unbound side or probably any plans to add it in the future?


  • Banned

    I just ran into an issue and wanted to put it out here.

    I run Private Internet Access as a VPN for one of my interfaces, I also use their DNS's.

    I guess it's not really too surprising that this happened but my internet went down due to DNS and it was because the PIA DNS were being blocked by one of my pfBlockerNG lists, specifically "http://www.stopforumspam.com/downloads/bannedips.zip".

    Anyways I know I'm not the only person who uses PIA DNS so I thought I'd put it out on here.


  • Moderator

    @pfBasic:

    I just ran into an issue and wanted to put it out here.

    I run Private Internet Access as a VPN for one of my interfaces, I also use their DNS's.

    I guess it's not really too surprising that this happened but my internet went down due to DNS and it was because the PIA DNS were being blocked by one of my pfBlockerNG lists, specifically "http://www.stopforumspam.com/downloads/bannedips.zip".

    Anyways I know I'm not the only person who uses PIA DNS so I thought I'd put it out on here.

    Best to submit this to Stop Forum Spam.



  • Is anyone else seeing this lately after a cron?

    There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]:

    Has only started happening in the last few weeks.

    Some googling suggested setting NIC speed manually from auto.

    If anyone has got any other ideas please post.

    Thanks

    Rob


  • Moderator

    Looks like a bad aliastable possibly. Try a Force reload and/or a reboot.

    Check the pfblockerng.log and system logs for further clues. If you hover over the Firewall rules, do you get the aliastable popup? Could also be unrelated to the package. You could temporarily disable the package and see if it re occurs.



  • Ok thanks for that does this look ok as it is the only alias that didn't look right to me?

    It only seemed to appear after i ticked de-duplication is it.

    Rob

    ![Screen Shot 2016-11-07 at 13.19.46.jpg](/public/imported_attachments/1/Screen Shot 2016-11-07 at 13.19.46.jpg)
    ![Screen Shot 2016-11-07 at 13.19.46.jpg_thumb](/public/imported_attachments/1/Screen Shot 2016-11-07 at 13.19.46.jpg_thumb)


  • Moderator

    That is used for the IP Suppression… Its ok as is...



  • Hi guys

    i am busy running a test server where i plan to simulate our current Squid+squidguard setup, but add DNSBL to it.

    one think i note is that some ad's are still getting through ( comparing with AdBlock on chrome)

    Is it possible to load an AdBlock list ( and from where?) , so that user experience is similar to having AdBlock extension activate?

    Thx


  • Banned

    It looks like I got one thing working on my setup but killed DNSBL in the process.

    I recently setup policy based routing where my pass firewall rules determine the Gateway to be used. I did this so that I could have one interface route all traffic through my VPN client and another interface not utilize the VPN at all. That works great, but ever since I set that up I have 0 hits on all of my DNSBL feeds and ads are coming through.

    Any ideas on how I can use pfBlockerNG DNSBL while still utilizing policy based routing for my purposes?



  • My DNSBL has stopped working. When I do a Fprce Update, I get this in the log

    ------------------------------------------
    Assembling database... completed
    Validating database... completed
    Reloading Unbound... Failed to Reload... Restoring previous database.... Not completed.
    
    *** DNSBL update [ 0 ] [ 8654 ] ... OUT OF SYNC ! ***
    ------------------------------------------
    

    When I execute "Shell Output - unbound-control -c /var/unbound/unbound.conf status"
    I get this:

    error: SSL handshake failed
    34386131464:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:/builder/pfsense-232/tmp/FreeBSD-src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1191:
    

    I've just uninstalled PfBlockerNG and after that a reboot, but it still persists. Any ideas?



  • the cron/reload/update job stalls for me. no error, just stops at dnsbl. works if i disable alexa whitelist or if i stop the dnsbl component. anyone else having this problem?


  • Moderator

    @laviniuc:

    the cron/reload/update job stalls for me. no error, just stops at dnsbl. works if i disable alexa whitelist or if i stop the dnsbl component. anyone else having this problem?

    It might be related to Alexa retiring its TOP1M feed… They have since restored the feed, but uncertain of its ultimate fate...

    https://twitter.com/adamcaudill/status/800813805009768448
    https://twitter.com/Alexa_Support/status/801167423726489600

    I would try the following and see if it fixes this issue:

    mv /var/db/pfblockerng/top-1m.csv /var/db/pfblockerng/top-1m.csv.old
    

    Then run a "Force Reload" and see if it re-downloads the Alexa Archive and rebuilds the Alexa whitelist…

    As reference, the Alexa TOP1M feed url:

    https://s3.amazonaws.com/alexa-static/top-1m.csv.zip
    


  • @BBcan177:

    @laviniuc:

    the cron/reload/update job stalls for me. no error, just stops at dnsbl. works if i disable alexa whitelist or if i stop the dnsbl component. anyone else having this problem?

    It might be related to Alexa retiring its TOP1M feed… They have since restored the feed, but uncertain of its ultimate fate...

    https://twitter.com/adamcaudill/status/800813805009768448
    https://twitter.com/Alexa_Support/status/801167423726489600

    I would try the following and see if it fixes this issue:

    mv /var/db/pfblockerng/top-1m.csv /var/db/pfblockerng/top-1m.csv.old
    

    Then run a "Force Reload" and see if it re-downloads the Alexa Archive and rebuilds the Alexa whitelist…

    As reference, the Alexa TOP1M feed url:

    https://s3.amazonaws.com/alexa-static/top-1m.csv.zip
    

    thanks, much obliged, it works perfectly now. i'll keep on the lookout for if they drop the feed again… maybe you could check if anything weird happens on the import code if the url can't be found? before i stopped the import / disabled dnsbl it was stalling the cpu/dns resolver (guessing weird things happened when the first import took as long as the refresh timeout and then the second import started wihtout the first one completing?)



  • Hi
    I've just installed pfBlockerNG 2.1.1_4 on a test pfSense install.

    I've enabled it and then gone to update.
    I've selected the option to reload but nothing is downloaded.

    Checking /var/db/aliastables there is nothing showing.
    Can anyone point me in the right direction ?

     UPDATE PROCESS START [ 12/01/16 11:26:27 ]
    
    Clearing all DNSBL Feeds... 
    ** DNSBL Disabled **
    
    ===[  Continent Process  ]============================================
    
    ===[  IPv4 Process  ]=================================================
    
    ===[  IPv6 Process  ]=================================================
    
    ===[  Aliastables / Rules  ]==========================================
    
    No changes to Firewall rules, skipping Filter Reload
    No Changes to Aliases, Skipping pfctl Update
    
     UPDATE PROCESS ENDED
    

    Thanks



  • Did you define any tables ?





  • @tonymorella:

    @Atlan:

    I am receiving certificate invalid errors after enabling the DNSBL when navigating certain sites - so far the culprit seems to mainly be googleads.g.doubleclick.net when trying to serve content over an HTTPS enabled website.

    The attempt still shows in the Alerts tab of pfBlockerNG, the main site still loads, but it seems to be wanting to use the firewall's self signed certificate as the certificate for the ad server host.

    Did I miss a setting somewhere to prevent this from happening?

    Ya ran into the same problem while beta testing, completely forgot about this issue until I read your post :) For a quick fix I created a CA and web certificate via pfsense GUI then copies this cert to /var/unbound/dnsbl_cert.pem which dnsbl light http will use.  Then push out the CA to all the desktops, could be done via GPO or login scripts. With https://letsencrypt.org/ picking up traction, everything is going to be running over TLS or SSL soon going to have to implement this type of fix.

    **EDIT: Sorry guys did a bad job on this post and what I was doing, it does not fix the underlying issue with the TLS stream being bad because the cert is not valid, reggie14 did a much better job testing this than I. In my setup is simply helped limit the errors in Chrome. Again just a quick fix that might help. Again sorry for the confusion.

    FYI the update from Chrome 45 to 46 made a difference in what come up under security warnings:
    http://arstechnica.com/information-technology/2015/10/chrome-finally-kills-off-the-http-https-mixed-content-warning/**

    Thanks
    Tony M

    I am new to pfsense and new to this forum. I do not tend to go digging up the past but was there any conclusion to the https page errors received while using DNSBL? I initially scrapped my setup of squidguard due to its failure to even pass through to HTTPS sites. Now the DNSBL is working adequately but if I am to block certain domains (such as www.yahoo.com), I receive an insecure connection error on my browser instead of the 1x1 pixel error page. Is there a way for me to get rid of this error? Or is there a way to potentially redirect this https domain being blocked to an internal webpage?

    Also if this is in the wrong section or if this has already been answered I apologize in advance.


  • Moderator

    @Knyte:

    Is it possible to use an alternative list, such as:

    https://blog.majestic.com/development/alexa-top-1-million-sites-retired-heres-majestic-million/

    Alexa has re-instated the Alexa Top1M Feed, not sure if/when they will discontinue it again….
        https://twitter.com/Alexa_Support/status/801167423726489600

    However, Cisco has recently released an alternative TOP1M version:
        https://blog.opendns.com/2016/12/14/cisco-umbrella-1-million/

    I will be adding this to the next release as an alternative option to the package...


  • Moderator

    @BoxMacNCheese:

    …https page errors received while using DNSBL?

    Usually this occurs with Safari Browsers only… I don't know of any workarounds, except for whitelisting those domains that are causing this behaviour...



  • Hi BBcan177,

    Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

    It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

    Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

    FYI, the two lists that I added that killed the system were these:

    http://osint.bambenekconsulting.com/feeds/dga-feed.gz
    http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

    Thanks,

    AK



  • Try to raise System/ Advanced / Firewall & NAT : Firewall Maximum Table Entries, but this may only be useful for IP tables.

    Maybe you are just running out of memory, those DNSBL tables are huge.
    -rw-r–r--  1 root  wheel  45701811 Dec 18 14:22 BBC_DGA.txt



  • Okay so as I understand I will not be able to avoid the HTTPS errors in my browser due to the nature of DNS. For example, if I were to block www.aol.com in a DNSBL feed, I would be redirected to the 1x1 pixel block page at my DNSBL VIP of 192.168.1.10. If I went to https://www.aol.com I receive a browser error page of "ssl_error_bad_cert_domain" because the DNSBL certificate obviously doesn't match up with the DNS entry of www.aol.com and the internal DNSBL VIP.

    1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

    2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

    Thanks.



  • @BoxMacNCheese:

    2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

    https://forum.pfsense.org/index.php?topic=120253.0



  • @BoxMacNCheese:

    1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

    Thanks.



  • @BBcan177:

    Here are some basic instructions to get started with DNSBL.

    1. Open the pfBNG "DNSBL" Tab:

    (Use the defaults unless you have a need to use otherwise)
    Enter the DNSBL VIP as 10.10.10.1
    Enter the DNSBL Listening Port as 8081
    Enter the DNSBL SSL Listening port as 8443
    Select the DNSBL Listening Interface as Lan

    For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
    Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

    DNSBL IP Firewall Rule Settings:
    Select Deny outbound or as per your requirements
    select Enable logging

    Alexa  (is optional, you can skip this until later if you wish)
    Select Top 1K
    Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

    In the Custom List you may enter any domain you wish to Whitelist.

    Save your settings

    1. Open the "DNSBL Feeds" Tab:

    Create a new DNSBL Alias

    Enter DNS Group Name as ADs
    Enter Description as DNSBL ADverts

    DNSBL:

    Enter the Header/Label and Source URL as follows:
    (Use copy/paste as plain text for the URL)

    Format Auto and State ON

    yoyo
    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

    hpHosts_ads
    http://hosts-file.net/ad_servers.txt

    Adaway
    https://adaway.org/hosts.txt

    Cameleon
    http://sysctl.org/cameleon/hosts

    Select List Action as Unbound
    Select Update Frequency as Once a day

    Alexa:
    Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

    Add any other domains that you wish to block in the Custom List.

    Save your settings

    1. Open the "DNSBL" Tab:

    Click DNSBL Enable checkbox.
    Save your settings

    1. Open the "Update" Tab:

    Select Force Update

    You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

    1. Goto the pfBlockerNG "Alerts" Tab:

    Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

    As a test, goto www.aol.com  and www.yahoo.com  and see some alerts populate.

    There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

    I tried to look for an answer, but I cannot find it (maybe it's obvious), but can someone assure me?
    BBcan177 advises not to filter the DNSBL feeds mentioned above with Alexa. So  does that mean that for all DNSBL feeds, other than advertising, you should enable Alexa?



  • alexa is a preference thing.

    Enabling it means you will not accidentally block popular websites.  On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

    My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's?  If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.



  • @akishore:

    Hi BBcan177,

    Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

    It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

    Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

    FYI, the two lists that I added that killed the system were these:

    http://osint.bambenekconsulting.com/feeds/dga-feed.gz
    http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

    Thanks,

    AK

    My device is higher end than yours, it didnt crash but unbound started been weird, it kept restarting and on each restart it was unresponsive.

    I then did a bit of research as I found it very hard to believe that dga list is all live domains and it turns out I am right.

    Basically the dga list is someone has inputted known seeds used by botnets and made a list of all the possible domains to be used from those seeds, these are not verified live domains so given the resource complications I removed the dga, the c2 list is verified active domains,. but of course is also much smaller so shouldnt give you problems to use.



  • @chrcoluk:

    alexa is a preference thing.

    Enabling it means you will not accidentally block popular websites.  On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

    My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's?  If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.

    Thanks, I will try it and see then wager the results.


  • Banned

    Still looking for some guidance on this topic? Policy Based Routing to a VPN Gateway results in no pfblockerNG filtering. How can I fix this?

    It looks like I got one thing working on my setup but killed DNSBL in the process.

    I recently setup policy based routing where my pass firewall rules determine the Gateway to be used. I did this so that I could have one interface route all traffic through my VPN client and another interface not utilize the VPN at all. That works great, but ever since I set that up I have 0 hits on all of my DNSBL feeds and ads are coming through.

    Any ideas on how I can use pfBlockerNG DNSBL while still utilizing policy based routing for my purposes?



  • quick update since my post is now out of date as to what is going on with my device.

    unbound was not crashing but it was the restarts for the dhcp renewals which I now stopped by patching services.inc, I did enable the DGA list again but as I said before the DGA list is not live domains but a domain list generated from discovered seeds of malware domain generators, of course the domains could be made live at any given moment so its a way to preemt them.


  • Banned

    I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

    But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

    I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.



  • @pfBasic:

    I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

    But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

    I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.

    In order for DNSBL to block domain name, all clients DNS requests must use the pfSense DNS resolver. https://forum.pfsense.org/index.php?topic=102470.msg572527#msg572527

    Normally, in General setup : you leave the DNS servers fields empty, you disable Allow DNS server list to be overridden by DHCP/PPP on WAN
    in DHCP / Server you leave the DNS Servers fields empty.


  • Banned

    So my understanding with that setup is that all clients will use the pfsense default (ISP) DNS that will be filtered by pfbng w/ dnsbl since we haven't specified anything else for it to use.

    Is there a way present or future (maybe some advanced settings entry?) that allows us to tell pfbng w/ dnsbl directly to use a specific DNS or list of DNS?

    It seems like that should work (not that I'd know) but I don't see why it is necessary that pfbng w/ dnsbl uses an ISP's DNS. If it can filter requests going through the ISP couldn't it do the same with any public DNS?




  • Banned

    Im not sure what im supposed to be picking up from that? No DNS forwarder? I've never used DNS Forwarder so I really dont know what is implied by that? It also says that you can use the DNS forwarding mode of DNS Resolver, but implies that if I use forwarding mode then i'll use my ISP's DNS, whereas if i leave it in resolver mode i'll use the root DNS (don't know what that is?).

    Im using DNS resolver, not forwarder, and not using forwarding mode, but my queries are still going through my ISP DNS.



  • @pfBasic:

    So my understanding with that setup is that all clients will use the pfsense default (ISP) DNS that will be filtered by pfbng w/ dnsbl since we haven't specified anything else for it to use.

    All clients will use the pfsense DNS resolver, that you can configure in forwarding mode to use your ISP DNS server, but by default, why not use the pfsense DNS Resolver in Root mode without having to rely on an external DNS server.

    @pfBasic:

    Is there a way present or future (maybe some advanced settings entry?) that allows us to tell pfbng w/ dnsbl directly to use a specific DNS or list of DNS?

    That what pfSense DNS Resolver forwarding mode is for.



  • @pfBasic:

    Im not sure what im supposed to be picking up from that? No DNS forwarder? I've never used DNS Forwarder so I really dont know what is implied by that? It also says that you can use the DNS forwarding mode of DNS Resolver, but implies that if I use forwarding mode then i'll use my ISP's DNS, whereas if i leave it in resolver mode i'll use the root DNS (don't know what that is?).

    Im using DNS resolver, not forwarder, and not using forwarding mode, but my queries are still going through my ISP DNS.

    Check the hosts DNS configuration to see what DNS server they use.


  • Banned

    All clients will use the pfsense DNS resolver, that you can configure in forwarding mode to use your ISP DNS server, but by default, why not use the pfsense DNS Resolver in Root mode without having to rely on an external DNS server.

    OK, awesome, thank-you. This is what I must be doing wrong. As far as I thought, I am using the DNS resolver (not using forwarded, forwarding mode not turned on, and I've removed all entries in general setup & dhcp for alternative DNS servers).
    But whenever I run dnsleaktest.com, I still get my ISP's DNS server?

    What am I doing wrong? How can I run DNS off of pfsenses internal DNS?

    I didn't even know pfsense did this natively, I thought if I wanted to use an internal DNS server then I had to set up and run the whole thing on a real or virtual machine, which I dont know how to do.



  • yup pfsense has a DNS Resolver  ;)

    So now you have to check your hosts/device DNS configuration.

    If they get DNS server configuration from a DHCP server, then configure the DHCP server to send pfsense ip.

    @ :

    Leave blank to use the system default DNS servers: this interface's IP if DNS Forwarder or Resolver is enabled, otherwise the servers configured on the System / General Setup page.


  • Banned

    awesome, i got it working as its meant to be, thank you for sticking with my ignorant line of questioning!

    One last question, how does the "root server" resolve the DNS lookups? as in what tells my pfsense box the ip address that google.com resolves to?


Log in to reply