PfBlockerNG v2.0 w/DNSBL

laviniuc

@laviniuc:

the cron/reload/update job stalls for me. no error, just stops at dnsbl. works if i disable alexa whitelist or if i stop the dnsbl component. anyone else having this problem?

It might be related to Alexa retiring its TOP1M feed… They have since restored the feed, but uncertain of its ultimate fate...

https://twitter.com/adamcaudill/status/800813805009768448
https://twitter.com/Alexa_Support/status/801167423726489600

I would try the following and see if it fixes this issue:
mv /var/db/pfblockerng/top-1m.csv /var/db/pfblockerng/top-1m.csv.old
Then run a "Force Reload" and see if it re-downloads the Alexa Archive and rebuilds the Alexa whitelist…

As reference, the Alexa TOP1M feed url:
https://s3.amazonaws.com/alexa-static/top-1m.csv.zip

thanks, much obliged, it works perfectly now. i'll keep on the lookout for if they drop the feed again… maybe you could check if anything weird happens on the import code if the url can't be found? before i stopped the import / disabled dnsbl it was stalling the cpu/dns resolver (guessing weird things happened when the first import took as long as the refresh timeout and then the second import started wihtout the first one completing?)

TomT

Hi
I've just installed pfBlockerNG 2.1.1_4 on a test pfSense install.

I've enabled it and then gone to update.
I've selected the option to reload but nothing is downloaded.

Checking /var/db/aliastables there is nothing showing.
Can anyone point me in the right direction ?

 UPDATE PROCESS START [ 12/01/16 11:26:27 ]

Clearing all DNSBL Feeds... 
** DNSBL Disabled **

===[  Continent Process  ]============================================

===[  IPv4 Process  ]=================================================

===[  IPv6 Process  ]=================================================

===[  Aliastables / Rules  ]==========================================

No changes to Firewall rules, skipping Filter Reload
No Changes to Aliases, Skipping pfctl Update

 UPDATE PROCESS ENDED

Thanks

RonpfS

Did you define any tables ?

DigitalJer

Is it possible to use an alternative list, such as:

https://blog.majestic.com/development/alexa-top-1-million-sites-retired-heres-majestic-million/

BoxMacNCheese

@tonymorella:

@Atlan:

I am receiving certificate invalid errors after enabling the DNSBL when navigating certain sites - so far the culprit seems to mainly be googleads.g.doubleclick.net when trying to serve content over an HTTPS enabled website.

The attempt still shows in the Alerts tab of pfBlockerNG, the main site still loads, but it seems to be wanting to use the firewall's self signed certificate as the certificate for the ad server host.

Did I miss a setting somewhere to prevent this from happening?

Ya ran into the same problem while beta testing, completely forgot about this issue until I read your post :) For a quick fix I created a CA and web certificate via pfsense GUI then copies this cert to /var/unbound/dnsbl_cert.pem which dnsbl light http will use. Then push out the CA to all the desktops, could be done via GPO or login scripts. With https://letsencrypt.org/ picking up traction, everything is going to be running over TLS or SSL soon going to have to implement this type of fix.

**EDIT: Sorry guys did a bad job on this post and what I was doing, it does not fix the underlying issue with the TLS stream being bad because the cert is not valid, reggie14 did a much better job testing this than I. In my setup is simply helped limit the errors in Chrome. Again just a quick fix that might help. Again sorry for the confusion.

FYI the update from Chrome 45 to 46 made a difference in what come up under security warnings:
http://arstechnica.com/information-technology/2015/10/chrome-finally-kills-off-the-http-https-mixed-content-warning/**

Thanks
Tony M

I am new to pfsense and new to this forum. I do not tend to go digging up the past but was there any conclusion to the https page errors received while using DNSBL? I initially scrapped my setup of squidguard due to its failure to even pass through to HTTPS sites. Now the DNSBL is working adequately but if I am to block certain domains (such as www.yahoo.com), I receive an insecure connection error on my browser instead of the 1x1 pixel error page. Is there a way for me to get rid of this error? Or is there a way to potentially redirect this https domain being blocked to an internal webpage?

Also if this is in the wrong section or if this has already been answered I apologize in advance.

BBcan177

@Knyte:

Is it possible to use an alternative list, such as:

https://blog.majestic.com/development/alexa-top-1-million-sites-retired-heres-majestic-million/

Alexa has re-instated the Alexa Top1M Feed, not sure if/when they will discontinue it again….
https://twitter.com/Alexa_Support/status/801167423726489600

However, Cisco has recently released an alternative TOP1M version:
https://blog.opendns.com/2016/12/14/cisco-umbrella-1-million/

I will be adding this to the next release as an alternative option to the package...

BBcan177

@BoxMacNCheese:

…https page errors received while using DNSBL?

Usually this occurs with Safari Browsers only… I don't know of any workarounds, except for whitelisting those domains that are causing this behaviour...

akishore

Hi BBcan177,

Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

FYI, the two lists that I added that killed the system were these:

http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

Thanks,

AK

RonpfS

Try to raise System/ Advanced / Firewall & NAT : Firewall Maximum Table Entries, but this may only be useful for IP tables.

Maybe you are just running out of memory, those DNSBL tables are huge.
-rw-r–r-- 1 root wheel 45701811 Dec 18 14:22 BBC_DGA.txt

BoxMacNCheese

Okay so as I understand I will not be able to avoid the HTTPS errors in my browser due to the nature of DNS. For example, if I were to block www.aol.com in a DNSBL feed, I would be redirected to the 1x1 pixel block page at my DNSBL VIP of 192.168.1.10. If I went to https://www.aol.com I receive a browser error page of "ssl_error_bad_cert_domain" because the DNSBL certificate obviously doesn't match up with the DNS entry of www.aol.com and the internal DNSBL VIP.

1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

Thanks.

RonpfS

@BoxMacNCheese:

2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

https://forum.pfsense.org/index.php?topic=120253.0

BoxMacNCheese

@BoxMacNCheese:

1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

Thanks.

Qinn

@BBcan177:

Here are some basic instructions to get started with DNSBL.

Open the pfBNG "DNSBL" Tab:

(Use the defaults unless you have a need to use otherwise)
Enter the DNSBL VIP as 10.10.10.1
Enter the DNSBL Listening Port as 8081
Enter the DNSBL SSL Listening port as 8443
Select the DNSBL Listening Interface as Lan

For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

DNSBL IP Firewall Rule Settings:
Select Deny outbound or as per your requirements
select Enable logging

Alexa (is optional, you can skip this until later if you wish)
Select Top 1K
Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

In the Custom List you may enter any domain you wish to Whitelist.

Save your settings

Open the "DNSBL Feeds" Tab:

Create a new DNSBL Alias

Enter DNS Group Name as ADs
Enter Description as DNSBL ADverts

DNSBL:

Enter the Header/Label and Source URL as follows:
(Use copy/paste as plain text for the URL)

Format Auto and State ON

yoyo
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

hpHosts_ads
http://hosts-file.net/ad_servers.txt

Adaway
https://adaway.org/hosts.txt

Cameleon
http://sysctl.org/cameleon/hosts

Select List Action as Unbound
Select Update Frequency as Once a day

Alexa:
Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

Add any other domains that you wish to block in the Custom List.

Save your settings

Open the "DNSBL" Tab:

Click DNSBL Enable checkbox.
Save your settings

Open the "Update" Tab:

Select Force Update

You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

Goto the pfBlockerNG "Alerts" Tab:

Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

As a test, goto www.aol.com and www.yahoo.com and see some alerts populate.

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

I tried to look for an answer, but I cannot find it (maybe it's obvious), but can someone assure me?
BBcan177 advises not to filter the DNSBL feeds mentioned above with Alexa. So does that mean that for all DNSBL feeds, other than advertising, you should enable Alexa?

chrcoluk

alexa is a preference thing.

Enabling it means you will not accidentally block popular websites. On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's? If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.

chrcoluk

@akishore:

Hi BBcan177,

Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

FYI, the two lists that I added that killed the system were these:

http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

Thanks,

AK

My device is higher end than yours, it didnt crash but unbound started been weird, it kept restarting and on each restart it was unresponsive.

I then did a bit of research as I found it very hard to believe that dga list is all live domains and it turns out I am right.

Basically the dga list is someone has inputted known seeds used by botnets and made a list of all the possible domains to be used from those seeds, these are not verified live domains so given the resource complications I removed the dga, the c2 list is verified active domains,. but of course is also much smaller so shouldnt give you problems to use.

Qinn

@chrcoluk:

alexa is a preference thing.

Enabling it means you will not accidentally block popular websites. On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's? If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.

Thanks, I will try it and see then wager the results.

pfBasic

Still looking for some guidance on this topic? Policy Based Routing to a VPN Gateway results in no pfblockerNG filtering. How can I fix this?

It looks like I got one thing working on my setup but killed DNSBL in the process.

I recently setup policy based routing where my pass firewall rules determine the Gateway to be used. I did this so that I could have one interface route all traffic through my VPN client and another interface not utilize the VPN at all. That works great, but ever since I set that up I have 0 hits on all of my DNSBL feeds and ads are coming through.

Any ideas on how I can use pfBlockerNG DNSBL while still utilizing policy based routing for my purposes?

chrcoluk

quick update since my post is now out of date as to what is going on with my device.

unbound was not crashing but it was the restarts for the dhcp renewals which I now stopped by patching services.inc, I did enable the DGA list again but as I said before the DGA list is not live domains but a domain list generated from discovered seeds of malware domain generators, of course the domains could be made live at any given moment so its a way to preemt them.

pfBasic

I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.

RonpfS

@pfBasic:

I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.

In order for DNSBL to block domain name, all clients DNS requests must use the pfSense DNS resolver. https://forum.pfsense.org/index.php?topic=102470.msg572527#msg572527

Normally, in General setup : you leave the DNS servers fields empty, you disable Allow DNS server list to be overridden by DHCP/PPP on WAN
in DHCP / Server you leave the DNS Servers fields empty.