PfBlockerNG v2.0 w/DNSBL

BBcan177

@Knyte:

Is it possible to use an alternative list, such as:

https://blog.majestic.com/development/alexa-top-1-million-sites-retired-heres-majestic-million/

Alexa has re-instated the Alexa Top1M Feed, not sure if/when they will discontinue it again….
https://twitter.com/Alexa_Support/status/801167423726489600

However, Cisco has recently released an alternative TOP1M version:
https://blog.opendns.com/2016/12/14/cisco-umbrella-1-million/

I will be adding this to the next release as an alternative option to the package...

BBcan177

@BoxMacNCheese:

…https page errors received while using DNSBL?

Usually this occurs with Safari Browsers only… I don't know of any workarounds, except for whitelisting those domains that are causing this behaviour...

akishore

Hi BBcan177,

Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

FYI, the two lists that I added that killed the system were these:

http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

Thanks,

AK

RonpfS

Try to raise System/ Advanced / Firewall & NAT : Firewall Maximum Table Entries, but this may only be useful for IP tables.

Maybe you are just running out of memory, those DNSBL tables are huge.
-rw-r–r-- 1 root wheel 45701811 Dec 18 14:22 BBC_DGA.txt

BoxMacNCheese

Okay so as I understand I will not be able to avoid the HTTPS errors in my browser due to the nature of DNS. For example, if I were to block www.aol.com in a DNSBL feed, I would be redirected to the 1x1 pixel block page at my DNSBL VIP of 192.168.1.10. If I went to https://www.aol.com I receive a browser error page of "ssl_error_bad_cert_domain" because the DNSBL certificate obviously doesn't match up with the DNS entry of www.aol.com and the internal DNSBL VIP.

1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

Thanks.

RonpfS

@BoxMacNCheese:

2. Where is the filepath of DNSBL package, or more directly, where is the location of the 1x1 pixel block page (or image) since I would like to put my own error notice.

https://forum.pfsense.org/index.php?topic=120253.0

BoxMacNCheese

@BoxMacNCheese:

1. Would I be able to redirect all of the traffic that is blocked and goes to the DNSBL VIP address to go to an internal (or external) custom webpage. Or perhaps have it be redirected to Google? Even if it was HTTPS would it not receive an error and pass through?

Thanks.

Qinn

@BBcan177:

Here are some basic instructions to get started with DNSBL.

Open the pfBNG "DNSBL" Tab:

(Use the defaults unless you have a need to use otherwise)
Enter the DNSBL VIP as 10.10.10.1
Enter the DNSBL Listening Port as 8081
Enter the DNSBL SSL Listening port as 8443
Select the DNSBL Listening Interface as Lan

For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

DNSBL IP Firewall Rule Settings:
Select Deny outbound or as per your requirements
select Enable logging

Alexa (is optional, you can skip this until later if you wish)
Select Top 1K
Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

In the Custom List you may enter any domain you wish to Whitelist.

Save your settings

Open the "DNSBL Feeds" Tab:

Create a new DNSBL Alias

Enter DNS Group Name as ADs
Enter Description as DNSBL ADverts

DNSBL:

Enter the Header/Label and Source URL as follows:
(Use copy/paste as plain text for the URL)

Format Auto and State ON

yoyo
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

hpHosts_ads
http://hosts-file.net/ad_servers.txt

Adaway
https://adaway.org/hosts.txt

Cameleon
http://sysctl.org/cameleon/hosts

Select List Action as Unbound
Select Update Frequency as Once a day

Alexa:
Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

Add any other domains that you wish to block in the Custom List.

Save your settings

Open the "DNSBL" Tab:

Click DNSBL Enable checkbox.
Save your settings

Open the "Update" Tab:

Select Force Update

You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

Goto the pfBlockerNG "Alerts" Tab:

Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

As a test, goto www.aol.com and www.yahoo.com and see some alerts populate.

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

I tried to look for an answer, but I cannot find it (maybe it's obvious), but can someone assure me?
BBcan177 advises not to filter the DNSBL feeds mentioned above with Alexa. So does that mean that for all DNSBL feeds, other than advertising, you should enable Alexa?

chrcoluk

alexa is a preference thing.

Enabling it means you will not accidentally block popular websites. On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's? If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.

chrcoluk

@akishore:

Hi BBcan177,

Thanks for the awesome package. I had a quick question for you. I added quite a few feeds that you had mentioned in the beginning of this thread and everything was working great. Anyway, I found some more feeds, like really big ones and tried to add those.

It loaded, but the database grew to about 1.4 million entries. Firstly, it started giving me a message about a 2 million hard limit and also, my entire device basically crashed. I have the SG-2440, so it's not one of the higher end models, but I didn't think it would crash the device.

Anyway, Unbound could not be reloaded and it reverted back to the smaller database (which was about 500,000 URLs). Can my device just not handle that big of a table or is there some other way to fix this?

FYI, the two lists that I added that killed the system were these:

http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

Thanks,

AK

My device is higher end than yours, it didnt crash but unbound started been weird, it kept restarting and on each restart it was unresponsive.

I then did a bit of research as I found it very hard to believe that dga list is all live domains and it turns out I am right.

Basically the dga list is someone has inputted known seeds used by botnets and made a list of all the possible domains to be used from those seeds, these are not verified live domains so given the resource complications I removed the dga, the c2 list is verified active domains,. but of course is also much smaller so shouldnt give you problems to use.

Qinn

@chrcoluk:

alexa is a preference thing.

Enabling it means you will not accidentally block popular websites. On the flipside should one of those sites ever get compromised, you also wouldnt be filtering it.

My own reason for posting here is I wonder if anyone has an issue with unbound been unresponsive during reload's? If I add the cryptolocker domains of which there is almost 1 million of them (surely some are dead and not been cleaned up), unbound then needs 20+ seconds to reload at which time it is unresponsive.

Thanks, I will try it and see then wager the results.

pfBasic

Still looking for some guidance on this topic? Policy Based Routing to a VPN Gateway results in no pfblockerNG filtering. How can I fix this?

It looks like I got one thing working on my setup but killed DNSBL in the process.

I recently setup policy based routing where my pass firewall rules determine the Gateway to be used. I did this so that I could have one interface route all traffic through my VPN client and another interface not utilize the VPN at all. That works great, but ever since I set that up I have 0 hits on all of my DNSBL feeds and ads are coming through.

Any ideas on how I can use pfBlockerNG DNSBL while still utilizing policy based routing for my purposes?

chrcoluk

quick update since my post is now out of date as to what is going on with my device.

unbound was not crashing but it was the restarts for the dhcp renewals which I now stopped by patching services.inc, I did enable the DGA list again but as I said before the DGA list is not live domains but a domain list generated from discovered seeds of malware domain generators, of course the domains could be made live at any given moment so its a way to preemt them.

pfBasic

I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.

RonpfS

@pfBasic:

I've got everything back up and running again. pfblockerng w/ DNSBL, openvpn client & server.

But now all of my DNS requests go through my ISP. Not really a big deal, but is it possible to make my requests go through google or openvpn dns instead?

I have googe and opendns ip's listed under general setup, and under dhcp. but i keep getting ISP dns.

In order for DNSBL to block domain name, all clients DNS requests must use the pfSense DNS resolver. https://forum.pfsense.org/index.php?topic=102470.msg572527#msg572527

Normally, in General setup : you leave the DNS servers fields empty, you disable Allow DNS server list to be overridden by DHCP/PPP on WAN
in DHCP / Server you leave the DNS Servers fields empty.

pfBasic

So my understanding with that setup is that all clients will use the pfsense default (ISP) DNS that will be filtered by pfbng w/ dnsbl since we haven't specified anything else for it to use.

Is there a way present or future (maybe some advanced settings entry?) that allows us to tell pfbng w/ dnsbl directly to use a specific DNS or list of DNS?

It seems like that should work (not that I'd know) but I don't see why it is necessary that pfbng w/ dnsbl uses an ISP's DNS. If it can filter requests going through the ISP couldn't it do the same with any public DNS?

RonpfS

Read again https://forum.pfsense.org/index.php?topic=102470.msg572527#msg572527
about forwarding mode.

pfBasic

Im not sure what im supposed to be picking up from that? No DNS forwarder? I've never used DNS Forwarder so I really dont know what is implied by that? It also says that you can use the DNS forwarding mode of DNS Resolver, but implies that if I use forwarding mode then i'll use my ISP's DNS, whereas if i leave it in resolver mode i'll use the root DNS (don't know what that is?).

Im using DNS resolver, not forwarder, and not using forwarding mode, but my queries are still going through my ISP DNS.

RonpfS

@pfBasic:

So my understanding with that setup is that all clients will use the pfsense default (ISP) DNS that will be filtered by pfbng w/ dnsbl since we haven't specified anything else for it to use.

All clients will use the pfsense DNS resolver, that you can configure in forwarding mode to use your ISP DNS server, but by default, why not use the pfsense DNS Resolver in Root mode without having to rely on an external DNS server.

@pfBasic:

Is there a way present or future (maybe some advanced settings entry?) that allows us to tell pfbng w/ dnsbl directly to use a specific DNS or list of DNS?

That what pfSense DNS Resolver forwarding mode is for.

RonpfS

@pfBasic:

Im not sure what im supposed to be picking up from that? No DNS forwarder? I've never used DNS Forwarder so I really dont know what is implied by that? It also says that you can use the DNS forwarding mode of DNS Resolver, but implies that if I use forwarding mode then i'll use my ISP's DNS, whereas if i leave it in resolver mode i'll use the root DNS (don't know what that is?).

Im using DNS resolver, not forwarder, and not using forwarding mode, but my queries are still going through my ISP DNS.

Check the hosts DNS configuration to see what DNS server they use.