PfBlockerNG v2.0 w/DNSBL

BBcan177

pfBlockerNG v2.0

This release brings several improvements and enhancements to the existing pfBlockerNG IP Download manager capabilities, and also introduces domain name blocking ("DNSBL") via Unbound DNS Resolver. Domain blocking can be used for ADvert servers, malicious domains and/or domain filtering.

DNSBL overview: When a DNS request is made for a domain that is listed in DNSBL, the request is redirected to a Virtual IP address where an instance of Lighttpd Web Server will collect the packet statistics and push a '1x1' GIF image to the Browser. Rejected domain activity can be viewed in the "Alerts" tab along with the IP Alerts.

Due to the increased security measures in Browsers, HTTPS alerts will not display the SRC IP or Requested URL, however, the Requested Domain name will be logged to the Alerts tab along with all HTTP alerts.

To reduce False Positives, the "Alexa" Top 1M sites can be used to suppress the most popular sites. Settings allow for the number of Top sites and which TLDs (.com, .org etc) to include in the Alexa Suppression list.

Improved Features

[ [color=red]cURL ]

PHP cURL library is now used to download all Lists/Feeds/Maxmind/Alexa files. This allows for increased security settings and collection of HTTP response codes. Allowed SSL Ciphers (TLSv1.2 and TLSv1) and default settings (SSL_VERIFYPEER, SSL_VERIFYHOST).

Added a "Flex" option to allow for the downgrade of the SSL connection, if required. The "Flex" option (when enabled) will downgrade on the following HTTP (RFC7231) response codes:

35 - GET_SERVER_HELLO:sslv3
51 - NO alternative certificate
60 - Local Issuer Certificate Subject

The following settings will be downgraded:

disable verifypeer
disable verifyhost
adds SSLv3 SSL Cipher.

(It is not recommended to use the "Flex" setting; however not all Lists/Feeds support the increased security measures. Expect overtime that these Lists/Feeds will improve their security.)

[ [color=red]Alerts Tab ]

The Alerts tab has been refactored. Improvements in efficiency and accuracy in the IP/CIDR for the reported Alerts. The Suppression Icon is now only visible for /32 and /24 CIDRS, all other suppression's require an 'Outbound' allow firewall rule to circumvent the blocklist. Hostnames for local devices are now displayed under each Alerted IP. When 'Filter options' are used, they will be preserved on page refreshes.

[ [color=red]Reputation ]

All reputation functions have been refactored and improvements made to the overall efficiency.

[[color=red] IP Parser ]

The IP parser has been converted to use String functions vs regex functions, for a significant improvement in efficiency.

[ [color=red]List/Feed Format ]

The package will now "automagically" determine what 'Format' is required to extract and parse any given URL.

[ [color=red]Firewall Rules ]

The "Firewall Rule" creation function has been improved. Please verify your firewall rules post installation to ensure they are as expected.

[ [color=red]Config Save ]

Only when changes are made to the settings, will pfBlockerNG save its settings to the pfSense config file. This will reduce the spammed backups that were being created at each hourly CRON task interval.

[ [color=red]XMLRPC ]

XMLRPC sync has been improved to include support for hostnames and IPv6 addresses.

[ [color=red]XSS potential ]

Fixed XSS potentials in several functions of the code.

New Features

[ [color=red]Whois ]

A whois feature has been added to convert an AS number into its respective IP Addresses. Entering a Domain name will also collect its host IP Address.

[ [color=red]CIDR Aggregation ]

Aggregation will optimize contiguous network prefixes in a List, and remove any superfluous prefixes which are already included in another prefix. Depending on the hardware used, Aggregation can use a significant amount of resources if the list being aggregated is large.

[ [color=red]Max daily download failure attempts ]

Allows for the configuration of a "max daily download failure threshold" for the CRON updates. If this setting is defined, attempts will be made to re-download up-to the threshold defined. The 'failed download' counters can be clearing in the widget.

[ [color=red]Restore previous download on failure ]

When enabled, the previously downloaded file contents are used on a download failure. (Defaulted to 'on')

[[color=red] Kill States ]

After a cron event or any 'Force' commands (when enabled), any IPs listed in any Blocklists; if found in the Firewall states will be cleared and logged. States removal can be bypassed on an alias by alias basis if required.

[[color=red] Tracker IDs ]

The pfBlockerNG Firewall Rules now include a 'Tracker ID', this ID is based upon the Alias name (Sum of the ASCII characters ORD values). This ID is a 10 digit number, prefixed with '177'. This ID will remain static unless the Alias name is changed. If the Firewall Logs are pushed to a remote syslog, you can use the Tracker ID as a reference to the pfBlockerNG Alias.

Further details are available in the following forum thread:
https://forum.pfsense.org/index.php?topic=86212.0

q54e3w

awesome job, thanks BBCan!

mygeeknc

Great work sir! I'm looking forward to this release. Will there be any special instructions for upgrading?

trinidadrancheria

Guess I will hold off manually editing the INC file then to add DNSBL :)
Any ETA on this new version?

THANKS!!!

pfcode

Great. Thanks!

wcrowder

ETA is as soon as the Dev's validate the package, package has been submitted… Yeah!!!!! :)

https://github.com/pfsense/pfsense-packages/pull/1174

trinidadrancheria

@wcrowder:

ETA is as soon as the Dev's validate the package, package has been submitted… Yeah!!!!! :)

https://github.com/pfsense/pfsense-packages/pull/1174

HOLY Poop!!! 23 changed files with 8,591 additions and 5,267 deletions Total re-write? :D

Panja

Looking forward to this release!
Keep up the good work mate. Awesome! 8)

BBcan177

@mygeeknc:

Great work sir! I'm looking forward to this release. Will there be any special instructions for upgrading?

There should be no concerns in migrating to pfBNG v2.0 from the existing v1.10.

pfSense has two types of DNS Services

DNS Forwarder
DNS Resolver

If you plan on using the DNSBL feature, you will need to use the DNS Resolver for your DNS queries, the DNS Forwarder is not an option for DNSBL. Its probably best to ensure that the DNS Resolver is working before using DNSBL.

The DNS Resolver is developed by NLnet Labs and is named 'Unbound'. It is a validating, recursive and caching DNS resolver. https://www.unbound.net/index.html

Some recommendations:

The DNS Resolver can also be used in 'Forwardering mode'; however its best to not use this 'Forwarding mode' and keep it in 'resolver mode' as this will query the Root DNS servers for the DNS queries instead of relying on an ISPs DNS etc…
If you use the 'DNS Resolver Forwarder mode', only configure 'DNSSEC' if the configured DNS servers support DNSSEC. The enabling of 'DNSSEC' to harden your DNS security is highly recommended.
Disable the two "DHCP registrations" checkboxes, unless you really require those options.

Here is a good primer about the DNS Resolver (Unbound) https://calomel.org/unbound_dns.html

gsiemon

@BBcan177:

If you plan on using the DNSBL feature, you will need to use the DNS Resolver for your DNS queries, the DNS Forwarder is not an option for DNSBL. Its probably best to ensure that the

I'm using the Bind package instead of Unbound. Will V2.0 support that or is it hard coded to use only Unbound? If it only supports Unbound, is it possible to support Bind in the future?

doktornotor

@gsiemon:

I'm using the Bind package instead of Unbound. Will V2.0 support that

No.

fraglord

You doin a great job! I appreciate it and many others do so as well. It is a leightweight alternative to squid when it comes to ad-blocking.
I was allowed ot participate in the beta test program and the DNSBL feature is my favourite. Minor issues have been ironed out with the latest upgdates so it for sure feels ready to be released.

a-a-ron

@fraglord:

You doin a great job! I appreciate it and many others do so as well. It is a leightweight alternative to squid when it comes to ad-blocking.
I was allowed ot participate in the beta test program and the DNSBL feature is my favourite. Minor issues have been ironed out with the latest upgdates so it for sure feels ready to be released.

Same here, it was a pleasure to help BBcan177. Hopefully I wasn't too much of a pain :D

musicwizard

yup the beta one works really nice :)
Running my own DNS server and not from my ISP/google/ etc
so dnsbl is really nice on it also. Keep it up BBcan

cmb

This has been merged, it'll be live in the package repo within at most ~15 minutes of the time of this post.

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance. If there are issues, BBCan, please follow up with pull requests and I'll make sure one of us gets them merged promptly. Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

doktornotor

@cmb:

Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Done, all works pretty well (as before). :D

BBcan177

@cmb:

This has been merged, it'll be live in the package repo within at most ~15 minutes of the time of this post.

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance. If there are issues, BBCan, please follow up with pull requests and I'll make sure one of us gets them merged promptly. Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

Thanks Chris! :)

hda

Package update succeeded on the second reinstall from v1.10, after ~11 min. on an Alix pfSense 2.2.5 (ro). Thank you BBcan :)

Update: I followed BBcan's instructions below for starting DNSBL and that worked too. Excellent extension.

BBcan177

Here are some basic instructions to get started with DNSBL.

Open the pfBNG "DNSBL" Tab:

(Use the defaults unless you have a need to use otherwise)
Enter the DNSBL VIP as 10.10.10.1
Enter the DNSBL Listening Port as 8081
Enter the DNSBL SSL Listening port as 8443
Select the DNSBL Listening Interface as Lan

For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

DNSBL IP Firewall Rule Settings:
Select Deny outbound or as per your requirements
select Enable logging

Alexa (is optional, you can skip this until later if you wish)
Select Top 1K
Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

In the Custom List you may enter any domain you wish to Whitelist.

Save your settings

Open the "DNSBL Feeds" Tab:

Create a new DNSBL Alias

Enter DNS Group Name as ADs
Enter Description as DNSBL ADverts

DNSBL:

Enter the Header/Label and Source URL as follows:
(Use copy/paste as plain text for the URL)

Format Auto and State ON

yoyo
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

hpHosts_ads
http://hosts-file.net/ad_servers.txt

Adaway
https://adaway.org/hosts.txt

Cameleon
http://sysctl.org/cameleon/hosts

Select List Action as Unbound
Select Update Frequency as Once a day

Alexa:
Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

Add any other domains that you wish to block in the Custom List.

Save your settings

Open the "DNSBL" Tab:

Click DNSBL Enable checkbox.
Save your settings

Open the "Update" Tab:

Select Force Update

You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

Goto the pfBlockerNG "Alerts" Tab:

Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

As a test, goto www.aol.com and www.yahoo.com and see some alerts populate.

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

n3by

@cmb:

…

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance.
...
Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

uninstalled test 77 DEV and installed 2.0 from official on 2 boxes v2.2.4 & v2.2.5

At first look it was ok, up and running but I see that I have to manual start service```
dnsbl pfBlockerNG DNSBL Web Server

This service not running after update is something that I had before when beta testing on all boxes.

ntct

I also update from dev to 2.0, it work properly, a powerful package

Thanks! BBcan177

wcrowder

@cmb:

Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

Done, ran the standard tests, all good. Thanks for all the work BB!

pfcode

Work like a charm, Thanks! For some reason, First attempt stuck at the 'Downloading MaxMind databases' part. Second attempt worked.

vito

@pfcode:

Work like a charm, Thanks! For some reason, First attempt stuck at the 'Downloading MaxMind databases' part. Second attempt worked.

I had this also, second time worked fine
prior, i was on the latest build of PFB 1.x

BBcan177

Looks like I have a file caching issue with the download function. I will get that resolved asap and add the relevant download code directly to the install code.

Bastion

Installed and configured with DNSBL, all went smoothly and working flawlessly since.

Great package by great people ! Many thanks BBcan177

Wolf666

After months of testing I just installed pfBlockerNG 2.0 from official repository, everything is ok.
Thanks BBCan, ottimo lavoro amico mio!

JasonJoel

@pfcode:

For some reason, First attempt stuck at the 'Downloading MaxMind databases' part. Second attempt worked.

Ditto. Worked on 2nd try.

I setup DNSBL per the instructions above, and it looks like everything is worked as advertised. Great work!

Jason

Ibor Daru

Excellent!!!

Confirmed also: second try (simply reinstalled package after first update attempt)

Really great job. Where can I donate?

trinidadrancheria

Installed, It stopped at the same place as most others, being lazy I refreshed which reinstalled it again :D (3 less clicks than packages/reinstall/confirm) :P

Working GREAT! Gonna enable DNSBL for INBOUND Monday (using Alias) :)

BBcan177

Here is a PR to fix the Installation issue being reported…

https://github.com/pfsense/pfsense-packages/pull/1189

Once this is merged, if anyone has similar issues, please report back. Thanks!

fraglord

If you move your configuration from DNS Forwarder to DNS Resolver in order to use pfBlockerNG: For multi-WAN configurations you have to have forwarding mode enabled. See also here: https://doc.pfsense.org/index.php/Unbound_DNS_Resolver

Hugovsky

Thanks BBcan177, and everyone that made this package possible. Great jpb.

@fraglord:

If you move your configuration from DNS Forwarder to DNS Resolver in order to use pfBlockerNG: For multi-WAN configurations you have to have forwarding mode enabled. See also here: https://doc.pfsense.org/index.php/Unbound_DNS_Resolver

Can you elaborate? I have multi-wan and forwarding disabled and everything works.

cmb

Latest PR merged.

Dpain

@BBcan177:

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

Huge thanks BBcan177!!! First off really looking forward to you posting more feeds, more importantly though where can I donate? I'd like to thank you, you have no idea I've been trying to block ads on my wireless devices and had been trying to do this on an Asus RT-68U (AP mode) with both tomato and merlin and was having issues. PfBlockerNG 2.0 has solved this issue for me, I no longer have to mess around with the Asus router and don't have to worry bout getting ads on my sons ipad/pc. I added a feed from http://winhelp2002.mvps.org/hosts.htm would this work for host files as well? In the log it does seem to have downloaded the .txt file (see attached screenshot) Thanks again BBcan177 for your work!

BBcan177

Here are more DNSBL Feeds that can be used in pfBlockerNG.
(Copy and paste URLS as plain text)

Create a new alias for these.
These are not necessarily ADvert domains. So I named mine "Malicious"

hpHosts
http://hosts-file.net/download/hosts.zip

SWC
http://someonewhocares.org/hosts/hosts

spam404
~~https://spam404bl.com/blacklist.txt~~
https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt

malc0de
https://malc0de.com/bl/BOOT

MDS (use 'Flex' state)
https://mirror1.malwaredomains.com/files/justdomains

MVPS
http://winhelp2002.mvps.org/hosts.txt

MDL
http://www.malwaredomainlist.com/hostslist/hosts.txt

Discontinued
GJTech
http://adblock.gjtech.net/?format=unix-hosts

dShield_SD (They also have a conservative list available)
https://www.dshield.org/feeds/suspiciousdomains_High.txt

Zeus
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist

These two feeds post full URLs, so there can be some more false positives.
Create a new Alias, and use Alexa as a recommendation.

PhishTank
https://data.phishtank.com/data/online-valid.csv.bz2

OpenPhish
https://www.openphish.com/feed.txt

MPatrol (You need to register - Free or Paid subscription. Use Danguardian feed)
https://lists.malwarepatrol.net

This is a feed that I manage (as time permits)
MS_2
https://gist.githubusercontent.com/BBcan177/4a8bf37c131be4803cb2/raw
Use this in its own Alias:

BBC_DGA (This is a large feed of DGA for the likes of Cryptolocker et al…)
http://osint.bambenekconsulting.com/feeds/dga-feed.gz

BBC_C2
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt

Use this feed in its own alias as it is updated more frequently.
So you can update it more often than once per day.

hpHosts_partial
http://hosts-file.net/hphosts-partial.asp

If users find other feeds, please post back so that others may benefit also.
Its also important to donate to the feeds provider (IP and/or Domain) as they all need support.

tonymorella

@BBcan177:

Here are some basic instructions to get started with DNSBL.

Open the pfBNG "DNSBL" Tab:

(Use the defaults unless you have a need to use otherwise)
Enter the DNSBL VIP as 10.10.10.1
Enter the DNSBL Listening Port as 8081
Enter the DNSBL SSL Listening port as 8443
Select the DNSBL Listening Interface as Lan

For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

DNSBL IP Firewall Rule Settings:
Select Deny outbound or as per your requirements
select Enable logging

Alexa (is optional, you can skip this until later if you wish)
Select Top 1K
Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

In the Custom List you may enter any domain you wish to Whitelist.

Save your settings

Open the "DNSBL Feeds" Tab:

Create a new DNSBL Alias

Enter DNS Group Name as ADs
Enter Description as DNSBL ADverts

DNSBL:

Enter the Header/Label and Source URL as follows:
(Use copy/paste as plain text for the URL)

Format Auto and State ON

yoyo
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

hpHosts_ads
http://hosts-file.net/ad_servers.txt

Adaway
https://adaway.org/hosts.txt

Cameleon
http://sysctl.org/cameleon/hosts

Select List Action as Unbound
Select Update Frequency as Once a day

Alexa:
Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

Add any other domains that you wish to block in the Custom List.

Save your settings

Open the "DNSBL" Tab:

Click DNSBL Enable checkbox.
Save your settings

Open the "Update" Tab:

Select Force Update

You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

Goto the pfBlockerNG "Alerts" Tab:

Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

As a test, goto www.aol.com and www.yahoo.com and see some alerts populate.

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

Have been using the beta code for some time and noticed a list of domains in the Alexa top 1K that serve up ads, since I don't want this I added the following custom block list to my Ads DNS Group which will remove them from any DNS group that is using the Alexa top 1K filter.

popcash.net
www.popcash.net
outbrain.com
www.outbrain.com
onclickads.net
www.onclickads.net
googleadservices.com
www.googleadservices.com
adcash.com
www.adcash.com
popads.net
www.popads.net

To confirm make sure "Enable Alexa Whitelist" is NOT checked for this DNS group.

BBcan177

@Dpain:

@BBcan177:

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

Huge thanks BBcan177!!! First off really looking forward to you posting more feeds, more importantly though where can I donate?

Thanks again BBcan177 for your work!

I could use a lifetime pfSense Gold subscription… :) But I think all developers should get that as a bonus :)

My email is at the bottom of the pfBNG general tab.

Dpain

@BBcan177:

@Dpain:

@BBcan177:

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

Huge thanks BBcan177!!! First off really looking forward to you posting more feeds, more importantly though where can I donate?

Thanks again BBcan177 for your work!

I could use a lifetime pfSense Gold subscription… :) But I think all developers should get that as a bonus :)

My email is at the bottom of the pfBNG general tab.

:o lifetime ? That would be $99 x how many ever years you have left ;D :P so is the email there attached to a paypal account? Not sure if donation talk is allowed on forum if not, I apologize mods. Also , It does seem to be working for host files from http://winhelp2002.mvps.org/hosts.htm Thanks!

fraglord

Can you elaborate? I have multi-wan and forwarding disabled and everything works.

If you enable forwarding mode then a foward-zone named "." is created as well as some forward-addr entries with the DNS servers you sepcified system->general setup.
Default behavior of unbound is to query the Root servers. But if you have a multi-WAN configuration and need to specifiy a different DNS server for each WAN(gateway) you need forwarding mode to be enabled. So yeah, works without forwarding mode for some multi-WAN configs that are fine with just the Root servers.