PfBlockerNG v2.0 w/DNSBL

BBcan177

pfBlockerNG v2.0

This release brings several improvements and enhancements to the existing pfBlockerNG IP Download manager capabilities, and also introduces domain name blocking ("DNSBL") via Unbound DNS Resolver. Domain blocking can be used for ADvert servers, malicious domains and/or domain filtering.

DNSBL overview: When a DNS request is made for a domain that is listed in DNSBL, the request is redirected to a Virtual IP address where an instance of Lighttpd Web Server will collect the packet statistics and push a '1x1' GIF image to the Browser. Rejected domain activity can be viewed in the "Alerts" tab along with the IP Alerts.

Due to the increased security measures in Browsers, HTTPS alerts will not display the SRC IP or Requested URL, however, the Requested Domain name will be logged to the Alerts tab along with all HTTP alerts.

To reduce False Positives, the "Alexa" Top 1M sites can be used to suppress the most popular sites. Settings allow for the number of Top sites and which TLDs (.com, .org etc) to include in the Alexa Suppression list.

Improved Features

[ [color=red]cURL ]

PHP cURL library is now used to download all Lists/Feeds/Maxmind/Alexa files. This allows for increased security settings and collection of HTTP response codes. Allowed SSL Ciphers (TLSv1.2 and TLSv1) and default settings (SSL_VERIFYPEER, SSL_VERIFYHOST).

Added a "Flex" option to allow for the downgrade of the SSL connection, if required. The "Flex" option (when enabled) will downgrade on the following HTTP (RFC7231) response codes:

35 - GET_SERVER_HELLO:sslv3
51 - NO alternative certificate
60 - Local Issuer Certificate Subject

The following settings will be downgraded:

disable verifypeer
disable verifyhost
adds SSLv3 SSL Cipher.

(It is not recommended to use the "Flex" setting; however not all Lists/Feeds support the increased security measures. Expect overtime that these Lists/Feeds will improve their security.)

[ [color=red]Alerts Tab ]

The Alerts tab has been refactored. Improvements in efficiency and accuracy in the IP/CIDR for the reported Alerts. The Suppression Icon is now only visible for /32 and /24 CIDRS, all other suppression's require an 'Outbound' allow firewall rule to circumvent the blocklist. Hostnames for local devices are now displayed under each Alerted IP. When 'Filter options' are used, they will be preserved on page refreshes.

[ [color=red]Reputation ]

All reputation functions have been refactored and improvements made to the overall efficiency.

[[color=red] IP Parser ]

The IP parser has been converted to use String functions vs regex functions, for a significant improvement in efficiency.

[ [color=red]List/Feed Format ]

The package will now "automagically" determine what 'Format' is required to extract and parse any given URL.

[ [color=red]Firewall Rules ]

The "Firewall Rule" creation function has been improved. Please verify your firewall rules post installation to ensure they are as expected.

[ [color=red]Config Save ]

Only when changes are made to the settings, will pfBlockerNG save its settings to the pfSense config file. This will reduce the spammed backups that were being created at each hourly CRON task interval.

[ [color=red]XMLRPC ]

XMLRPC sync has been improved to include support for hostnames and IPv6 addresses.

[ [color=red]XSS potential ]

Fixed XSS potentials in several functions of the code.

New Features

[ [color=red]Whois ]

A whois feature has been added to convert an AS number into its respective IP Addresses. Entering a Domain name will also collect its host IP Address.

[ [color=red]CIDR Aggregation ]

Aggregation will optimize contiguous network prefixes in a List, and remove any superfluous prefixes which are already included in another prefix. Depending on the hardware used, Aggregation can use a significant amount of resources if the list being aggregated is large.

[ [color=red]Max daily download failure attempts ]

Allows for the configuration of a "max daily download failure threshold" for the CRON updates. If this setting is defined, attempts will be made to re-download up-to the threshold defined. The 'failed download' counters can be clearing in the widget.

[ [color=red]Restore previous download on failure ]

When enabled, the previously downloaded file contents are used on a download failure. (Defaulted to 'on')

[[color=red] Kill States ]

After a cron event or any 'Force' commands (when enabled), any IPs listed in any Blocklists; if found in the Firewall states will be cleared and logged. States removal can be bypassed on an alias by alias basis if required.

[[color=red] Tracker IDs ]

The pfBlockerNG Firewall Rules now include a 'Tracker ID', this ID is based upon the Alias name (Sum of the ASCII characters ORD values). This ID is a 10 digit number, prefixed with '177'. This ID will remain static unless the Alias name is changed. If the Firewall Logs are pushed to a remote syslog, you can use the Tracker ID as a reference to the pfBlockerNG Alias.

Further details are available in the following forum thread:
https://forum.pfsense.org/index.php?topic=86212.0

q54e3w

awesome job, thanks BBCan!

mygeeknc

Great work sir! I'm looking forward to this release. Will there be any special instructions for upgrading?

trinidadrancheria

Guess I will hold off manually editing the INC file then to add DNSBL :)
Any ETA on this new version?

THANKS!!!

pfcode

Great. Thanks!

wcrowder

ETA is as soon as the Dev's validate the package, package has been submitted… Yeah!!!!! :)

https://github.com/pfsense/pfsense-packages/pull/1174

trinidadrancheria

@wcrowder:

ETA is as soon as the Dev's validate the package, package has been submitted… Yeah!!!!! :)

https://github.com/pfsense/pfsense-packages/pull/1174

HOLY Poop!!! 23 changed files with 8,591 additions and 5,267 deletions Total re-write? :D

Panja

Looking forward to this release!
Keep up the good work mate. Awesome! 8)

BBcan177

@mygeeknc:

Great work sir! I'm looking forward to this release. Will there be any special instructions for upgrading?

There should be no concerns in migrating to pfBNG v2.0 from the existing v1.10.

pfSense has two types of DNS Services

DNS Forwarder
DNS Resolver

If you plan on using the DNSBL feature, you will need to use the DNS Resolver for your DNS queries, the DNS Forwarder is not an option for DNSBL. Its probably best to ensure that the DNS Resolver is working before using DNSBL.

The DNS Resolver is developed by NLnet Labs and is named 'Unbound'. It is a validating, recursive and caching DNS resolver. https://www.unbound.net/index.html

Some recommendations:

The DNS Resolver can also be used in 'Forwardering mode'; however its best to not use this 'Forwarding mode' and keep it in 'resolver mode' as this will query the Root DNS servers for the DNS queries instead of relying on an ISPs DNS etc…
If you use the 'DNS Resolver Forwarder mode', only configure 'DNSSEC' if the configured DNS servers support DNSSEC. The enabling of 'DNSSEC' to harden your DNS security is highly recommended.
Disable the two "DHCP registrations" checkboxes, unless you really require those options.

Here is a good primer about the DNS Resolver (Unbound) https://calomel.org/unbound_dns.html

gsiemon

@BBcan177:

If you plan on using the DNSBL feature, you will need to use the DNS Resolver for your DNS queries, the DNS Forwarder is not an option for DNSBL. Its probably best to ensure that the

I'm using the Bind package instead of Unbound. Will V2.0 support that or is it hard coded to use only Unbound? If it only supports Unbound, is it possible to support Bind in the future?

doktornotor

@gsiemon:

I'm using the Bind package instead of Unbound. Will V2.0 support that

No.

fraglord

You doin a great job! I appreciate it and many others do so as well. It is a leightweight alternative to squid when it comes to ad-blocking.
I was allowed ot participate in the beta test program and the DNSBL feature is my favourite. Minor issues have been ironed out with the latest upgdates so it for sure feels ready to be released.

a-a-ron

@fraglord:

You doin a great job! I appreciate it and many others do so as well. It is a leightweight alternative to squid when it comes to ad-blocking.
I was allowed ot participate in the beta test program and the DNSBL feature is my favourite. Minor issues have been ironed out with the latest upgdates so it for sure feels ready to be released.

Same here, it was a pleasure to help BBcan177. Hopefully I wasn't too much of a pain :D

musicwizard

yup the beta one works really nice :)
Running my own DNS server and not from my ISP/google/ etc
so dnsbl is really nice on it also. Keep it up BBcan

cmb

This has been merged, it'll be live in the package repo within at most ~15 minutes of the time of this post.

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance. If there are issues, BBCan, please follow up with pull requests and I'll make sure one of us gets them merged promptly. Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

doktornotor

@cmb:

Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Done, all works pretty well (as before). :D

BBcan177

@cmb:

This has been merged, it'll be live in the package repo within at most ~15 minutes of the time of this post.

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance. If there are issues, BBCan, please follow up with pull requests and I'll make sure one of us gets them merged promptly. Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

Thanks Chris! :)

hda

Package update succeeded on the second reinstall from v1.10, after ~11 min. on an Alix pfSense 2.2.5 (ro). Thank you BBcan :)

Update: I followed BBcan's instructions below for starting DNSBL and that worked too. Excellent extension.

BBcan177

Here are some basic instructions to get started with DNSBL.

Open the pfBNG "DNSBL" Tab:

(Use the defaults unless you have a need to use otherwise)
Enter the DNSBL VIP as 10.10.10.1
Enter the DNSBL Listening Port as 8081
Enter the DNSBL SSL Listening port as 8443
Select the DNSBL Listening Interface as Lan

For the DNSBL Firewall Rule select all of the LAN subnets that access the DNS Resolver.
Ensure that all Devices that use the DNS Resolver, have the Resolver as its only DNS setting for DNSBL to function properly.

DNSBL IP Firewall Rule Settings:
Select Deny outbound or as per your requirements
select Enable logging

Alexa (is optional, you can skip this until later if you wish)
Select Top 1K
Select the TLD Inclusions as ca,co,com,io,me,net,org or as required.

In the Custom List you may enter any domain you wish to Whitelist.

Save your settings

Open the "DNSBL Feeds" Tab:

Create a new DNSBL Alias

Enter DNS Group Name as ADs
Enter Description as DNSBL ADverts

DNSBL:

Enter the Header/Label and Source URL as follows:
(Use copy/paste as plain text for the URL)

Format Auto and State ON

yoyo
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext

hpHosts_ads
http://hosts-file.net/ad_servers.txt

Adaway
https://adaway.org/hosts.txt

Cameleon
http://sysctl.org/cameleon/hosts

Select List Action as Unbound
Select Update Frequency as Once a day

Alexa:
Do not enable the Alexa Whitelist for this ADverts based alias, as Alexa also posts the top ADvert servers. So using Alexa whitelist, will interfere with ADvert Blocking.

Add any other domains that you wish to block in the Custom List.

Save your settings

Open the "DNSBL" Tab:

Click DNSBL Enable checkbox.
Save your settings

Open the "Update" Tab:

Select Force Update

You should now see the DNSBL Feeds being downloaded and parsed. Once that is complete, goto the Dashboard, and confirm that the widget is populated correctly.

Goto the pfBlockerNG "Alerts" Tab:

Any domain that is blocked will be reported here. For HTTPS alerts, the SRC IP and URL are not captured due to Browser security measures.

As a test, goto www.aol.com and www.yahoo.com and see some alerts populate.

There are several other DNSBL Feeds that can be used with pfBNG DNSBL. I will post that at a later date, once users get their basic configurations working. There is also an ADBlock Easylist tab, which is pretty self-explanatory.

n3by

@cmb:

…

I largely relied on all your reports here for testing, only checking that the basics function and there isn't anything malicious, or grossly wrong at a glance.
...
Those who were beta testing, it'd help if you can reinstall latest from the standard package repo and report back.

Thanks BBCan!

uninstalled test 77 DEV and installed 2.0 from official on 2 boxes v2.2.4 & v2.2.5

At first look it was ok, up and running but I see that I have to manual start service```
dnsbl pfBlockerNG DNSBL Web Server

This service not running after update is something that I had before when beta testing on all boxes.