TLD blacklist, exclusion and whitelist

AWeidner · Apr 9, 2021, 6:33 PM

Prerequisites:
pfSense 2.5.0, pfblockerng 2.1.4_25, both freshly installed.

What i want to do:
block the entire TLD 'ru' with exception for "www.gov.spb.ru"

What i did:

enabled TLD in DNSBL
added 'ru' to TLD blacklist
force reload
check that all sites within '.ru' are blocked: it works
added

gov.spb.ru|46.243.177.41
www.gov.spb.ru|46.243.177.41

to TLD whitelist

force reload: www.gov.spb.ru is still blocked
added spb.ru, gov.spb.ru and www.gov.spb.ru to TLD exclusion list
force reload: site is still blocked

Restarting pfSense did not help.

What am i doing wrong? These are the only settings i made so far.

RonpfS · Apr 9, 2021, 6:35 PM

@aweidner said in TLD blacklist, exclusion and whitelist:

'ru' to TLD blacklist

From the DNSBL Settings page :

When the 'python Blocking mode' feature is enabled. The TLD Whitelist is not utilized and instead uses the DNSBL Whitelist.

fireodo · Apr 9, 2021, 6:46 PM

@ronpfs said in TLD blacklist, exclusion and whitelist:

@aweidner said in TLD blacklist, exclusion and whitelist:

When the 'python Blocking mode' feature is enabled. The TLD Whitelist is not utilized and instead uses the DNSBL Whitelist.

pfblockerng 2.1.4_25 - has a Python mode?

BBcan177 · Apr 9, 2021, 6:58 PM

@aweidner

With Unbound Python mode, you need to use the TLD Blacklist with the TLD Whitelist.

In Unbound Python Mode, you can use the TLD Blacklist, and then just whitelist the sites with the DNSBL Whitelist as all other whitelisting.

If you stay with Unbound Mode:

You need to add each variant of the domains to the TLD Whitelist.
So both www.example.com and example.com
Keep in mind, that you don't need to add the "|x.x.x.x" IP as it will now collect the IP each cron run to ensure it stays up to date.

AWeidner · Apr 9, 2021, 8:01 PM

@bbcan177 said in TLD blacklist, exclusion and whitelist:

@aweidner

With Unbound Python mode, you need to use the TLD Blacklist with the TLD Whitelist.

In Unbound Python Mode, you can use the TLD Blacklist, and then just whitelist the sites with the DNSBL Whitelist as all other whitelisting.

Where is the difference between "With Unbound Python mode" and In "Unbound Python Mode"?!

If you stay with Unbound Mode:

You need to add each variant of the domains to the TLD Whitelist.

So both www.example.com and example.com

Keep in mind, that you don't need to add the "|x.x.x.x" IP as it will now collect the IP each cron run to ensure it stays up to date.

For a new-to-pfblockerng-person like me this sounds unnecessary unintuitive. If there is a blacklist, why can't i use the whitelist in the same realm?

RonpfS · Apr 9, 2021, 8:09 PM

@aweidner said in TLD blacklist, exclusion and whitelist:

Where is the difference between "With Unbound Python mode" and In "Unbound Python Mode"?!

A typo? maybe In "Unbound Mode" you can use the TLD Blacklist, ?

BBcan177 · Apr 9, 2021, 8:19 PM

@aweidner said in [TLD blacklist, exclusion and whitelist]

For a new-to-pfblockerng-person like me this sounds unnecessary unintuitive. If there is a blacklist, why can't i use the whitelist in the same realm?

Unbound mode is the original/first generation of DNSBL. It relies on Unbound local-zone and local-data entries to block domains.

TLDR;
Unbound Mode is too restrictive in wildcard blocking. Unbound Python mode (the next generation of DNSBL) doesn't use Unbound's local-zone/local-data entries, and removes all these types of restrictions. Therefore, you can use the DNSBL Whitelist feature with Python mode in this circumstance.

Existing pfBlockerNG DNSBL Unbound mode:

In this mode, entries are added to Unbound with an include file located at:

    /var/unbound/pfb_dnsbl.conf

For each single domain to be blocked, a local-data entry is added:

    local-data: "example.com A 10.10.10.1"

Any DNS request for example.com would be sinkholed to 10.10.10.1 where the DNSBL Webserver will respond to that request.

If the TLD Wildcard blocking option is enabled, entries will be added to Wildcard block domains:

    local-zone: "xyz" "transparent"
    local-data: "example.xyz 60 IN A 10.10.10.1"
    local-data: "example2.xyz 60 IN A 10.10.10.1"

For each TLD (ie: xyz), a transparent zone is created, and any xyz domains that need to be wildcard blocked are added to the associated transparent zone entry with local-data entries.

When users want to Wildcard block a whole TLD, TLDs would be added to the TLD Blacklist with a local-zone redirect entry:

    local-zone: "ru" redirect local-data: "ru 60 IN A 10.10.10.1"

If "ru" is wilcard blocked, and a user wants to allow a specific ru domain to be allowed, domains can be added to the

TLD Whitelist. The entries in Unbound will use a local-zone static entry to block all domains except for domains that are
explicitly defined with a local-date entry;

    local-zone: "ru" "static"
    local-data: "ramblr.ru A 31.31.205.163"

AWeidner · Apr 12, 2021, 2:17 PM

@bbcan177
I appreciate your effort in explaining things, but i am still lost.

What needs to be done in order to achive the following:

Block everything with the TLD '.ru'
Allow access to 'gov.spb.ru' and 'www.gov.spb.ru'

Preferrably via the UI and not having to edit files form the command line. It can't be that hard...

Still on pfSense 2.5.0 and pfBlockerNG 2.1.4_25 with default settings.

BBcan177 · Apr 12, 2021, 2:36 PM

@aweidner
Do you see any errors in the pfblockerng.log when its performing the TLD Analysis?

If you ping those domains from pfSense and from a LAN device do you get the IP, or the DNSBL VIP Sinkhole address?

Are you blocking via GeoIP also?

AWeidner · Apr 12, 2021, 3:13 PM

@bbcan177
No, i did not use GeoIP.

I just reset the pfSense to Factory defaults (5 minutes ago)
Installed the package pfBlockerNG 2.1.4_25
Enabled pfBlockerNG, DNSBL and TLD
Added 'ru' to TLD Blacklist
Ran Update

 UPDATE PROCESS START [ 04/12/21 17:08:26 ]

===[  DNSBL Process  ]================================================

  DNSBL: Flush DNSBL_IP
Clearing all DNSBL Feeds...  completed
Executing TLD
 Blocking full TLD/Sub-Domain(s)... |ru| completed
TLD analysis completed
Finalizing TLD... head: 1: No such file or directory
tail: 1: No such file or directory
 completed
 ----------------------------------------
 Original    Matches    Removed    Final     
 ----------------------------------------
 0           0          -1         1         
 -----------------------------------------
Validating database... completed

DNSBL enabled FAIL - restoring Unbound conf
/var/unbound/pfb_dnsbl.conf:1: error: unknown keyword '.ru'
/var/unbound/pfb_dnsbl.conf:1: error: unknown keyword '60'
read /var/unbound/unbound.tmp failed: 2 errors in configuration file

===[  Continent Process  ]============================================


===[  Aliastables / Rules  ]==========================================

No changes to Firewall rules, skipping Filter Reload
No Changes to Aliases, Skipping pfctl Update

 UPDATE PROCESS ENDED

Now it does not even block anything with the TLD '.ru'

The file /var/unbound/pfb_dnsbl.conf is empty (0 bytes).

BBcan177 · Apr 12, 2021, 3:20 PM

@aweidner
In TLD Blacklist, you can't prefix with a ".", just add "ru".
Click on the blue infoblock icons for more details.

BBcan177 · Apr 12, 2021, 3:21 PM

Would also recommend installing pfBlockerNG-devel instead.

AWeidner · Apr 13, 2021, 8:02 AM

@bbcan177 said in TLD blacklist, exclusion and whitelist:

@aweidner
In TLD Blacklist, you can't prefix with a ".", just add "ru".
Click on the blue infoblock icons for more details.

It is without the Dot in the blacklist.

I don't want to use developement software in a production environment.

How something this simple seems to be so complicated is really frustrating.

I just need a simple answer:
How do i block everything russian with the exception of one site? Is it possible with the plain pfblockerNG?

BBcan177 · Apr 13, 2021, 6:03 PM

@aweidner said in TLD blacklist, exclusion and whitelist:

I don't want to use developement software in a production environment.
How something this simple seems to be so complicated is really frustrating.
I just need a simple answer:
How do i block everything russian with the exception of one site? Is it possible with the plain pfblockerNG?

I have reviewed this configuration, and its a scenario that I have not tested. In essence, you don't have any DNSBL Feeds enabled, but are only using the TLD Blacklist/TLD Whitelist functionality.

So in this case, pfBlockerNG will not work as-is.

If you are still reading .....

The actual TLD to be blocked is not "ru", but actually "spb.ru". So this TLD would need to be used in TLD Blacklist. You can add both "ru" and "spb.ru"
See DNSBL TLD Reference.
You would need to add a DNSBL Group with atleast one Feed defined, or add a single domain to be blocked in the DNSBL Customlist (could be any ru domain).
There is a single line of code that needs to be changed to resolve an issue with the TLD processing. (This is fixed already in pfBlockerNG-devel)

Reference:
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.sh#L449

File:
/usr/local/pkg/pfblockerng/pfblockerng.sh

Line: #449

From:

dnsbl_tmp_files="$(grep -c ^ ${dnsbl_tmp}DNSBL_*.txt | sort -t : -k 2,2n | cut -d':' -f1)"

To:

dnsbl_tmp_files="$(grep -Hc ^ ${dnsbl_tmp}DNSBL_*.txt | sort -t : -k 2,2n | cut -d':' -f1)"

BBcan177 · Apr 13, 2021, 6:00 PM

A PR has been submitted to the pfSense Devs for review and approval to resolve that line of code:

pfBlockerNG v2.1.4_26
https://github.com/pfsense/FreeBSD-ports/pull/1062

AWeidner · Apr 16, 2021, 10:42 AM

@bbcan177
Thank you for your patience. I just could not imagine it being so hard to achieve this.

I have some experience with Squid, where URL blocking/whitelisting is relatively easy. But i want to migrate away from it and pfBlockerNG seemed like a good alternative.