pfBlockerNG not blocking everything in blocklists

vjizzle · Apr 4, 2021, 1:26 PM

Hi,

I am running the latest pfBlockerNG on pFsense but the widget in the dashboard always showed a low blocked percentage. In my mind that percentage should be a lot higher! So this weekend I took some time and setup AdGuard Home on a Raspberry Pi and did some testing.

After running for about 24 hours the AdGuard Home Raspberry Pi showed a blocked percentage of roughly 11% while pfBlockerNG did not manage to ever go over 2%! I started digging and soon found out that pfBlockerNG is not blocking all the domains altho they are in the blocklist. I am using the same blocklists on Adguard Home as in pfBlockerNG. pfBlockerNG is running in python mode and I am using Ubound resolver mode on pfSense. My Adguard Home Raspberry Pi is also running local Unbound Server.

One domain for testing: www.conduit.com. This domain is in the oisd.nl blocklist. AdGuard blocks this flawlessly (resolves to 0.0.0.0) while pfBlockerNG just resolves it fine. So now I suspect that pfBlockerNG is somehow missing even more and this is why it is showing the low blocked percentage. I love pfBlockerNG but now it seems like it is giving me a false feeling of security and privacy :|.

My pfSense is version 2.5 has 8GB of RAM, local SSD and is a core i5 running 3Ghz. Ram usage is about 25% and CPU is always around 2%. I don't us the TLD option in pfBlockerNG but I don't believe that this will resolve the behavior I am seeing. I already tried several reboots of pfSense and also completely reinstalled pfBlockerNG without the "Keep settings" option. I remember that the low percentage was also the case when I had pfSense 2.4.5 p1. The blocked percentage then just never was above 2%. Back then I didn't investigate it because I was waiting for pfSense 2.5. All help is welcome!

--edit: typos

Gertjan · Apr 5, 2021, 8:42 AM

I loaded the feed oisd.nl :

🔒 Log in to view

Test :

🔒 Log in to view

Also :

🔒 Log in to view

edit :

2% blocked isn't a bad number.

After all, you won't need to block what so ever if you stopped visiting BS sites ^^

My Alerts represent just the usual crap :

🔒 Log in to view

vjizzle · Apr 5, 2021, 1:49 PM

Hi Gertjan,

Thank you for taking your time and doing some tests. The reason why I stumbled upon www.conduit.com was because I was using GRC's DNS Benchmark tool for testing pfSense with pfBlockerNG and AdGuard Home. Going trough the log file I was surprised that www.conduit.com did show up in the AdGuard Home log file but not in pfBlockerNG. That's when I found out that pfBlockerNG was resolving that domain fine while AdGuard Home blocked it properly. It is not that I am willingly visiting BS sites ^^.

Just to make sure that I was not doing something stupid I did some tests again and found another domain which shows the same behavior. pfBlockerNG is allowing this domain while AdGuard Home is properly blocking it. The domain is in the oisd.nl blocklist. For some reason I am not able to upload images so... :|.

Below is the AdGuard Home Server:

server 192.168.150.10
Default Server: adguard.vXXXXXX.lan
Address: 192.168.150.10

www.ku6.com
Server: adguard.vXXXXXXX.lan
Address: 192.168.150.10

Non-authoritative answer:
Name: www.ku6.com
Address: 0.0.0.0

Here is my pfSense with pfBlockerNG:

server 192.168.100.1
Default Server: odin.vXXXXXXX.lan
Address: 192.168.100.1

www.ku6.com
Server: odin.vXXXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: 3a1d905b3d90ef4e-cl2.qcloudcjgj.com
Address: 118.89.206.86
Aliases: www.ku6.com
fa2a686c758fa3d110164df993a0b8c4.qcloudzygj.com

I can see www.ku6.com in the logs of AdGuard home while it is not showing up in pfBlockerNG reports. That is of course correct because it is not blocking that response. I checked that CNAME Validation is turned on in pfBlockerNG.

Gertjan · Apr 5, 2021, 2:22 PM

@vjizzle :

C:\Users\gwkro>nslookup www.ku6.com
Serveur :   pfsense.brit-hotel-fumel.net
Address:  2001:470:1f13:5c0:2::1

Nom :    www.ku6.com
Address:  0.0.0.0

@vjizzle said in pfBlockerNG not blocking everything in blocklists:

It is not that I am willingly visiting BS sites ^^.

I know ^^.

Btw : the feed "oisd.nl" works fine for me.
One of your settings is doing what you see.
As, if we use the same code (pfBlockerNG) and the same feed (oisd.nl) only our local setting differ.

These are my "Firewall > pfBlockerNG > DNSBL" settings :

🔒 Log in to view

You already saw "Firewall > pfBlockerNG > DNSBL > DNSBL Groups".

Here are the settings for the feed :

🔒 Log in to view

When you force update, all looks well ?

192.168.100.1 == pfSense (== resolver), right ?

vjizzle · Apr 5, 2021, 3:13 PM

@gertjan
Thank you again for your help! Thanks for sharing your settings for DNSBL, mine are same without the IPV6 options and I am not using the TOP1M lists.

Yes 192.168.100.1 is my pfSense with Unbound running in resolver mode. I think I found something.

When I load the lists in pfBlockerNG I can see that it is not blocking the domain www.ku6.com, www.conduit.com or www.vmn.net. Those are domains which are being triggered when using GRC's DNS testing tool. BUT I got pfSense working briefly as I wanted when I restarted Unbound manually after a force reload of pfBlockerNG! But then after cron of pfBlockerNG runs I and I check those domains again...pfBlockerNG is allowing them again.

So I think that there is something wrong with my unbound - pfBlockerNG config.

See:

Before Ubound restart:

www.conduit.com
Server: odin.vXXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: www.va.conduit.com
Address: 198.49.162.233
Aliases: www.conduit.com

www.vmn.net
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: www.vmn.net
Address: 54.146.142.3

And then I do the restart of Unbound:

www.conduit.com
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Name: www.conduit.com
Address: 0.0.0.0

www.vmn.net
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Name: www.vmn.net
Address: 0.0.0.0

www.ku6.com
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Name: www.ku6.com
Address: 0.0.0.0

After a reload of pfBlockerNG:

www.ku6.com
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: 3a1d905b3d90ef4e-cl2.qcloudcjgj.com
Address: 118.89.206.86
Aliases: www.ku6.com
fa2a686c758fa3d110164df993a0b8c4.qcloudzygj.com

www.vmn.net
Server: odin.vXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: www.vmn.net
Address: 54.146.142.3

www.conduit.com
Server: odin.vXXXXXXX.lan
Address: 192.168.100.1

Non-authoritative answer:
Name: www.va.conduit.com
Address: 198.49.162.233
Aliases: www.conduit.com

It looks like something is broken on my end of pfSense and pfBlockerNG but can't imagine what. Maybe it is time to try a clean install. Thank you for troubleshooting with me :)

Gertjan · Apr 6, 2021, 6:05 AM

You can do some more checking.

Locate the file /var/unbound/pfb_py_data.txt - load it into, for example, notepad++.
(a SSH access and SFTP with WinSCP comes in handy here)

rw-r--r--  1 root  unbound  63782900 Apr  5 10:29 /var/unbound/pfb_py_data.txt

A 63 Megabytes file (and I'm just using a couple of feeds).

When the oisd.nl feed file is loaded and parsed in, you'll find lines in this file like

....
,ku6.com,,0,OISD,DNSBL_Compilation
...
,www.ku6.com,,0,OISD,DNSBL_Compilation
....
etc.

....

unbound, when it receive a DNS lookup request, first looks in the local cache. Part of the lookup is a call to the python script written for pfblockerNG. And this script uses /var/unbound/pfb_py_data.txt as it's local lookup table.
If a host name is found in that file, it will be blocked.

I tend to say that when "ku6.com" resolves, it isn't present in var/unbound/pfb_py_data.txt or : the feed didn't get loaded.

vjizzle · Apr 6, 2021, 9:55 AM

@gertjan

I have checked the files and see the screenshots below:

🔒 Log in to view

Then I checked the entries for the domains in the file. They are there:

🔒 Log in to view

And then when I check with nslookup pfBlockerNG is not blocking them:

🔒 Log in to view

I think something seriously is broken and I hope that it is only on my end. I am planning to do a clean install tomorrow and restore my backup and hopefully that will fix this.

vjizzle · Apr 6, 2021, 10:13 AM

@vjizzle

I can confirm that reloading (force or normal) of pfBlockerNG does not solve this problem. However (like said before) when I restart unbound service it start working properly and pfBlockerNG blocks those domains.

After restarting Unbound I go back to force reload pfblockerNG (to test) and it keeps working.

After that I hit pfSense again with GRC DNS benchmark tool. This tool generates a LOT of DNS requests. After running the benchmark tool pfBlockerNG is broken and pfSense unbound resolves those blocked domains. So it look like when you hit pfSense Unbound with a lot of DNS requests something breaks under the load and pfBlockerNG is not working anymore with unbound. The pfb_py_data.txt is still there containing the entries from the blocked feeds. Force reloading pfBlockerNG does nog fix the problem at this point. I have to restart Unbound service manually again fixes the problem :|

@Gertjan : can you test your side also with the GRC DNS Benchmark tool and see of pfsense with pfblocker hold up? I would appreciate it!

Gertjan · Apr 6, 2021, 10:28 AM

@vjizzle

Actually, there is nothing to restart.
Check the unbound config :

[2.5.0-RELEASE][root@mypfsense.net]/root: tail -n 3 /var/unbound/unbound.conf
# Python Module
python:
python-script: pfb_unbound.py

unbound works - and uses the pfb_unbound.py script file to do some extra work.

Btw :

tail -f /var/unbound/var/log/pfblockerng/dns_reply.log

shows the progress during DNS benchmark test. There are not that many DNS requests, a couple per seconds, and that is not much.

I've got another test for you :

cat /var/log/resolver.log | grep 'start'

How often does your unbound restart ? per hour / per day ?

vjizzle · Apr 6, 2021, 10:52 AM

@gertjan

Here are the restarts from Unbound. I did some testing and restarting but nothing I think it looks good:

Apr  5 16:58:59 odin unbound[81926]: [81926:0] info: start of service (unbound 1.13.1).
Apr  5 17:15:14 odin unbound[70024]: [70024:0] info: start of service (unbound 1.13.1).
Apr  5 17:16:22 odin unbound[57071]: [57071:0] info: start of service (unbound 1.13.1).
Apr  6 11:38:11 odin unbound[11398]: [11398:0] info: start of service (unbound 1.13.1).
Apr  6 11:43:22 odin unbound[51771]: [51771:0] info: start of service (unbound 1.13.1).
Apr  6 11:57:55 odin unbound[9894]: [9894:0] info: start of service (unbound 1.13.1).
Apr  6 11:58:37 odin unbound[78660]: [78660:0] info: start of service (unbound 1.13.1).
Apr  6 12:01:13 odin unbound[7024]: [7024:0] info: start of service (unbound 1.13.1).
Apr  6 12:15:23 odin unbound[78621]: [78621:0] info: start of service (unbound 1.13.1).
Apr  6 12:23:14 odin unbound[93856]: [93856:0] info: start of service (unbound 1.13.1).

pfBlockerNG is not blocking those domains now and when I tail the unbound.conf I see this:

[2.5.0-RELEASE][admin@odin.vXXXXXX.lan]/root: tail -n 3 /var/unbound/unbound.conf
# Python Module
python:
python-script: pfb_unbound.py
[2.5.0-RELEASE][admin@odin.vXXXXXX.lan]/root:

In the dns_reply.log I can see all the queries when using the GRC tool. Looks to me like a lot. I restart Unbound service to make sure that pfBlockerNG is blocking those domains. Here is the output when everything is working fine and the domain www.conduit.com is being blocked:

DNS-reply,Apr 6 12:46:02,reply,A,SOA,3600,www.conduit.com.vXXXXX.lan,192.168.100.25,SOA,unk
DNS-reply,Apr 6 12:46:02,reply,AAAA,SOA,3600,www.conduit.com.vXXXXX.lan,192.168.100.25,SOA,unk
DNS-reply,Apr 6 12:46:02,servfail,AAAA,AAAA,Unk,www.conduit.com,192.168.100.25,ServFail,unk

Then I run GRC tool and see a lot of requests going by. Now when I query www.conduit.com I get an IP in nslookup and I see the following output and pfBlockerNG is broken:

DNS-reply,Apr 6 12:48:40,cache,A,A,3442,www.conduit.com.vXXXXXX.lan,192.168.100.25,NXDOMAIN,unk
DNS-reply,Apr 6 12:48:40,reply,AAAA,SOA,3442,www.conduit.com.vXXXXXX.lan,192.168.100.25,SOA,unk
DNS-reply,Apr 6 12:48:40,cache,A,A,86385,www.conduit.com,192.168.100.25,198.49.162.233,IL
DNS-reply,Apr 6 12:48:40,servfail,AAAA,AAAA,Unk,www.conduit.com,192.168.100.25,ServFail,unk

Gertjan · Apr 6, 2021, 1:10 PM

I see what you mean :

@vjizzle said in pfBlockerNG not blocking everything in blocklists:

DNS-reply,Apr 6 12:48:40,cache,A,A,86385,www.conduit.com,192.168.100.25,198.49.162.233,IL

dono where and when unbound (?) resolves "www.conduit.com".

Check these files :
dnsbl_error.log (mine is empty)
py_error.log (mine is empty)
dnsbl_parsed_error.log (some data, not related to this issue)

vjizzle · Apr 6, 2021, 1:12 PM

@gertjan
Yes for some reason every time I run the GRC lookup tool and query my pfSense, pfblockerng is broken afterwards. I can reproduce it every time and when I restart unbound server it is fixed.

I am using unbound on pfSense in resolver mode (forwarding option is disabled) and I have no DNS server entries in General Setup tab in pfSense. I have set the option DNS Resolution Behavior to Use local DNS (ignore remote DNS servers).

py_error.log is empty.
dnsbl_error.log shows a few entries

[ DNSBL_Abuse_CH - Abuse_ch_hostfile ] Download Fail
 Firewall and/or IDS (Legacy mode only) are not blocking download.

 Restoring previously downloaded file


[ DNSBL_Abuse_CH - Abuse_ch_hostfile ] Download Fail
 Firewall and/or IDS (Legacy mode only) are not blocking download.

 Restoring previously downloaded file

dnsbl_parsed_error.log has some entries like these:

04/6/21 14:16:12,No_Tracking_Hostnames,from,:: from
04/6/21 14:16:12,No_Tracking_Hostnames,hostwinds,0.0.0.0 hostwinds
04/6/21 14:16:12,No_Tracking_Hostnames,hostwinds,:: hostwinds
04/6/21 14:16:30,MS_2,beritanowtk,beritanowtk

Gertjan · Apr 6, 2021, 2:55 PM

Hummm.

Everything looks fine to me / identical to me.

Except : I can run that GRC tool as hard as I can, it won't break 'unbound'.

Did you try the old method "unbound mode" ?

vjizzle · Apr 6, 2021, 3:06 PM

@gertjan
Good tip @Gertjan. I immediately tested it and low-and-behold. Everything keeps working even if I bombard pfsense with the GRC benchmarking tool. Here the output before and after the GRC testing:

> server 192.168.100.1
Default Server:  odin.vXXXX.lan
Address:  192.168.100.1

> www.conduit.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.conduit.com
Address:  10.10.10.1

> www.vmn.net
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.vmn.net
Address:  10.10.10.1

> www.ku6.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.ku6.com
Address:  10.10.10.1

>

Looks like there is something the matter with the python integration here. Also as an added bonus in Unbound mode I can see that pfSense is now faster in answering queries compared to the AdGuard home server I have running here. In python mode pfSense was always slower:

🔒 Log in to view

Not that the speed difference is that big but still ^^. So in the end something is broken here. I will initiate a reinstall tomorrow unless someone else has another good idea. Thank you very. much for helping Gertjan!

vjizzle · Apr 6, 2021, 4:40 PM

Ok I just did a clean install of pfsense and then restored a backup. I have the exact same behavior again. Python mode pfblockerng stops working after I hammer it with GRC DNS benchmark.

For now I am switching over to AdGuard Home. Maybe better luck with pfSense in the next release :(

Gertjan · Apr 6, 2021, 10:28 PM

I'm at home now.

Did some testing.
The dns benchmark running, doing it thing.

In a console/SSH, I ran this :

tail -f /var/unbound/var/log/pfblockerng/dns_reply.log | grep 'conduit'

According the https://dbl.oisd.nl/ feed ( the oisd.nl feed) :

,www.conduit.com,,0,OISD,DNSBL_Compilation

and also :

,conduit.com,,0,MS_2,DNSBL_BBcan177

I was spamming

www.conduit.com

in a nslookup (Windows 10) session.

Suddenly, during the test :

......
> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

When the 0.0.0.0 came back, I saw this in the 'tail' console :

DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk

and then :

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1
Réponse ne faisant pas autorité :
Nom :    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

DNS-reply,Apr 6 23:59:12,cache,A,A,86399,www.conduit.com,2001:470:xxxx:65c::2001,198.49.162.233,IL

198.49.162.233 popped up.

I guess I can confirm what you saw.

Btw :

I saw :

Assembling DNSBL database...... completed [ 04/7/21 00:05:47 ]
TLD:
TLD analysis..xxxxxxxxxxx completed [ 04/7/21 00:05:51 ]

  ** TLD Domain count exceeded. [ 200000 ] All subsequent Domains listed as-is **

during the force update. The OISD feed is BIG : 1,2 million domain names.

I'll test tommorow agagin without the OISD feed, as the base domain name itself "conduit.com" is already in the BBCan 77 feed - as shown above.

edit : without the oisd.nl feed, and thus a rather (for me) small set of blocked domains, env 16962, the "0.0.0.0" blocking of "conduit.com" works.
www.conduit.com is resolved.....

vjizzle · Apr 7, 2021, 5:52 AM

Interesting! So maybe I am on to something :p.

I got curious and did the test with the BBCan77 feed. The domain conduit.com is indeed there. So I decided to disable the oisd.nl feed and force reload pfblockerNG. After that I bombarded pfSense with the DNS benchmarking tool. Guess what...everything keeps running fine for conduit.com. Multiple reloads of pfBlockerNG, multiple restart of Unbound but everything kept working as expected:

> conduit.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    conduit.com
Address:  0.0.0.0

Then I enabled the https://dbl.oisd.nl feed did a force reload of pfBlockerNG. I don't have TLD blocking enabled so I don't get that message you have. You can see the log about loading the oisd.nl list:

[ dbl_list ]			 Downloading update .. 200 OK.
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  1251871  1251871    154015     0          0          1097856              
  ----------------------------------------------------------------------

Saving DNSBL statistics... completed [ 04/7/21 00:31:48 ]
------------------------------------------------------------------------
Assembling DNSBL database...... completed [ 04/7/21 00:31:49 ]
Reloading Unbound Resolver (DNSBL python).
Stopping Unbound Resolver.
Unbound stopped in 2 sec.
Additional mounts (DNSBL python):
  No changes required.
Starting Unbound Resolver... completed [ 04/7/21 00:31:53 ]
Resolver cache restored
DNSBL update [ 1379690 | PASSED  ]... completed

And of course now pfBlockerNG is broken and not blocking the domain anymore BUT conduit.com keeps getting blocked by pfblockerng fine:

> server 192.168.100.1
Default Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

> conduit.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Name:    conduit.com
Address:  0.0.0.0

> www.conduit.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

> www.ku6.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    3a1d905b3d90ef4e-cl2.qcloudcjgj.com
Address:  118.89.206.86
Aliases:  www.ku6.com
          fa2a686c758fa3d110164df993a0b8c4.qcloudzygj.com

> www.vmn.net
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    www.vmn.net
Address:  54.146.142.3

>

So www.conduit.com is broken but conduit.com is being blocked fine. Even after I run the DNS benchmark tool multiple times conduit.com keeps working.

I am glad you are seeing the same behavior like me. Looks like something is crashing when stressed with multiple (and fast) DNS queries .

Gertjan · Apr 7, 2021, 5:52 AM

Same test, back on work.
Home is a 1 Mbytes 'small' VM, work use its own i5, with 8 Mbyyes - although I don't think this matters.

First test :
nslookup with full debug :

set db2

> www.conduit.com.
Serveur :   pfsense.my-work-domain.net
Address:  2001:470:xxxx:5c0:2::1

------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 20, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN

------------
------------
Got answer (49 bytes):
    HEADER:
        opcode = QUERY, id = 20, rcode = NOERROR
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN
    ANSWERS:
    ->  www.conduit.com
        type = A, class = IN, dlen = 4
        internet address = 0.0.0.0
        ttl = 60 (1 min)

------------
------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 21, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
------------
Got answer (33 bytes):
    HEADER:
        opcode = QUERY, id = 21, rcode = SERVFAIL
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
Nom :    www.conduit.com
Address:  0.0.0.0

Fine, "www.conduit.com" yields 0.0.0.0

Now I launch a DNS benchmark, and have it finished (30 seconds later)

I "nslookup" again :

> www.conduit.com.
Serveur :   pfsense.my-work-domain.net
Address:  2001:470:xxxx:5c0:2::1

------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 22, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN

------------
------------
Got answer (70 bytes):
    HEADER:
        opcode = QUERY, id = 22, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 2,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN
    ANSWERS:
    ->  www.conduit.com
        type = CNAME, class = IN, dlen = 9
        canonical name = www.va.conduit.com
        ttl = 86227 (23 hours 57 mins 7 secs)
    ->  www.va.conduit.com
        type = A, class = IN, dlen = 4
        internet address = 198.49.162.233
        ttl = 86227 (23 hours 57 mins 7 secs)

------------
Réponse ne faisant pas autorité :
------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 23, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
------------
Got answer (33 bytes):
    HEADER:
        opcode = QUERY, id = 23, rcode = SERVFAIL
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
Nom :    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

Note the difference :
www.conduit.com resolves to the CNAME "www.va.conduit.com" (domain not listed), and that one resolves to "198.49.162.233"

vjizzle · Apr 7, 2021, 7:34 AM

Hey Gertjan. This is correct about eventually resolving to a CNAME. I can see it on my end also that after running the GRC benchmark testing tool the domain resolves to a CNAME (www.va.conduit.com) and so pfblockerNG is not blocking it anymore. I think the same thing is happening with "www.ku6.com" and "www.vmn.net".

Can we conclude that the CNAME blocking on pfBlockerNG is not working as expected?
Because the same test on AdguardHome server always gets the domains blocked. I think that is also the reason why I am getting the low percentage in the pfblockerNG dashboard widget (AdguardHome is at 10% blocked DNS requests at the moment).

I also think the same thing is happening with other domains as well. Just found out that "diagnostics.meethue.com" gets blocked but after benchmarking with GRC it is not blocked anymore.
I have never seen "diagnostics.meethue.com" in the pfblockerNG DNSBL Block Stats tab being in the list. After running my whole network for 24 hrs on AdGuardhome it is the domain being blocked the most.

BBcan177 · Apr 7, 2021, 5:05 PM

@vjizzle

Thanks for the report! Its appreciated!

The issue is that GRC is using a mixture of lowercase and uppercase in their DNS Tests.

I have fixed the code to force everything to lowercase before the domains are validated.

A fix can be downloaded here until the next version is available:

curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/1e400f170b7ef7e8cb76b548d6524b31/raw"

Then Restart Unbound

Thanks!