pfBlockerNG not blocking domain after first DNS lookup attempt
-
When I add msn.com to a custom DNSBL Group for blocking, pfBlockerNG works perfectly and keeps blocking msn.com and www.msn.com. That goes for every domain I add to my custom blocklist.
The problems starts to occur when there is CNAME resolving involved. As you can see www.msn.com resolves to a CNAME a-0003.a-msedge.net. This hostname is in the blocklist I am using:
I have TLD blocking and CNAME blocking options enabled in pfBlockerNG:
That is why I would expect pfBlockerNG to keep blocking www.msn.com because the CNAME it resolves to is in the blocklist.
This is also the behaviour in pihole. When it resolves to a CNAME in a blocklist, the initial domain is blocked:
-
@jrey Please find the screenshots below on how I have setup the DNSBL group:
-
That's why I want to see the log files ;-)
Editand the DNSBL Groups settings for the specific group/list you have setup containing the list you referenced.oops your second post appeared lated.You also have Regex Blocking enabled, do you have any entries in the list that appears below on the screen when that is enabled?
-
@jrey Ah yes. Are the screenshots I just posted enough? What logfiles do you want to see?
-
@vjizzle
pfblocker -> logs tabpfblockerng.log
error.log (if there is anything in it)You only need to provide the sections related to processing your CustomFeed and TLD processing.
-
Below a snippet of the logfile:
[ Force Reload Task - DNSBL ] UPDATE PROCESS START [ v3.2.0_7 ] [ 11/30/23 16:54:38 ] ===[ DNSBL Process ]================================================ Loading DNSBL Statistics... completed Loading DNSBL SafeSearch... enabled Loading DNSBL Whitelist... completed [ AdAway_Default_Blocklist ] Reload [ 11/30/23 16:54:39 ] . completed .. Whitelist: XXXXXXXXXnet| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 6540 6540 0 2 0 6538 ---------------------------------------------------------------------- [ adguarddns_justdomains ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 48994 48993 902 0 0 48091 ---------------------------------------------------------------------- [ AmazonFire_Tracking ] Reload [ 11/30/23 16:54:40 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 16 16 4 0 0 12 ---------------------------------------------------------------------- [ AndroidTracking ] Reload . completed .. Whitelist: XXXXXXXom| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 80 80 56 1 0 23 ---------------------------------------------------------------------- [ Disconnect_Me ] Reload [ 11/30/23 16:54:41 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2701 2701 738 0 0 1963 ---------------------------------------------------------------------- [ easylist_justdomains ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 22066 22045 21350 0 0 695 ---------------------------------------------------------------------- [ easyprivacy_justdomains ] Reload [ 11/30/23 16:54:42 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 20270 19116 7260 0 0 11856 ---------------------------------------------------------------------- [ FB_AdAway_Hosts ] Reload [ 11/30/23 16:54:43 ] . completed .. Whitelist: XXXXXXXXpulse.net| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 6540 6540 6538 2 0 0 ---------------------------------------------------------------------- [ FB_Adguard ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 57509 57508 28109 0 0 29399 ---------------------------------------------------------------------- [ FB_Admiral ] Reload [ 11/30/23 16:54:44 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1014 1001 919 0 0 82 ---------------------------------------------------------------------- [ FB_Anudeep ] Reload [ 11/30/23 16:54:45 ] . completed .. Whitelist:XXXXXXXXse.net| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 42536 42363 2955 1 0 39407 ---------------------------------------------------------------------- [ FB_EasyList ] Reload [ 11/30/23 16:54:46 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 31527 31527 31403 0 0 124 ---------------------------------------------------------------------- [ FB_jdlingyu ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1707 1645 574 0 0 1071 ---------------------------------------------------------------------- [ Firebog_easy_privacy ] Reload [ 11/30/23 16:54:47 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 21758 21758 10606 0 0 11152 ---------------------------------------------------------------------- [ nextdns_cname_cloacking ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 35 35 19 0 0 16 ---------------------------------------------------------------------- [ Public_proxies ] Reload [ 11/30/23 16:54:48 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 265 244 1 0 0 243 ---------------------------------------------------------------------- [ StevenBlack ] Reload . completed .. Whitelist: XXXXXXXXXXXn| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 160200 160200 13687 3 0 146510 ---------------------------------------------------------------------- [ TikTok ] Reload [ 11/30/23 16:54:51 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3699 3699 78 0 0 3621 ---------------------------------------------------------------------- [ WindowsLies ] Reload [ 11/30/23 16:54:53 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 107 107 82 0 0 25 ---------------------------------------------------------------------- [ CustomFeeds_custom ] Reload [ 11/30/23 16:54:54 ]. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 11 11 7 0 0 4 ----------------------------------------------------------------------Assembling DNSBL database...... completed [ 11/30/23 16:57:08 ] TLD: Blocking full TLD/Sub-Domain(s)... |cn|by|zip|mov| completed TLD analysis..................................... completed [ 11/30/23 16:57:25 ] TLD finalize... ---------------------------------------- Original Matches Removed Final ---------------------------------------- 3649180 1412714 1873986 1775194 ----------------------------------------- TLD finalize... completed [ 11/30/23 16:57:52 ] Saving DNSBL statistics... completed [ 11/30/23 16:58:01 ] Reloading Unbound Resolver (DNSBL python). Stopping Unbound Resolver. Unbound stopped in 2 sec. Additional mounts (DNSBL python): No changes required. Starting Unbound Resolver... completed [ 11/30/23 16:58:05 ] Resolver cache restored DNSBL update [ 1775194 | PASSED ]... completed [ 11/30/23 16:58:07 ]
-
-
@jrey
I am offline for the night but I am curious what you find! -
Wow that's a large list
TLD finalize...
Original Matches Removed Final
3649180 1412714 1873986 1775194Only reason I mention this is the one specific list you are providing hasn't been touched in something like 8 years. Many of the domain names in it don't even exist anymore. You might want to review your lists for being current and accurate
That said with settings similar to yours, I was able to recreate the issue, with a much smaller list set.
As I mentioned above I think it is a cache issue, and once I got the system into a state of working like yours, the only way I could get it functional again was:(also flush the DNS cache on the clients)
on the netgate
Turn off CNAME Validation
Turn off Resolver Cache,
restart the system
let the system start fully, don't query anything.I added the top 3 entries on this list (there are many others, I checked my upstream DNS and what it had been resolving) keep in mind these are all the result of outbound queries, and won't happen unless there is a specific DNS request for them).
Once running, you can re-enable both
CNAME and Resolver Cache
Reload the DNSBL as shown above
then once that competes, wait a couple of minutes visit Status -> Services and restart unbound one more time.Test.
I need to go back and setup the failure again, but I'd like to see your results when you do this.are you running DHCP Server on the device?
also Status -> System Logs (System Tab/ DNS Resolver)
filter like this,
Thanks
-
please note that once I completed the above steps,
all of these lookups are within the same session, with the steps taken on pfSenses noted within.nslookup > server 10.168.1.1 Default server: 10.168.1.1 Address: 10.168.1.1#53 > > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: www.msn.com Address: 0.0.0.0 ** server can't find www.msn.com: SERVFAIL * at this point I turned off the list, removed the 3 entries and force reload the DNSBL * > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: www.msn.com canonical name = www-msn-com.a-0003.a-msedge.net. www-msn-com.a-0003.a-msedge.net canonical name = a-0003.a-msedge.net. Name: a-0003.a-msedge.net Address: 204.79.197.203 * at this point I turned the list on again, and added the three entries, force reload the DNSBL * > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: www.msn.com Address: 0.0.0.0 ** server can't find www.msn.com: SERVFAIL > a.msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: a.msedge.net Address: 0.0.0.0 ** server can't find a.msedge.net: SERVFAIL > a-msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: a-msedge.net Address: 0.0.0.0 ** server can't find a-msedge.net: SERVFAIL > dc-msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: dc-msedge.net Address: 0.0.0.0 ** server can't find dc-msedge.net: SERVFAIL >all results above from the client are within the same nslookup session. (didn't exit and/or do anything special on the client) the system is just returning the expected results again.
-
@jrey
Hi. I did the steps you asked but still the same.- disabled cname and resolver cache
- restarted pfsense
- waited a good 5 minutes after restart
- re-enabled cname and resolver cache
- reloaded the pfblockerng config using the Update tab
The first query succeeds and returns 0.0.0.0 but after that it starts resolving.
Please find the screenshot below of the DNS Resolver log:
-
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
the DNS Resolver log:
that's it one entry? I'm trying to get a feel for how often the resolver is starting/restarting..
did you add the 3 specific blocks on the custom list ? you're not clear on those being added. if you did not please do and force reload the DNSBL
flush the cache on the client (windows) and try again.the results you provide suggest strongly that another DNS is providing the response in the chain of responses
above you provided
Please find an overview of my DNS Resolver and DNSBL settings in the screenshots below.
but I don't see in that screen capture the fields that are above the "strict outgoing network setting" the point where you start the screen capture
Thanks
-
@jrey said in pfBlockerNG not blocking domain after first DNS lookup attempt:
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
the DNS Resolver log:
that's it one entry? I'm trying to get a feel for how often the resolver is starting/restarting..
did you add the 3 specific blocks on the custom list ? you're not clear on those being added. if you did not please do and force reload the DNSBL
flush the cache on the client (windows) and try again.the results you provide suggest strongly that another DNS is providing the response in the chain of responses
above you provided
Please find an overview of my DNS Resolver and DNSBL settings in the screenshots below.
but I don't see in that screen capture the fields that are above the "strict outgoing network setting" the point where you start the screen capture
Thanks
I did add the entries to the custom blocklist on my DNSBL group, did the force reload and also the restart. Flushed the cache from my Windows and MacOS client, but then again when using nslookup via cli there is no cache, that's the whole point of using nslookup of course.
But here is the thing, when add them to my custom blocklist they are always blocked and everything is working as expected.
In the strict outgoing setting I have enforced the DNS resolver in pfSense to use my VPN. Now I know that several VPN providers are doing DNS intercept, mine is not. Checked and verified with them. Also, my pihole is using the same VPN route to go to the internet and the unbound resolver on my pihole in combination with pihole itself never misses a beat when blocking.
-
so then if you add the three items I provided it blocks, but when they are not there it doesn't.
you realize that this actually puts those entries on the same effective DNSBL list.
from pfsense diagnostics ping - ping this
ad.doubleclick.net
what do you get?from your screen image provided earlier, you have this checked, so what group policy do you have that might potential allows bypass.
my pihole is using the same VPN route to go to the internet and the unbound resolver on my pihole in combination with pihole itself never misses a beat when blocking
sure - but how is the network interface configured on the pihole vs the pfsense
specifically in the section on the pfsense page just above the strict outgoing network, setting where your screen capture starts, what are the settings for
Network Interfaces and Outgoing Network Interfaces
drag the low right corner to expand the lists so we can see what is selected. -
here are the results of every name on the block list you provided. no issue with anything on the list. All are returning "blocked" even though some of the names don't even exist anymore, they would still be blocked first.
When blocked
DNSBL Results.txtWhen not blocked (those names where the name no longer exists are returning NXDOMAIN and those on another list still return 0.0.0.0, and those names that are still active and not on another list return IP addresses.
DNSBL is working as I would expect.
-
@jrey
So what you are seeing if domain names don’t exist anymore and are in fact returning a NXDOMAIN, this throws pfblockerng into despair as it then starts to answer?If I compare Pihole with pfblockerng, it does not matter what its resolver is doing, or the forwarder or the actual dns record on the internet, when a domain name is in a blocklist…Pihole just blocks it.
-
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
NXDOMAIN, this throws pfblockerng into despair as it then starts to answer
Not something I said, and not what happens. DNSBL is not in a state of despair and does not then start to answer on subsequent queries for a domain that does not exist.
for example corpext.msitadfs.glbdns2.microsoft.com
when the name is on an active list you get the DNSBL response of 0.0.0.0 as the non-authoritative answer:
(the DNSBL just blocked it, it was told to return 0.0.0.0 and it does)> corpext.msitadfs.glbdns2.microsoft.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: corpext.msitadfs.glbdns2.microsoft.com Address: 0.0.0.0 ** server can't find corpext.msitadfs.glbdns2.microsoft.com: SERVFAILWhen it is not on a DNSBL list and name does not exist you get a direct response, but there is no resolution (no IP can be provided)
> corpext.msitadfs.glbdns2.microsoft.com Server: 10.168.1.1 Address: 10.168.1.1#53 ** server can't find corpext.msitadfs.glbdns2.microsoft.com: NXDOMAINvisit any online dnslookup and they will tell you the domain does not exist.
here is the log from a site that does DNS
it does not matter what its resolver is doing, or the forwarder or the actual dns record on the internet
yeah, actually it does.
when a domain name is in a blocklist…Pihole just blocks
and so does DNSBL from the pfBlockerNG, when name is on an active list, as shown above, you get 0.0.0.0
of course, if you prefer pihole there is no reason you can't continue to use it for your DNS blocking/resolution, in combination with pfsense. There are 100's of how-to's on the subject.
I, like many others, have not had a problem with the DNSBL it just does what you tell it to do - block
and yes, I did say earlier that I can recreate the issue you see, but in order to do that I had to change settings and force the system to do bad things in the first place.there are still a couple of outstanding requests for you to provide your configuration information, (parts that you haven't already provided) and you have not, so not sure how else we can continue. If your settings are correct it will work for you, just like it does for me.
Thanks
Have a great weekend. -
I really appreciate all the help. Next week when I am at home I will dive into this again. In the meantime Pihole just the job.