pfBlockerNG not blocking domain after first DNS lookup attempt
-
@johnpoz said in pfBlockerNG not blocking domain after first DNS lookup attempt:
doh in your browser would be the most likely suspect.
unlikely doh, it is an nslookup - port 53
it is a good question, I've preformed the same tests as the OP, and my DoH rule has never tripping, but at the same time it just blocks every single time.
Let's ask this @vjizzle are you loading / have you loaded msn at any point during testing in a browser.
as for the browser, on my windows machine
and the nslookup all still returned 0.0.0.0 after. No DoH recorded.
but something could be loading the cache,
"Message cache elements are prefected..."
"Serve cache records even with TTL of 0"Personally I've never had much use for either those settings, OP may have a requirement ?
-
I have not loaded www.msn.com (or the CNAME) in my browser or other device in my network. The DNS options for prefetch and server TTL of 0 is only there to speedup host lookups. The same options are enabled on my pihole which is using local installed unbound in resolver mode.
Somehow the pfSense resolver manages to bypass pfBlockerNG or CNAME blocking fails in pfBlockerNG, looks like to me at least.
-
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
CNAME blocking fails in pfBlockerNG
you could have some problem with something looking up the cname but not the original that is blocked..
;; QUESTION SECTION: ;www.msn.com. IN A ;; ANSWER SECTION: www.msn.com. 11194 IN CNAME www-msn-com.a-0003.a-msedge.net. www-msn-com.a-0003.a-msedge.net. 56 IN CNAME a-0003.a-msedge.net. a-0003.a-msedge.net. 56 IN A 204.79.197.203Blocking msn.com doesn't prevent lookup of a-msedge.net.
If I wanted to block msn.com what I would do is put this directly in unbound custom box, since I don't use blocking in pfblocker..
local-zone: "msn.com" always_nxdomain
-
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
www.msn.com is blocked via CNAME a-0003.a-msedge.net
look above when I ran the test the canonical name returned is not on the list you provided.
I'd like to see the specific setup you have for this list you are providing.
(ie the DNSBL setup containing that list)Hopefully you have it added under an entry within DNSBL Groups?
also in the pfblockerng.log there should be a section showing it being processed.
provide that log section please.
anything in the error.log (pfblockerng)then most likely what you are seeing is that www.msn.com is not on the list, and you are returning the CNAME on subsequent queries some of those are multi-named
not also that the CNAME I returned is www.msn.com.a-0003.a-msedge.net
and the CNAME you see is a-0003.a-msedge.nettraceroute a-0003.a-msedge.net traceroute to a-0003.a-msedge.net (204.79.197.203), 64 hops max, 52 byte packetstraceroute www.msn.com.a-0003.a-msedge.net traceroute to a-0003.a-msedge.net (204.79.197.203), 64 hops max, 52 byte packetson your DNSBL category, where you have the list being added, goto the bottom DNSBL Custom_list and add the entry msn.com
no quotes, no other sub just msn.com
then force reload the DNSBL on the update tab
check the same 2 logs referenced above, for the processing of the list and the new custom entry
try the test again.
-
yup, I really want to see the settings of the specific DNSBL category. being used,
using the local-zone would also work, but in this case the OP already has a "list" in pfBocker, that blocks a bunch of other specific things, just not all the "specific things"
msn.com on the DNSBL category, would keep it all together for the OP.
certainly I don't use the list, and blocking the lookup by DNSBL works and does not return a CNAME.
However there might be a problem with the TLD because from the list
a-0003.a-msedge.net is listed.The setting, CNAME Validation set being enable MIGHT be the source of the "leak". Still trying to recreate the OP's various settings to see if I can recreate the issue.
-
When I add msn.com to a custom DNSBL Group for blocking, pfBlockerNG works perfectly and keeps blocking msn.com and www.msn.com. That goes for every domain I add to my custom blocklist.
The problems starts to occur when there is CNAME resolving involved. As you can see www.msn.com resolves to a CNAME a-0003.a-msedge.net. This hostname is in the blocklist I am using:
I have TLD blocking and CNAME blocking options enabled in pfBlockerNG:
That is why I would expect pfBlockerNG to keep blocking www.msn.com because the CNAME it resolves to is in the blocklist.
This is also the behaviour in pihole. When it resolves to a CNAME in a blocklist, the initial domain is blocked:
-
@jrey Please find the screenshots below on how I have setup the DNSBL group:
-
That's why I want to see the log files ;-)
Editand the DNSBL Groups settings for the specific group/list you have setup containing the list you referenced.oops your second post appeared lated.You also have Regex Blocking enabled, do you have any entries in the list that appears below on the screen when that is enabled?
-
@jrey Ah yes. Are the screenshots I just posted enough? What logfiles do you want to see?
-
@vjizzle
pfblocker -> logs tabpfblockerng.log
error.log (if there is anything in it)You only need to provide the sections related to processing your CustomFeed and TLD processing.
-
Below a snippet of the logfile:
[ Force Reload Task - DNSBL ] UPDATE PROCESS START [ v3.2.0_7 ] [ 11/30/23 16:54:38 ] ===[ DNSBL Process ]================================================ Loading DNSBL Statistics... completed Loading DNSBL SafeSearch... enabled Loading DNSBL Whitelist... completed [ AdAway_Default_Blocklist ] Reload [ 11/30/23 16:54:39 ] . completed .. Whitelist: XXXXXXXXXnet| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 6540 6540 0 2 0 6538 ---------------------------------------------------------------------- [ adguarddns_justdomains ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 48994 48993 902 0 0 48091 ---------------------------------------------------------------------- [ AmazonFire_Tracking ] Reload [ 11/30/23 16:54:40 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 16 16 4 0 0 12 ---------------------------------------------------------------------- [ AndroidTracking ] Reload . completed .. Whitelist: XXXXXXXom| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 80 80 56 1 0 23 ---------------------------------------------------------------------- [ Disconnect_Me ] Reload [ 11/30/23 16:54:41 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 2701 2701 738 0 0 1963 ---------------------------------------------------------------------- [ easylist_justdomains ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 22066 22045 21350 0 0 695 ---------------------------------------------------------------------- [ easyprivacy_justdomains ] Reload [ 11/30/23 16:54:42 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 20270 19116 7260 0 0 11856 ---------------------------------------------------------------------- [ FB_AdAway_Hosts ] Reload [ 11/30/23 16:54:43 ] . completed .. Whitelist: XXXXXXXXpulse.net| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 6540 6540 6538 2 0 0 ---------------------------------------------------------------------- [ FB_Adguard ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 57509 57508 28109 0 0 29399 ---------------------------------------------------------------------- [ FB_Admiral ] Reload [ 11/30/23 16:54:44 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1014 1001 919 0 0 82 ---------------------------------------------------------------------- [ FB_Anudeep ] Reload [ 11/30/23 16:54:45 ] . completed .. Whitelist:XXXXXXXXse.net| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 42536 42363 2955 1 0 39407 ---------------------------------------------------------------------- [ FB_EasyList ] Reload [ 11/30/23 16:54:46 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 31527 31527 31403 0 0 124 ---------------------------------------------------------------------- [ FB_jdlingyu ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 1707 1645 574 0 0 1071 ---------------------------------------------------------------------- [ Firebog_easy_privacy ] Reload [ 11/30/23 16:54:47 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 21758 21758 10606 0 0 11152 ---------------------------------------------------------------------- [ nextdns_cname_cloacking ] Reload . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 35 35 19 0 0 16 ---------------------------------------------------------------------- [ Public_proxies ] Reload [ 11/30/23 16:54:48 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 265 244 1 0 0 243 ---------------------------------------------------------------------- [ StevenBlack ] Reload . completed .. Whitelist: XXXXXXXXXXXn| ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 160200 160200 13687 3 0 146510 ---------------------------------------------------------------------- [ TikTok ] Reload [ 11/30/23 16:54:51 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 3699 3699 78 0 0 3621 ---------------------------------------------------------------------- [ WindowsLies ] Reload [ 11/30/23 16:54:53 ] . completed .. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 107 107 82 0 0 25 ---------------------------------------------------------------------- [ CustomFeeds_custom ] Reload [ 11/30/23 16:54:54 ]. ---------------------------------------------------------------------- Orig. Unique # Dups # White # TOP1M Final ---------------------------------------------------------------------- 11 11 7 0 0 4 ----------------------------------------------------------------------Assembling DNSBL database...... completed [ 11/30/23 16:57:08 ] TLD: Blocking full TLD/Sub-Domain(s)... |cn|by|zip|mov| completed TLD analysis..................................... completed [ 11/30/23 16:57:25 ] TLD finalize... ---------------------------------------- Original Matches Removed Final ---------------------------------------- 3649180 1412714 1873986 1775194 ----------------------------------------- TLD finalize... completed [ 11/30/23 16:57:52 ] Saving DNSBL statistics... completed [ 11/30/23 16:58:01 ] Reloading Unbound Resolver (DNSBL python). Stopping Unbound Resolver. Unbound stopped in 2 sec. Additional mounts (DNSBL python): No changes required. Starting Unbound Resolver... completed [ 11/30/23 16:58:05 ] Resolver cache restored DNSBL update [ 1775194 | PASSED ]... completed [ 11/30/23 16:58:07 ]
-
-
@jrey
I am offline for the night but I am curious what you find! -
Wow that's a large list
TLD finalize...
Original Matches Removed Final
3649180 1412714 1873986 1775194Only reason I mention this is the one specific list you are providing hasn't been touched in something like 8 years. Many of the domain names in it don't even exist anymore. You might want to review your lists for being current and accurate
That said with settings similar to yours, I was able to recreate the issue, with a much smaller list set.
As I mentioned above I think it is a cache issue, and once I got the system into a state of working like yours, the only way I could get it functional again was:(also flush the DNS cache on the clients)
on the netgate
Turn off CNAME Validation
Turn off Resolver Cache,
restart the system
let the system start fully, don't query anything.I added the top 3 entries on this list (there are many others, I checked my upstream DNS and what it had been resolving) keep in mind these are all the result of outbound queries, and won't happen unless there is a specific DNS request for them).
Once running, you can re-enable both
CNAME and Resolver Cache
Reload the DNSBL as shown above
then once that competes, wait a couple of minutes visit Status -> Services and restart unbound one more time.Test.
I need to go back and setup the failure again, but I'd like to see your results when you do this.are you running DHCP Server on the device?
also Status -> System Logs (System Tab/ DNS Resolver)
filter like this,
Thanks
-
please note that once I completed the above steps,
all of these lookups are within the same session, with the steps taken on pfSenses noted within.nslookup > server 10.168.1.1 Default server: 10.168.1.1 Address: 10.168.1.1#53 > > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: www.msn.com Address: 0.0.0.0 ** server can't find www.msn.com: SERVFAIL * at this point I turned off the list, removed the 3 entries and force reload the DNSBL * > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: www.msn.com canonical name = www-msn-com.a-0003.a-msedge.net. www-msn-com.a-0003.a-msedge.net canonical name = a-0003.a-msedge.net. Name: a-0003.a-msedge.net Address: 204.79.197.203 * at this point I turned the list on again, and added the three entries, force reload the DNSBL * > www.msn.com Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: www.msn.com Address: 0.0.0.0 ** server can't find www.msn.com: SERVFAIL > a.msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: a.msedge.net Address: 0.0.0.0 ** server can't find a.msedge.net: SERVFAIL > a-msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: a-msedge.net Address: 0.0.0.0 ** server can't find a-msedge.net: SERVFAIL > dc-msedge.net Server: 10.168.1.1 Address: 10.168.1.1#53 Non-authoritative answer: Name: dc-msedge.net Address: 0.0.0.0 ** server can't find dc-msedge.net: SERVFAIL >all results above from the client are within the same nslookup session. (didn't exit and/or do anything special on the client) the system is just returning the expected results again.
-
@jrey
Hi. I did the steps you asked but still the same.- disabled cname and resolver cache
- restarted pfsense
- waited a good 5 minutes after restart
- re-enabled cname and resolver cache
- reloaded the pfblockerng config using the Update tab
The first query succeeds and returns 0.0.0.0 but after that it starts resolving.
Please find the screenshot below of the DNS Resolver log:
-
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
the DNS Resolver log:
that's it one entry? I'm trying to get a feel for how often the resolver is starting/restarting..
did you add the 3 specific blocks on the custom list ? you're not clear on those being added. if you did not please do and force reload the DNSBL
flush the cache on the client (windows) and try again.the results you provide suggest strongly that another DNS is providing the response in the chain of responses
above you provided
Please find an overview of my DNS Resolver and DNSBL settings in the screenshots below.
but I don't see in that screen capture the fields that are above the "strict outgoing network setting" the point where you start the screen capture
Thanks
-
@jrey said in pfBlockerNG not blocking domain after first DNS lookup attempt:
@vjizzle said in pfBlockerNG not blocking domain after first DNS lookup attempt:
the DNS Resolver log:
that's it one entry? I'm trying to get a feel for how often the resolver is starting/restarting..
did you add the 3 specific blocks on the custom list ? you're not clear on those being added. if you did not please do and force reload the DNSBL
flush the cache on the client (windows) and try again.the results you provide suggest strongly that another DNS is providing the response in the chain of responses
above you provided
Please find an overview of my DNS Resolver and DNSBL settings in the screenshots below.
but I don't see in that screen capture the fields that are above the "strict outgoing network setting" the point where you start the screen capture
Thanks
I did add the entries to the custom blocklist on my DNSBL group, did the force reload and also the restart. Flushed the cache from my Windows and MacOS client, but then again when using nslookup via cli there is no cache, that's the whole point of using nslookup of course.
But here is the thing, when add them to my custom blocklist they are always blocked and everything is working as expected.
In the strict outgoing setting I have enforced the DNS resolver in pfSense to use my VPN. Now I know that several VPN providers are doing DNS intercept, mine is not. Checked and verified with them. Also, my pihole is using the same VPN route to go to the internet and the unbound resolver on my pihole in combination with pihole itself never misses a beat when blocking.
-
so then if you add the three items I provided it blocks, but when they are not there it doesn't.
you realize that this actually puts those entries on the same effective DNSBL list.
from pfsense diagnostics ping - ping this
ad.doubleclick.net
what do you get?from your screen image provided earlier, you have this checked, so what group policy do you have that might potential allows bypass.
my pihole is using the same VPN route to go to the internet and the unbound resolver on my pihole in combination with pihole itself never misses a beat when blocking
sure - but how is the network interface configured on the pihole vs the pfsense
specifically in the section on the pfsense page just above the strict outgoing network, setting where your screen capture starts, what are the settings for
Network Interfaces and Outgoing Network Interfaces
drag the low right corner to expand the lists so we can see what is selected. -
here are the results of every name on the block list you provided. no issue with anything on the list. All are returning "blocked" even though some of the names don't even exist anymore, they would still be blocked first.
When blocked
DNSBL Results.txtWhen not blocked (those names where the name no longer exists are returning NXDOMAIN and those on another list still return 0.0.0.0, and those names that are still active and not on another list return IP addresses.
DNSBL is working as I would expect.
