How to unblock duckduckgo and find why it's being blocked.

thewismit

Re: PfblockerNG-devel SafeSearch Redirection

Hi folks. Based on the referenced thread it seems changes need to be made to the configuration to allow duckduckgo to render. Turning off safe search is not an option for me.

So my questions are

list itemhow can I allow duckduckgo.com to work?
list itemhow can I search for what about the domain is being blocked? (ads, subdomains, etc.) I want to be able to track down the source of a page getting blocked in general and understand how to whitelist it.

Thanks for any help you can provide and I'm happy to be here!

provels

@thewismit
I had the same problem, so I just disabled SafeSearch. Took a while to figure it out. And it seems (according to the line item description at least) that DDG should work fine. I'm a little foggy on this, but I think the problem was caused by DDG's IP addresses changing several/many months ago. And perhaps whatever is hard-coded into pfB for DDG is still the old address? Or I could be completely wrong... Yeah, probably that... Maybe I'll see if I can figure it out, too.

provels

OK, I enabled SS, reloaded, flushed the DNS cache... and I can get to DDG just fine. Huh.

https://start.duckduckgo.com/

thewismit

@provels said in How to unblock duckduckgo and find why it's being blocked.:

https://start.duckduckgo.com/

When I do a search from my browser bar, it calls this URL: https://duckduckgo.com/?t=ffab&q=testing, which isn't working.

Is there a way to see exactly what is being blocked when this page is requested?

provels

@thewismit
OK, it turns out although https://start.duckduckgo.com/ will load, any search from there fails. I saw a block in the DNSBL log blocking improving.duckduckgo.com blocked by https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt if you happen to use that,. Added .duckduckgo.com to the DNSBL Whitelist w/o benefit. So short story is I don't know. I ran a quick packet capture, but didn't see anything obvious. Maybe you can try that.

thewismit

Thanks @provels. I don't even know how you found improving.duckduckgo.com in the DNSBL log. I've tried finding anything and haven't been successful beyond IP addresses so far.

I also have no idea how to read a packet capture yet but I do appreciate you trying and replying.

provels

Perhaps @BBcan177 can take a look when convenient. It seems SafeSearch should work with DDG, at least according to the description on the 'SafeSearch Redirection' line item.

rjamesm

Any fix on this problem? I don't want to disable safe search in order to fix this problem.

Patch

@rjamesm said in How to unblock duckduckgo and find why it's being blocked.:

I don't want to disable safe search

My understanding is safe search is a Google function.
So blocking Google but white listing safe search may work.
A likely issue is Google rapidly changes what URL resolve to to enable them to harvest timing data. This can be blocked by overriding the minimum time in pfsense however white listing the specific Google sites you want to access is likely to require pfsense white lists also recent history (as client will use old IP addresses)

mrtumnus

duckduckgo.com isn't being "blocked" by pfblockerng, per se. pfblockerng uses unbound to work its magic for DNSBL, including the Safe Search feature. It has a big list of search engine host overrides mapped to the IP address of the corresponding web host that enforces safe search for that engine. However, for duckduckgo.com (and pixabay.com), a CNAME is used to redirect each domain to a DNS name of the forced-safesearch host. This would work if unbound were setup as an authoritative name server, but it is not by default. So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

I got around this by creating host overrides for duckduckgo.com to whatever the current IP of safe.duckduckgo.com is. This will break if the IP ever changes, but I have a feeling it should be stable for a while. I tried to mess around getting BIND to resolve the CNAME, but I couldn't figure it out. At least this is a temporary work-around.

EDIT: Here's my host override in DNS Resolver:

johnpoz

@mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

Huh? Maybe not understanding what your saying. But unbound doesn't have to be authoritative to follow a cname.. There was a fqdn that came up today with a 8 cname chain, which is insane and it resolves..

;logincdn.msauth.net.           IN      A

;; ANSWER SECTION:
logincdn.msauth.net.    3600    IN      CNAME   lgincdn.trafficmanager.net.
lgincdn.trafficmanager.net. 3600 IN     CNAME   lgincdnmsftuswe2.azureedge.net.
lgincdnmsftuswe2.azureedge.net. 3600 IN CNAME   lgincdnmsftuswe2.afd.azureedge.net.
lgincdnmsftuswe2.afd.azureedge.net. 3600 IN CNAME firstparty-azurefd-prod.trafficmanager.net.
firstparty-azurefd-prod.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.t-msedge.net.
dual.part-0023.t-0009.t-msedge.net. 3600 IN CNAME global-entry-afdthirdparty-fallback.trafficmanager.net.
global-entry-afdthirdparty-fallback.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.fbs1-t-msedge.net.
dual.part-0023.t-0009.fbs1-t-msedge.net. 3600 IN CNAME part-0023.t-0009.fbs1-t-msedge.net.
part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.219.51
part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.227.51

;; Query time: 390 msec
;; SERVER: 192.168.9.253#53(192.168.9.253)

Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

mrtumnus

@johnpoz said in How to unblock duckduckgo and find why it's being blocked.:

Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

Yes, that's what I'm saying. I realize an example would have been helpful. I was using not only Windows tools, but also some open source ones like dig (presuming that the unholy cygwin doesn't entirely rely on Windows DNS - maybe it does?). In addition, the Net Analyzer app on my phone only retrieved a CNAME record, no A or AAAA, and ping fails.

The entry in pfblockerng is:

local-data: "duckduckgo.com IN CNAME safe.duckduckgo.com"

Are you suggesting that this results in successful DNS resolution on some clients? I'll have to try with dig oi a *nix environment.

mrtumnus

dig-ging on a nix box produces the same result - no DNS resolution via browser, ping, etc. So, this is not a Windows problem.

I took a look at what NextDNS does for its SafeSearch redirect. Here's the result:

$ dig duckduckgo.com

; <<>> DiG 9.11.9 <<>> duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37077
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;duckduckgo.com.                        IN      A

;; ANSWER SECTION:
duckduckgo.com.         300     IN      CNAME   safe.duckduckgo.com.
safe.duckduckgo.com.    300     IN      A       40.89.244.237

;; Query time: 38 msec
;; SERVER: 192.168.77.71#53(192.168.77.71)
;; WHEN: Fri Jun 10 11:54:41 EDT 2022
;; MSG SIZE  rcvd: 67

It would be great if we could get unbound to query the CNAME and provide the result as an A record, but from everything I'm reading it won't.

Tom8

I had the same problem.
With duckduckgo.com in the DNSBL whitelist it didn´t work, so I just added www.duckduckgo.com to the list.
Now it,s running.

BBcan177

I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

Change here:
https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/revisions

From the Shell or pfSense Diag/Command Prompt:

cp /var/unbound/pfb_unbound.py /var/unbound/pfb_unbound.py.bk

curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/raw/pfb_unbound.py"

Follow that with a Restart of Unbound. Might need to clear the OS and Browser cache.

mrtumnus

@bbcan177 said in How to unblock duckduckgo and find why it's being blocked.:

I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

What difference should I expect the change to make? I'm not seeing any resolution of the CNAME in a dig query.

Looking at the changes, I'm confused what line 1181 is about:

                elif isSafeSearch['A'] == 'cname':
why this? >>>       if isSafeSearch['AAAA'] is not None and isSafeSearch['AAAA'] != '':
                        if q_type == RR_TYPE_AAAA:

Why does an AAAA element of isSafeSearch[] need to exist? It is not checked for any of the if/else sibling conditions. Could this be bypassing the CNAME handling?

BBcan177

@mrtumnus
The /var/unbound/pfb_py_ss.txt is formatted like:

duckduckgo.com,cname,safe.duckduckgo.com
www.duckduckgo.com,cname,safe.duckduckgo.com
pixabay.com,cname,safesearch.pixabay.com
www.pixabay.com,cname,safesearch.pixabay.com
yandex.ru,213.180.193.56,
www.yandex.ru,213.180.193.56,

When the ss file is loaded it records 'cname' in the A field, and the acutal CNAME in the AAAA field.

safeSearchDB[row[0]] = {'A': row[1], 'AAAA': row[2]}

https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749#file-pfb_unbound-py-L335

I am short on time today, but shoot me an email to bbcan177@gmail.com and I try to look at it over the weekend.

mrtumnus

@bbcan177 I suppose this change might fix the issue with duckduckgo being blocked entirely? I misunderstood and thought you were addressing the CNAME resolution issue, but that is an issue with Unbound itself (I think).

I can confirm that duckduckgo does load properly now.

Ir0nsh007er

This post is deleted!

Ir0nsh007er

Same probleme here.. it's already fix or not?