How to unblock duckduckgo and find why it's being blocked.

provels · Dec 3, 2020, 4:39 PM

@thewismit
OK, it turns out although https://start.duckduckgo.com/ will load, any search from there fails. I saw a block in the DNSBL log blocking improving.duckduckgo.com blocked by https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt if you happen to use that,. Added .duckduckgo.com to the DNSBL Whitelist w/o benefit. So short story is I don't know. I ran a quick packet capture, but didn't see anything obvious. Maybe you can try that.

thewismit · Dec 3, 2020, 7:02 PM

Thanks @provels. I don't even know how you found improving.duckduckgo.com in the DNSBL log. I've tried finding anything and haven't been successful beyond IP addresses so far.

I also have no idea how to read a packet capture yet but I do appreciate you trying and replying.

provels · Dec 3, 2020, 11:43 PM

Perhaps @BBcan177 can take a look when convenient. It seems SafeSearch should work with DDG, at least according to the description on the 'SafeSearch Redirection' line item.

rjamesm · Mar 24, 2022, 10:50 PM

Any fix on this problem? I don't want to disable safe search in order to fix this problem.

Patch · Mar 25, 2022, 2:35 AM

@rjamesm said in How to unblock duckduckgo and find why it's being blocked.:

I don't want to disable safe search

My understanding is safe search is a Google function.
So blocking Google but white listing safe search may work.
A likely issue is Google rapidly changes what URL resolve to to enable them to harvest timing data. This can be blocked by overriding the minimum time in pfsense however white listing the specific Google sites you want to access is likely to require pfsense white lists also recent history (as client will use old IP addresses)

mrtumnus · Jun 9, 2022, 9:43 PM

duckduckgo.com isn't being "blocked" by pfblockerng, per se. pfblockerng uses unbound to work its magic for DNSBL, including the Safe Search feature. It has a big list of search engine host overrides mapped to the IP address of the corresponding web host that enforces safe search for that engine. However, for duckduckgo.com (and pixabay.com), a CNAME is used to redirect each domain to a DNS name of the forced-safesearch host. This would work if unbound were setup as an authoritative name server, but it is not by default. So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

I got around this by creating host overrides for duckduckgo.com to whatever the current IP of safe.duckduckgo.com is. This will break if the IP ever changes, but I have a feeling it should be stable for a while. I tried to mess around getting BIND to resolve the CNAME, but I couldn't figure it out. At least this is a temporary work-around.

EDIT: Here's my host override in DNS Resolver:

johnpoz · Jun 9, 2022, 9:57 PM

@mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

Huh? Maybe not understanding what your saying. But unbound doesn't have to be authoritative to follow a cname.. There was a fqdn that came up today with a 8 cname chain, which is insane and it resolves..

;logincdn.msauth.net.           IN      A

;; ANSWER SECTION:
logincdn.msauth.net.    3600    IN      CNAME   lgincdn.trafficmanager.net.
lgincdn.trafficmanager.net. 3600 IN     CNAME   lgincdnmsftuswe2.azureedge.net.
lgincdnmsftuswe2.azureedge.net. 3600 IN CNAME   lgincdnmsftuswe2.afd.azureedge.net.
lgincdnmsftuswe2.afd.azureedge.net. 3600 IN CNAME firstparty-azurefd-prod.trafficmanager.net.
firstparty-azurefd-prod.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.t-msedge.net.
dual.part-0023.t-0009.t-msedge.net. 3600 IN CNAME global-entry-afdthirdparty-fallback.trafficmanager.net.
global-entry-afdthirdparty-fallback.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.fbs1-t-msedge.net.
dual.part-0023.t-0009.fbs1-t-msedge.net. 3600 IN CNAME part-0023.t-0009.fbs1-t-msedge.net.
part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.219.51
part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.227.51

;; Query time: 390 msec
;; SERVER: 192.168.9.253#53(192.168.9.253)

Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

mrtumnus · Jun 9, 2022, 11:38 PM

@johnpoz said in How to unblock duckduckgo and find why it's being blocked.:

Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

Yes, that's what I'm saying. I realize an example would have been helpful. I was using not only Windows tools, but also some open source ones like dig (presuming that the unholy cygwin doesn't entirely rely on Windows DNS - maybe it does?). In addition, the Net Analyzer app on my phone only retrieved a CNAME record, no A or AAAA, and ping fails.

The entry in pfblockerng is:

local-data: "duckduckgo.com IN CNAME safe.duckduckgo.com"

Are you suggesting that this results in successful DNS resolution on some clients? I'll have to try with dig oi a *nix environment.

mrtumnus · Jun 10, 2022, 4:01 PM

dig-ging on a nix box produces the same result - no DNS resolution via browser, ping, etc. So, this is not a Windows problem.

I took a look at what NextDNS does for its SafeSearch redirect. Here's the result:

$ dig duckduckgo.com

; <<>> DiG 9.11.9 <<>> duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37077
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;duckduckgo.com.                        IN      A

;; ANSWER SECTION:
duckduckgo.com.         300     IN      CNAME   safe.duckduckgo.com.
safe.duckduckgo.com.    300     IN      A       40.89.244.237

;; Query time: 38 msec
;; SERVER: 192.168.77.71#53(192.168.77.71)
;; WHEN: Fri Jun 10 11:54:41 EDT 2022
;; MSG SIZE  rcvd: 67

It would be great if we could get unbound to query the CNAME and provide the result as an A record, but from everything I'm reading it won't.

Tom8 · Oct 14, 2022, 2:18 PM

I had the same problem.
With duckduckgo.com in the DNSBL whitelist it didn´t work, so I just added www.duckduckgo.com to the list.
Now it,s running.

BBcan177 · Oct 14, 2022, 2:18 PM

I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

Change here:
https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/revisions

From the Shell or pfSense Diag/Command Prompt:

cp /var/unbound/pfb_unbound.py /var/unbound/pfb_unbound.py.bk

curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/raw/pfb_unbound.py"

Follow that with a Restart of Unbound. Might need to clear the OS and Browser cache.

mrtumnus · Oct 14, 2022, 4:04 PM

@bbcan177 said in How to unblock duckduckgo and find why it's being blocked.:

I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

What difference should I expect the change to make? I'm not seeing any resolution of the CNAME in a dig query.

Looking at the changes, I'm confused what line 1181 is about:

                elif isSafeSearch['A'] == 'cname':
why this? >>>       if isSafeSearch['AAAA'] is not None and isSafeSearch['AAAA'] != '':
                        if q_type == RR_TYPE_AAAA:

Why does an AAAA element of isSafeSearch[] need to exist? It is not checked for any of the if/else sibling conditions. Could this be bypassing the CNAME handling?

BBcan177 · Oct 14, 2022, 4:12 PM

@mrtumnus
The /var/unbound/pfb_py_ss.txt is formatted like:

duckduckgo.com,cname,safe.duckduckgo.com
www.duckduckgo.com,cname,safe.duckduckgo.com
pixabay.com,cname,safesearch.pixabay.com
www.pixabay.com,cname,safesearch.pixabay.com
yandex.ru,213.180.193.56,
www.yandex.ru,213.180.193.56,

When the ss file is loaded it records 'cname' in the A field, and the acutal CNAME in the AAAA field.

safeSearchDB[row[0]] = {'A': row[1], 'AAAA': row[2]}

https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749#file-pfb_unbound-py-L335

I am short on time today, but shoot me an email to bbcan177@gmail.com and I try to look at it over the weekend.

mrtumnus · Oct 14, 2022, 6:04 PM

@bbcan177 I suppose this change might fix the issue with duckduckgo being blocked entirely? I misunderstood and thought you were addressing the CNAME resolution issue, but that is an issue with Unbound itself (I think).

I can confirm that duckduckgo does load properly now.

Ir0nsh007er · Oct 25, 2023, 1:40 AM

This post is deleted!

Ir0nsh007er · Oct 25, 2023, 1:42 AM

Same probleme here.. it's already fix or not?

Gertjan · Oct 25, 2023, 6:32 AM

@Ir0nsh007er

If your still using a very old version of pfBlocker from before October 2022, then no.
Because you didn't update/upgrade.

If you did upgrade : then yes, fixed :

@mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

I can confirm that duckduckgo does load properly now.