Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    How to unblock duckduckgo and find why it's being blocked.

    Scheduled Pinned Locked Moved pfBlockerNG
    21 Posts 10 Posters 18.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • provelsP
      provels @thewismit
      last edited by

      @thewismit
      OK, it turns out although https://start.duckduckgo.com/ will load, any search from there fails. I saw a block in the DNSBL log blocking improving.duckduckgo.com blocked by https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt if you happen to use that,. Added .duckduckgo.com to the DNSBL Whitelist w/o benefit. So short story is I don't know. I ran a quick packet capture, but didn't see anything obvious. Maybe you can try that.

      Peder

      MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
      BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

      1 Reply Last reply Reply Quote 0
      • thewismitT
        thewismit
        last edited by

        Thanks @provels. I don't even know how you found improving.duckduckgo.com in the DNSBL log. I've tried finding anything and haven't been successful beyond IP addresses so far.

        I also have no idea how to read a packet capture yet but I do appreciate you trying and replying.

        1 Reply Last reply Reply Quote 0
        • provelsP
          provels
          last edited by

          Perhaps @BBcan177 can take a look when convenient. It seems SafeSearch should work with DDG, at least according to the description on the 'SafeSearch Redirection' line item.

          Peder

          MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
          BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

          1 Reply Last reply Reply Quote 1
          • R
            rjamesm
            last edited by

            Any fix on this problem? I don't want to disable safe search in order to fix this problem.

            1 Reply Last reply Reply Quote 0
            • P
              Patch
              last edited by

              @rjamesm said in How to unblock duckduckgo and find why it's being blocked.:

              I don't want to disable safe search

              My understanding is safe search is a Google function.
              So blocking Google but white listing safe search may work.
              A likely issue is Google rapidly changes what URL resolve to to enable them to harvest timing data. This can be blocked by overriding the minimum time in pfsense however white listing the specific Google sites you want to access is likely to require pfsense white lists also recent history (as client will use old IP addresses)

              1 Reply Last reply Reply Quote 0
              • mrtumnusM
                mrtumnus
                last edited by mrtumnus

                duckduckgo.com isn't being "blocked" by pfblockerng, per se. pfblockerng uses unbound to work its magic for DNSBL, including the Safe Search feature. It has a big list of search engine host overrides mapped to the IP address of the corresponding web host that enforces safe search for that engine. However, for duckduckgo.com (and pixabay.com), a CNAME is used to redirect each domain to a DNS name of the forced-safesearch host. This would work if unbound were setup as an authoritative name server, but it is not by default. So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

                I got around this by creating host overrides for duckduckgo.com to whatever the current IP of safe.duckduckgo.com is. This will break if the IP ever changes, but I have a feeling it should be stable for a while. I tried to mess around getting BIND to resolve the CNAME, but I couldn't figure it out. At least this is a temporary work-around.

                EDIT: Here's my host override in DNS Resolver:
                d9946561-0e96-46c7-8ad1-4a62a26f8195-image.png

                johnpozJ 1 Reply Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator @mrtumnus
                  last edited by johnpoz

                  @mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

                  So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

                  Huh? Maybe not understanding what your saying. But unbound doesn't have to be authoritative to follow a cname.. There was a fqdn that came up today with a 8 cname chain, which is insane and it resolves..

                  ;logincdn.msauth.net.           IN      A
                  
                  ;; ANSWER SECTION:
                  logincdn.msauth.net.    3600    IN      CNAME   lgincdn.trafficmanager.net.
                  lgincdn.trafficmanager.net. 3600 IN     CNAME   lgincdnmsftuswe2.azureedge.net.
                  lgincdnmsftuswe2.azureedge.net. 3600 IN CNAME   lgincdnmsftuswe2.afd.azureedge.net.
                  lgincdnmsftuswe2.afd.azureedge.net. 3600 IN CNAME firstparty-azurefd-prod.trafficmanager.net.
                  firstparty-azurefd-prod.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.t-msedge.net.
                  dual.part-0023.t-0009.t-msedge.net. 3600 IN CNAME global-entry-afdthirdparty-fallback.trafficmanager.net.
                  global-entry-afdthirdparty-fallback.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.fbs1-t-msedge.net.
                  dual.part-0023.t-0009.fbs1-t-msedge.net. 3600 IN CNAME part-0023.t-0009.fbs1-t-msedge.net.
                  part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.219.51
                  part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.227.51
                  
                  ;; Query time: 390 msec
                  ;; SERVER: 192.168.9.253#53(192.168.9.253)
                  

                  Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                  mrtumnusM 1 Reply Last reply Reply Quote 0
                  • mrtumnusM
                    mrtumnus @johnpoz
                    last edited by

                    @johnpoz said in How to unblock duckduckgo and find why it's being blocked.:

                    Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

                    Yes, that's what I'm saying. I realize an example would have been helpful. I was using not only Windows tools, but also some open source ones like dig (presuming that the unholy cygwin doesn't entirely rely on Windows DNS - maybe it does?). In addition, the Net Analyzer app on my phone only retrieved a CNAME record, no A or AAAA, and ping fails.

                    The entry in pfblockerng is:

                    local-data: "duckduckgo.com IN CNAME safe.duckduckgo.com"
                    

                    Are you suggesting that this results in successful DNS resolution on some clients? I'll have to try with dig oi a *nix environment.

                    1 Reply Last reply Reply Quote 0
                    • mrtumnusM
                      mrtumnus
                      last edited by

                      dig-ging on a nix box produces the same result - no DNS resolution via browser, ping, etc. So, this is not a Windows problem.

                      I took a look at what NextDNS does for its SafeSearch redirect. Here's the result:

                      $ dig duckduckgo.com
                      
                      ; <<>> DiG 9.11.9 <<>> duckduckgo.com
                      ;; global options: +cmd
                      ;; Got answer:
                      ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37077
                      ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
                      
                      ;; QUESTION SECTION:
                      ;duckduckgo.com.                        IN      A
                      
                      ;; ANSWER SECTION:
                      duckduckgo.com.         300     IN      CNAME   safe.duckduckgo.com.
                      safe.duckduckgo.com.    300     IN      A       40.89.244.237
                      
                      ;; Query time: 38 msec
                      ;; SERVER: 192.168.77.71#53(192.168.77.71)
                      ;; WHEN: Fri Jun 10 11:54:41 EDT 2022
                      ;; MSG SIZE  rcvd: 67
                      

                      It would be great if we could get unbound to query the CNAME and provide the result as an A record, but from everything I'm reading it won't.

                      1 Reply Last reply Reply Quote 0
                      • T
                        Tom8
                        last edited by

                        I had the same problem.
                        With duckduckgo.com in the DNSBL whitelist it didn´t work, so I just added www.duckduckgo.com to the list.
                        Now it,s running.👍

                        BBcan177B 1 Reply Last reply Reply Quote 0
                        • BBcan177B
                          BBcan177 Moderator @Tom8
                          last edited by

                          I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

                          Change here:
                          https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/revisions

                          From the Shell or pfSense Diag/Command Prompt:

                          cp /var/unbound/pfb_unbound.py /var/unbound/pfb_unbound.py.bk
                          
                          curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/raw/pfb_unbound.py"
                          

                          Follow that with a Restart of Unbound. Might need to clear the OS and Browser cache.

                          "Experience is something you don't get until just after you need it."

                          Website: http://pfBlockerNG.com
                          Twitter: @BBcan177  #pfBlockerNG
                          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                          mrtumnusM 1 Reply Last reply Reply Quote 0
                          • mrtumnusM
                            mrtumnus @BBcan177
                            last edited by mrtumnus

                            @bbcan177 said in How to unblock duckduckgo and find why it's being blocked.:

                            I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

                            What difference should I expect the change to make? I'm not seeing any resolution of the CNAME in a dig query.

                            Looking at the changes, I'm confused what line 1181 is about:

                                            elif isSafeSearch['A'] == 'cname':
                            why this? >>>       if isSafeSearch['AAAA'] is not None and isSafeSearch['AAAA'] != '':
                                                    if q_type == RR_TYPE_AAAA:
                            

                            Why does an AAAA element of isSafeSearch[] need to exist? It is not checked for any of the if/else sibling conditions. Could this be bypassing the CNAME handling?

                            BBcan177B 1 Reply Last reply Reply Quote 0
                            • BBcan177B
                              BBcan177 Moderator @mrtumnus
                              last edited by

                              @mrtumnus
                              The /var/unbound/pfb_py_ss.txt is formatted like:

                              duckduckgo.com,cname,safe.duckduckgo.com
                              www.duckduckgo.com,cname,safe.duckduckgo.com
                              pixabay.com,cname,safesearch.pixabay.com
                              www.pixabay.com,cname,safesearch.pixabay.com
                              yandex.ru,213.180.193.56,
                              www.yandex.ru,213.180.193.56,
                              

                              When the ss file is loaded it records 'cname' in the A field, and the acutal CNAME in the AAAA field.

                              safeSearchDB[row[0]] = {'A': row[1], 'AAAA': row[2]}
                              

                              https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749#file-pfb_unbound-py-L335

                              I am short on time today, but shoot me an email to bbcan177@gmail.com and I try to look at it over the weekend.

                              "Experience is something you don't get until just after you need it."

                              Website: http://pfBlockerNG.com
                              Twitter: @BBcan177  #pfBlockerNG
                              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                              mrtumnusM 1 Reply Last reply Reply Quote 1
                              • mrtumnusM
                                mrtumnus @BBcan177
                                last edited by

                                @bbcan177 I suppose this change might fix the issue with duckduckgo being blocked entirely? I misunderstood and thought you were addressing the CNAME resolution issue, but that is an issue with Unbound itself (I think).

                                I can confirm that duckduckgo does load properly now.

                                1 Reply Last reply Reply Quote 0
                                • GertjanG Gertjan referenced this topic on
                                • GertjanG Gertjan referenced this topic on
                                • I
                                  Ir0nsh007er
                                  last edited by

                                  This post is deleted!
                                  1 Reply Last reply Reply Quote 0
                                  • I
                                    Ir0nsh007er
                                    last edited by

                                    Same probleme here.. it's already fix or not?

                                    GertjanG 1 Reply Last reply Reply Quote 0
                                    • GertjanG
                                      Gertjan @Ir0nsh007er
                                      last edited by

                                      @Ir0nsh007er

                                      If your still using a very old version of pfBlocker from before October 2022, then no.
                                      Because you didn't update/upgrade.

                                      If you did upgrade : then yes, fixed :

                                      @mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

                                      I can confirm that duckduckgo does load properly now.

                                      No "help me" PM's please. Use the forum, the community will thank you.
                                      Edit : and where are the logs ??

                                      1 Reply Last reply Reply Quote 2
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.