Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    How to unblock duckduckgo and find why it's being blocked.

    Scheduled Pinned Locked Moved pfBlockerNG
    21 Posts 10 Posters 18.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rjamesm
      last edited by

      Any fix on this problem? I don't want to disable safe search in order to fix this problem.

      1 Reply Last reply Reply Quote 0
      • P
        Patch
        last edited by

        @rjamesm said in How to unblock duckduckgo and find why it's being blocked.:

        I don't want to disable safe search

        My understanding is safe search is a Google function.
        So blocking Google but white listing safe search may work.
        A likely issue is Google rapidly changes what URL resolve to to enable them to harvest timing data. This can be blocked by overriding the minimum time in pfsense however white listing the specific Google sites you want to access is likely to require pfsense white lists also recent history (as client will use old IP addresses)

        1 Reply Last reply Reply Quote 0
        • mrtumnusM
          mrtumnus
          last edited by mrtumnus

          duckduckgo.com isn't being "blocked" by pfblockerng, per se. pfblockerng uses unbound to work its magic for DNSBL, including the Safe Search feature. It has a big list of search engine host overrides mapped to the IP address of the corresponding web host that enforces safe search for that engine. However, for duckduckgo.com (and pixabay.com), a CNAME is used to redirect each domain to a DNS name of the forced-safesearch host. This would work if unbound were setup as an authoritative name server, but it is not by default. So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

          I got around this by creating host overrides for duckduckgo.com to whatever the current IP of safe.duckduckgo.com is. This will break if the IP ever changes, but I have a feeling it should be stable for a while. I tried to mess around getting BIND to resolve the CNAME, but I couldn't figure it out. At least this is a temporary work-around.

          EDIT: Here's my host override in DNS Resolver:
          d9946561-0e96-46c7-8ad1-4a62a26f8195-image.png

          johnpozJ 1 Reply Last reply Reply Quote 0
          • johnpozJ
            johnpoz LAYER 8 Global Moderator @mrtumnus
            last edited by johnpoz

            @mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

            So, any DNS queries return only the CNAME with no A record, and no clients know how to resolve that.

            Huh? Maybe not understanding what your saying. But unbound doesn't have to be authoritative to follow a cname.. There was a fqdn that came up today with a 8 cname chain, which is insane and it resolves..

            ;logincdn.msauth.net.           IN      A
            
            ;; ANSWER SECTION:
            logincdn.msauth.net.    3600    IN      CNAME   lgincdn.trafficmanager.net.
            lgincdn.trafficmanager.net. 3600 IN     CNAME   lgincdnmsftuswe2.azureedge.net.
            lgincdnmsftuswe2.azureedge.net. 3600 IN CNAME   lgincdnmsftuswe2.afd.azureedge.net.
            lgincdnmsftuswe2.afd.azureedge.net. 3600 IN CNAME firstparty-azurefd-prod.trafficmanager.net.
            firstparty-azurefd-prod.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.t-msedge.net.
            dual.part-0023.t-0009.t-msedge.net. 3600 IN CNAME global-entry-afdthirdparty-fallback.trafficmanager.net.
            global-entry-afdthirdparty-fallback.trafficmanager.net. 3600 IN CNAME dual.part-0023.t-0009.fbs1-t-msedge.net.
            dual.part-0023.t-0009.fbs1-t-msedge.net. 3600 IN CNAME part-0023.t-0009.fbs1-t-msedge.net.
            part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.219.51
            part-0023.t-0009.fbs1-t-msedge.net. 3600 IN A   13.107.227.51
            
            ;; Query time: 390 msec
            ;; SERVER: 192.168.9.253#53(192.168.9.253)
            

            Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.11 | Lab VMs 2.8, 24.11

            mrtumnusM 1 Reply Last reply Reply Quote 0
            • mrtumnusM
              mrtumnus @johnpoz
              last edited by

              @johnpoz said in How to unblock duckduckgo and find why it's being blocked.:

              Are you trying to say if you create a cname via a local-data: entry in unbound the client will not resolve it correctly? That is true because clients like your windows box is stupid, that is not a problem with unbound perse

              Yes, that's what I'm saying. I realize an example would have been helpful. I was using not only Windows tools, but also some open source ones like dig (presuming that the unholy cygwin doesn't entirely rely on Windows DNS - maybe it does?). In addition, the Net Analyzer app on my phone only retrieved a CNAME record, no A or AAAA, and ping fails.

              The entry in pfblockerng is:

              local-data: "duckduckgo.com IN CNAME safe.duckduckgo.com"
              

              Are you suggesting that this results in successful DNS resolution on some clients? I'll have to try with dig oi a *nix environment.

              1 Reply Last reply Reply Quote 0
              • mrtumnusM
                mrtumnus
                last edited by

                dig-ging on a nix box produces the same result - no DNS resolution via browser, ping, etc. So, this is not a Windows problem.

                I took a look at what NextDNS does for its SafeSearch redirect. Here's the result:

                $ dig duckduckgo.com
                
                ; <<>> DiG 9.11.9 <<>> duckduckgo.com
                ;; global options: +cmd
                ;; Got answer:
                ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37077
                ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
                
                ;; QUESTION SECTION:
                ;duckduckgo.com.                        IN      A
                
                ;; ANSWER SECTION:
                duckduckgo.com.         300     IN      CNAME   safe.duckduckgo.com.
                safe.duckduckgo.com.    300     IN      A       40.89.244.237
                
                ;; Query time: 38 msec
                ;; SERVER: 192.168.77.71#53(192.168.77.71)
                ;; WHEN: Fri Jun 10 11:54:41 EDT 2022
                ;; MSG SIZE  rcvd: 67
                

                It would be great if we could get unbound to query the CNAME and provide the result as an A record, but from everything I'm reading it won't.

                1 Reply Last reply Reply Quote 0
                • T
                  Tom8
                  last edited by

                  I had the same problem.
                  With duckduckgo.com in the DNSBL whitelist it didn´t work, so I just added www.duckduckgo.com to the list.
                  Now it,s running.👍

                  BBcan177B 1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator @Tom8
                    last edited by

                    I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

                    Change here:
                    https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/revisions

                    From the Shell or pfSense Diag/Command Prompt:

                    cp /var/unbound/pfb_unbound.py /var/unbound/pfb_unbound.py.bk
                    
                    curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749/raw/pfb_unbound.py"
                    

                    Follow that with a Restart of Unbound. Might need to clear the OS and Browser cache.

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    mrtumnusM 1 Reply Last reply Reply Quote 0
                    • mrtumnusM
                      mrtumnus @BBcan177
                      last edited by mrtumnus

                      @bbcan177 said in How to unblock duckduckgo and find why it's being blocked.:

                      I have made a test file to hopefully address this issue. If it can be tested and reported back, will get this in the next upcoming release.

                      What difference should I expect the change to make? I'm not seeing any resolution of the CNAME in a dig query.

                      Looking at the changes, I'm confused what line 1181 is about:

                                      elif isSafeSearch['A'] == 'cname':
                      why this? >>>       if isSafeSearch['AAAA'] is not None and isSafeSearch['AAAA'] != '':
                                              if q_type == RR_TYPE_AAAA:
                      

                      Why does an AAAA element of isSafeSearch[] need to exist? It is not checked for any of the if/else sibling conditions. Could this be bypassing the CNAME handling?

                      BBcan177B 1 Reply Last reply Reply Quote 0
                      • BBcan177B
                        BBcan177 Moderator @mrtumnus
                        last edited by

                        @mrtumnus
                        The /var/unbound/pfb_py_ss.txt is formatted like:

                        duckduckgo.com,cname,safe.duckduckgo.com
                        www.duckduckgo.com,cname,safe.duckduckgo.com
                        pixabay.com,cname,safesearch.pixabay.com
                        www.pixabay.com,cname,safesearch.pixabay.com
                        yandex.ru,213.180.193.56,
                        www.yandex.ru,213.180.193.56,
                        

                        When the ss file is loaded it records 'cname' in the A field, and the acutal CNAME in the AAAA field.

                        safeSearchDB[row[0]] = {'A': row[1], 'AAAA': row[2]}
                        

                        https://gist.github.com/BBcan177/b95c2d9f5e78cf8ed902139dff600749#file-pfb_unbound-py-L335

                        I am short on time today, but shoot me an email to bbcan177@gmail.com and I try to look at it over the weekend.

                        "Experience is something you don't get until just after you need it."

                        Website: http://pfBlockerNG.com
                        Twitter: @BBcan177  #pfBlockerNG
                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                        mrtumnusM 1 Reply Last reply Reply Quote 1
                        • mrtumnusM
                          mrtumnus @BBcan177
                          last edited by

                          @bbcan177 I suppose this change might fix the issue with duckduckgo being blocked entirely? I misunderstood and thought you were addressing the CNAME resolution issue, but that is an issue with Unbound itself (I think).

                          I can confirm that duckduckgo does load properly now.

                          1 Reply Last reply Reply Quote 0
                          • GertjanG Gertjan referenced this topic on
                          • GertjanG Gertjan referenced this topic on
                          • I
                            Ir0nsh007er
                            last edited by

                            This post is deleted!
                            1 Reply Last reply Reply Quote 0
                            • I
                              Ir0nsh007er
                              last edited by

                              Same probleme here.. it's already fix or not?

                              GertjanG 1 Reply Last reply Reply Quote 0
                              • GertjanG
                                Gertjan @Ir0nsh007er
                                last edited by

                                @Ir0nsh007er

                                If your still using a very old version of pfBlocker from before October 2022, then no.
                                Because you didn't update/upgrade.

                                If you did upgrade : then yes, fixed :

                                @mrtumnus said in How to unblock duckduckgo and find why it's being blocked.:

                                I can confirm that duckduckgo does load properly now.

                                No "help me" PM's please. Use the forum, the community will thank you.
                                Edit : and where are the logs ??

                                1 Reply Last reply Reply Quote 2
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.