Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfBlockerNG not blocking everything in blocklists

    Scheduled Pinned Locked Moved pfBlockerNG
    21 Posts 3 Posters 4.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @vjizzle
      last edited by

      I see what you mean :

      @vjizzle said in pfBlockerNG not blocking everything in blocklists:

      DNS-reply,Apr 6 12:48:40,cache,A,A,86385,www.conduit.com,192.168.100.25,198.49.162.233,IL

      dono where and when unbound (?) resolves "www.conduit.com".

      Check these files :
      dnsbl_error.log (mine is empty)
      py_error.log (mine is empty)
      dnsbl_parsed_error.log (some data, not related to this issue)

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      V 1 Reply Last reply Reply Quote 0
      • V
        vjizzle @Gertjan
        last edited by vjizzle

        @gertjan
        Yes for some reason every time I run the GRC lookup tool and query my pfSense, pfblockerng is broken afterwards. I can reproduce it every time and when I restart unbound server it is fixed.

        I am using unbound on pfSense in resolver mode (forwarding option is disabled) and I have no DNS server entries in General Setup tab in pfSense. I have set the option DNS Resolution Behavior to Use local DNS (ignore remote DNS servers).

        py_error.log is empty.
        dnsbl_error.log shows a few entries

        [ DNSBL_Abuse_CH - Abuse_ch_hostfile ] Download Fail
 Firewall and/or IDS (Legacy mode only) are not blocking download.

 Restoring previously downloaded file


[ DNSBL_Abuse_CH - Abuse_ch_hostfile ] Download Fail
 Firewall and/or IDS (Legacy mode only) are not blocking download.

 Restoring previously downloaded file

        dnsbl_parsed_error.log has some entries like these:

        04/6/21 14:16:12,No_Tracking_Hostnames,from,:: from
04/6/21 14:16:12,No_Tracking_Hostnames,hostwinds,0.0.0.0 hostwinds
04/6/21 14:16:12,No_Tracking_Hostnames,hostwinds,:: hostwinds
04/6/21 14:16:30,MS_2,beritanowtk,beritanowtk
        GertjanG 1 Reply Last reply Reply Quote 0
        • GertjanG
          Gertjan @vjizzle
          last edited by

          Hummm.

          Everything looks fine to me / identical to me.

          Except : I can run that GRC tool as hard as I can, it won't break 'unbound'.

          Did you try the old method "unbound mode" ?

          No "help me" PM's please. Use the forum, the community will thank you.
          Edit : and where are the logs ??

          V 1 Reply Last reply Reply Quote 0
          • V
            vjizzle @Gertjan
            last edited by

            @gertjan
            Good tip @Gertjan. I immediately tested it and low-and-behold. Everything keeps working even if I bombard pfsense with the GRC benchmarking tool. Here the output before and after the GRC testing:

            > server 192.168.100.1
Default Server:  odin.vXXXX.lan
Address:  192.168.100.1

> www.conduit.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.conduit.com
Address:  10.10.10.1

> www.vmn.net
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.vmn.net
Address:  10.10.10.1

> www.ku6.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    www.ku6.com
Address:  10.10.10.1

>

            Looks like there is something the matter with the python integration here. Also as an added bonus in Unbound mode I can see that pfSense is now faster in answering queries compared to the AdGuard home server I have running here. In python mode pfSense was always slower:

            screen05.jpg

            Not that the speed difference is that big but still ^^. So in the end something is broken here. I will initiate a reinstall tomorrow unless someone else has another good idea. Thank you very. much for helping Gertjan!

            1 Reply Last reply Reply Quote 0
            • V
              vjizzle
              last edited by

              Ok I just did a clean install of pfsense and then restored a backup. I have the exact same behavior again. Python mode pfblockerng stops working after I hammer it with GRC DNS benchmark.

              For now I am switching over to AdGuard Home. Maybe better luck with pfSense in the next release :(

              GertjanG 1 Reply Last reply Reply Quote 0
              • GertjanG
                Gertjan @vjizzle
                last edited by Gertjan

                I'm at home now.

                Did some testing.
                The dns benchmark running, doing it thing.

                In a console/SSH, I ran this :

                tail -f /var/unbound/var/log/pfblockerng/dns_reply.log | grep 'conduit'

                According the https://dbl.oisd.nl/ feed ( the oisd.nl feed) :

                ,www.conduit.com,,0,OISD,DNSBL_Compilation

                and also :

                ,conduit.com,,0,MS_2,DNSBL_BBcan177

                I was spamming

                www.conduit.com

                in a nslookup (Windows 10) session.

                Suddenly, during the test :

                ......
> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

> www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1

Nom :    www.conduit.com
Address:  0.0.0.0

                When the 0.0.0.0 came back, I saw this in the 'tail' console :

                DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:08,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:09,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:10,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk
DNS-reply,Apr 6 23:59:11,servfail,AAAA,AAAA,Unk,www.conduit.com,2001:470:1f13:65c::2001,ServFail,unk

                and then :

                > www.conduit.com.
Serveur :   priv.my-domain.net
Address:  2001:470:xxxx:65c::1
Réponse ne faisant pas autorité :
Nom :    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

                DNS-reply,Apr 6 23:59:12,cache,A,A,86399,www.conduit.com,2001:470:xxxx:65c::2001,198.49.162.233,IL

                198.49.162.233 popped up.

                I guess I can confirm what you saw.

                Btw :

                I saw :

                Assembling DNSBL database...... completed [ 04/7/21 00:05:47 ]
TLD:
TLD analysis..xxxxxxxxxxx completed [ 04/7/21 00:05:51 ]

  ** TLD Domain count exceeded. [ 200000 ] All subsequent Domains listed as-is **

                during the force update. The OISD feed is BIG : 1,2 million domain names.

                I'll test tommorow agagin without the OISD feed, as the base domain name itself "conduit.com" is already in the BBCan 77 feed - as shown above.

                edit : without the oisd.nl feed, and thus a rather (for me) small set of blocked domains, env 16962, the "0.0.0.0" blocking of "conduit.com" works.
                www.conduit.com is resolved.....

                No "help me" PM's please. Use the forum, the community will thank you.
                Edit : and where are the logs ??

                1 Reply Last reply Reply Quote 0
                • V
                  vjizzle
                  last edited by vjizzle

                  Interesting! So maybe I am on to something :p.

                  I got curious and did the test with the BBCan77 feed. The domain conduit.com is indeed there. So I decided to disable the oisd.nl feed and force reload pfblockerNG. After that I bombarded pfSense with the DNS benchmarking tool. Guess what...everything keeps running fine for conduit.com. Multiple reloads of pfBlockerNG, multiple restart of Unbound but everything kept working as expected:

                  > conduit.com
Server:  odin.vXXXXX.lan
Address:  192.168.100.1

Name:    conduit.com
Address:  0.0.0.0

                  Then I enabled the https://dbl.oisd.nl feed did a force reload of pfBlockerNG. I don't have TLD blocking enabled so I don't get that message you have. You can see the log about loading the oisd.nl list:

                  [ dbl_list ]			 Downloading update .. 200 OK.
  ----------------------------------------------------------------------
  Orig.    Unique     # Dups     # White    # TOP1M    Final                
  ----------------------------------------------------------------------
  1251871  1251871    154015     0          0          1097856              
  ----------------------------------------------------------------------

Saving DNSBL statistics... completed [ 04/7/21 00:31:48 ]
------------------------------------------------------------------------
Assembling DNSBL database...... completed [ 04/7/21 00:31:49 ]
Reloading Unbound Resolver (DNSBL python).
Stopping Unbound Resolver.
Unbound stopped in 2 sec.
Additional mounts (DNSBL python):
  No changes required.
Starting Unbound Resolver... completed [ 04/7/21 00:31:53 ]
Resolver cache restored
DNSBL update [ 1379690 | PASSED  ]... completed

                  And of course now pfBlockerNG is broken and not blocking the domain anymore BUT conduit.com keeps getting blocked by pfblockerng fine:

                  > server 192.168.100.1
Default Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

> conduit.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Name:    conduit.com
Address:  0.0.0.0

> www.conduit.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

> www.ku6.com
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    3a1d905b3d90ef4e-cl2.qcloudcjgj.com
Address:  118.89.206.86
Aliases:  www.ku6.com
          fa2a686c758fa3d110164df993a0b8c4.qcloudzygj.com

> www.vmn.net
Server:  odin.vXXXXXX.lan
Address:  192.168.100.1

Non-authoritative answer:
Name:    www.vmn.net
Address:  54.146.142.3

>

                  So www.conduit.com is broken but conduit.com is being blocked fine. Even after I run the DNS benchmark tool multiple times conduit.com keeps working.

                  I am glad you are seeing the same behavior like me. Looks like something is crashing when stressed with multiple (and fast) DNS queries .

                  GertjanG 1 Reply Last reply Reply Quote 0
                  • GertjanG
                    Gertjan @vjizzle
                    last edited by

                    Same test, back on work.
                    Home is a 1 Mbytes 'small' VM, work use its own i5, with 8 Mbyyes - although I don't think this matters.

                    First test :
                    nslookup with full debug :

                    set db2

                    > www.conduit.com.
Serveur :   pfsense.my-work-domain.net
Address:  2001:470:xxxx:5c0:2::1

------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 20, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN

------------
------------
Got answer (49 bytes):
    HEADER:
        opcode = QUERY, id = 20, rcode = NOERROR
        header flags:  response, auth. answer, want recursion, recursion avail.
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN
    ANSWERS:
    ->  www.conduit.com
        type = A, class = IN, dlen = 4
        internet address = 0.0.0.0
        ttl = 60 (1 min)

------------
------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 21, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
------------
Got answer (33 bytes):
    HEADER:
        opcode = QUERY, id = 21, rcode = SERVFAIL
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
Nom :    www.conduit.com
Address:  0.0.0.0

                    Fine, "www.conduit.com" yields 0.0.0.0

                    Now I launch a DNS benchmark, and have it finished (30 seconds later)

                    I "nslookup" again :

                    > www.conduit.com.
Serveur :   pfsense.my-work-domain.net
Address:  2001:470:xxxx:5c0:2::1

------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 22, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN

------------
------------
Got answer (70 bytes):
    HEADER:
        opcode = QUERY, id = 22, rcode = NOERROR
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 2,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = A, class = IN
    ANSWERS:
    ->  www.conduit.com
        type = CNAME, class = IN, dlen = 9
        canonical name = www.va.conduit.com
        ttl = 86227 (23 hours 57 mins 7 secs)
    ->  www.va.conduit.com
        type = A, class = IN, dlen = 4
        internet address = 198.49.162.233
        ttl = 86227 (23 hours 57 mins 7 secs)

------------
Réponse ne faisant pas autorité :
------------
SendRequest(), len 33
    HEADER:
        opcode = QUERY, id = 23, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
------------
Got answer (33 bytes):
    HEADER:
        opcode = QUERY, id = 23, rcode = SERVFAIL
        header flags:  response, want recursion, recursion avail.
        questions = 1,  answers = 0,  authority records = 0,  additional = 0

    QUESTIONS:
        www.conduit.com, type = AAAA, class = IN

------------
Nom :    www.va.conduit.com
Address:  198.49.162.233
Aliases:  www.conduit.com

                    Note the difference :
                    www.conduit.com resolves to the CNAME "www.va.conduit.com" (domain not listed), and that one resolves to "198.49.162.233"

                    No "help me" PM's please. Use the forum, the community will thank you.
                    Edit : and where are the logs ??

                    1 Reply Last reply Reply Quote 0
                    • V
                      vjizzle
                      last edited by vjizzle

                      Hey Gertjan. This is correct about eventually resolving to a CNAME. I can see it on my end also that after running the GRC benchmark testing tool the domain resolves to a CNAME (www.va.conduit.com) and so pfblockerNG is not blocking it anymore. I think the same thing is happening with "www.ku6.com" and "www.vmn.net".

                      Can we conclude that the CNAME blocking on pfBlockerNG is not working as expected?
                      Because the same test on AdguardHome server always gets the domains blocked. I think that is also the reason why I am getting the low percentage in the pfblockerNG dashboard widget (AdguardHome is at 10% blocked DNS requests at the moment).

                      I also think the same thing is happening with other domains as well. Just found out that "diagnostics.meethue.com" gets blocked but after benchmarking with GRC it is not blocked anymore.
                      I have never seen "diagnostics.meethue.com" in the pfblockerNG DNSBL Block Stats tab being in the list. After running my whole network for 24 hrs on AdGuardhome it is the domain being blocked the most.

                      BBcan177B 1 Reply Last reply Reply Quote 0
                      • BBcan177B
                        BBcan177 Moderator @vjizzle
                        last edited by

                        @vjizzle

                        Thanks for the report! Its appreciated!

                        The issue is that GRC is using a mixture of lowercase and uppercase in their DNS Tests.

                        I have fixed the code to force everything to lowercase before the domains are validated.

                        A fix can be downloaded here until the next version is available:

                        curl -o /var/unbound/pfb_unbound.py "https://gist.githubusercontent.com/BBcan177/1e400f170b7ef7e8cb76b548d6524b31/raw"

                        Then Restart Unbound

                        Thanks!

                        "Experience is something you don't get until just after you need it."

                        Website: http://pfBlockerNG.com
                        Twitter: @BBcan177  #pfBlockerNG
                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                        V 1 Reply Last reply Reply Quote 3
                        • V
                          vjizzle @BBcan177
                          last edited by

                          @bbcan177

                          Thank you for being so fast in fixing this and the great work you are doing with pfBlockerNG!

                          With the modifications all is working well :).

                          1 Reply Last reply Reply Quote 3
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.