Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Update to 3.2.0_7 breaks DNSBL

    Scheduled Pinned Locked Moved pfBlockerNG
    35 Posts 4 Posters 3.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      py
      last edited by

      I have not had a chance to review the links and will do so later today or this evening.

      ./pfb_dnsbl.sh restart
      Resulted in "ssl.cipher-list is deprecated. Please prefer lighttpd secure TLS defaults, etc..." with no errors or strange messages.
      I then enabled DCO and confirmed DNSBL was not blocking.
      I then ran:
      ./pfb_dnsbl.sh restart
      again and got the same result.
      I then disabled DCO and DNSBL still would not block. I noticed that I didn't have DNS resolution and checked the DNS Resolver log which was continuously priming many of the same domains for over 15 minutes . I restarted unbound and both DNS and DNSBL began working within a minute or two, however blocked domains are not always logged in DNSBL.

      This makes me suspect it may be more of a problem with unbound, or perhaps with how the VPN is handling DNS, than with DNSBL.

      1 Reply Last reply Reply Quote 0
      • P
        py
        last edited by

        @jrey I reviewed the linked posts and they clarified some things, thank you.

        After reviewing the DCO limitations I'm giving up on that until it's ready for prime time. At least one of the limitations makes it completely unsuitable for my environment and I should have been more careful before trying it out.

        Right now DNSBL is working, but it won't log many of the blocked domains. It doesn't seem to log any blocked domains from one machine in particular. It does seem to be logging all IPv4 blocks and Whitelisted domains. So far I've tried the following:

        1. Restarting pfb_dnsbl.sh as previously suggested;
        2. Restartng dnsbl from the Services page;
        3. Restarting unbound from the Services page;
        4. Disabling DNSBL and pfBlockerNG and reinstalling;
        5. Clearing all the log files and restarting DNSBL;
        6. Restarting DHCP.
          No change.
          At this point I'm considering disabling "Keep Settings", uninstalling pfBlockerNG, rebooting pfSense and reinstalling pfBlockerNG. But reconfiguring is at least 2 hours of work.

        Any suggestions welcome. You all have been a big help already and I've learned a few things.

        GertjanG 1 Reply Last reply Reply Quote 0
        • GertjanG
          Gertjan @py
          last edited by Gertjan

          @py said in Update to 3.2.0_7 breaks DNSBL:

          At this point I'm considering disabling "Keep Settings", uninstalling pfBlockerNG, rebooting pfSense and reinstalling pfBlockerNG. But reconfiguring is at least 2 hours of work

          Don't.
          If there is an issue, its not the code or the scripts.
          Your "pfSense", "unbound" and "pfBlockerng" is the same as mine. The same like 'every bit ....' where a bit is a bit in a byte.
          The only thing that is different between your pfSense and mine is : the settings ..... (that you want to keep).
          So, win 2 hours direct. Re installing is only a solution if you suspect that files are damaged, something that nearly never happens. If it happens, its often the disk dying on you.

          I'm using right now :

          b9a745b4-8479-476c-ab9c-b0c5fcd7637b-image.png

          I have pfBlockerng with some small DNSBL, and I can see that this

          tail -f /var/unbound/var/log/pfblockerng/dns_reply.log
          

          df6cf199-8f70-464a-add8-122dcfc4aff9-image.png

          keeps on scrolling - it never stops.
          When I open a browser on my phone - with the OpenVPN client activated and logged into pfSense, I can see that it also resolves DNS requests from my phone .... the dns_reply.log log scrolls even faster.

          Keep in mind : pfblockerng can only do its job for DNS requests it sees. That is, DNS records that are handled by unbound, the resolver. DNS traffic not handled by the resolver will not be handled by pfblockerng neither.

          I'm using the pfBlockerng 'pythion' mode (of course).

          My pfSense is a Intel x86 23.09 running on a '4100'.

          @py said in Update to 3.2.0_7 breaks DNSBL:

          Resulted in "ssl.cipher-list is deprecated. Please prefer lighttpd secure TLS defaults, etc..." with no errors or strange message

          and @jrey :

          I saw the lighttpd message.
          Good news : Google saw the message. Here is the info needed to blast the warning.

          /usr/local/pkg/pfblockerng/pfblockerng.inc

          The way pfblockerng is creating the lighttpd config file is a bit outdated.

          Look ,for the 4 (four !) occurrences of "cipher-list".

          Replace the lines with the more modern, according the redmine sited above.

          10d7c409-de6b-4651-b471-83ce1971a240-image.png

          Btw : now you're edting the source ..... if things go bad : reinstall pfBlockerng now makes sense ^^
          Btw : the message from lighttpd while (re) starting is not a problem. It's just a warning.

          No "help me" PM's please. Use the forum, the community will thank you.
          Edit : and where are the logs ??

          J 1 Reply Last reply Reply Quote 0
          • J
            jrey @Gertjan
            last edited by

            @Gertjan said in Update to 3.2.0_7 breaks DNSBL:

            the message from lighttpd while (re) starting is not a problem. It's just a warning.

            the cipher message in the file, is exactly what I said, not fatal don't worry about it. That wasn't the message causing other folks to have issues. The "other" messages if they are there "are not logged" so you can view them in a log file, and that's a problem. They do cause problems. By checking, the OP, has simply ruled those potential issues out.

            I agree, with the "Don't." - in regards to uninstalling/reinstalling.

            I disagree somewhat with this: "If there is an issue, its not the code or the scripts."
            There are literally 100s of issues, and unless there is an investigation, nothing will ever change.

            For example, the one line of code, that changed in pfBlockerNG _7 is being perceived by several as the cause of their "new" problem. Somewhat true, but not because of that line of code, as the scope of when it is executed is very limited.

            There is an issue, because of something else that gets exposed when the package is updated. (but the pfb code has changed). Specifically on a system, where the underlying system versions have changed. I can recreate this "issue" all day long (in a Break/fix test cycle). Does it have anything to do specifically with the change in pfBlockerNG _7, yes, but also no.
            Yes, but only because the update was applied to an underlying system that has changed. No because the code in pfBlockerNG that is causing these issues, hasn't changed, but is being impacted by what has changed under the hood.

            Does it impact logging? Very possible, however there is not enough data to make that determination conclusively yet. When "broken", logging stops /delays as described. But there are also "other" clues when it is "broken", the OP in this case doesn't seem to have any of those "other" clues. The lack of other clues, of course may or may not be a result of settings.

            P 1 Reply Last reply Reply Quote 0
            • P
              py @jrey
              last edited by py

              @jrey @Gertjan Okay, so I'm not reinstalling pfBlockerNG.

              Thanks for telling me about the dns_reply.log because mine is empty and stays that way.

              Hardware is an Intel Xeon E5-4667v4 with 18 cores on an ASRock X99 Taichi, and the NICs are Intel PRO/1000 on a PCI card.

              error.log logged 4 of these since I cleared the log yesterday:
              PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 12/6/23 03:16:41 ] Failed validation [ - ]

              dnsbl.log and unified.log frequently log a block that is not reflected in the GUI "Alerts" tab, and they never log a block from one of my workstations, even though the ping itself is blocked.

              Updates are set at 04:15, so the entries from the dnsbl_parsed_error.log must be happening during the update:
              dnsbl_parsed_error.JPG

              I don't see any errors in the VPN log and it seems normal to me.

              The DNS Resolver log frequently shows THROWAWAY responses which I don't recall seeing before this incident:
              DNS Resolver.JPG

              I tried python mode for DNSBL about a year ago but ran into trouble with that and my research showed those problems were inherent in the python implementation so I stopped using it since. I've long since forgotten the specifics.

              J 1 Reply Last reply Reply Quote 0
              • J
                jrey @py
                last edited by

                @py said in Update to 3.2.0_7 breaks DNSBL:

                PFB_FILTER - 6 | pfb_daemon_dnsbl_index [ 12/6/23 03:16:41 ] Failed validation [ - ]

                ^^ in the error.log and dnsbl_parsed_error (screen image) time stamps don't match.

                in the pfblockerng.log find this section in the latest processing
                ===[ DNSBL Domain/IP Counts ]

                what is the number for total ?

                and what do the numbers under this section look like
                pfSense Table Stats

                and then at very end what is the timestamp here ?
                UPDATE PROCESS ENDED [ 12/6/23 10:30:53 ]

                Can you share your complete DNS resolver settings, General and Advance.
                also the from the DNSBL config page. (just the sections DNSBL, DNSBL Webserver Config, and DNSBL Configuration)

                Thanks

                P 1 Reply Last reply Reply Quote 0
                • P
                  py @jrey
                  last edited by

                  @jrey Those timestamps were not intended to match.

                  ===[ DNSBL Domain/IP Counts ] ===================================

                  3632259 total

                  pfSense Table Stats

                  table-entries hard limit 400000
                  Table Usage Count 36389

                  UPDATE PROCESS ENDED [ 12/6/23 04:34:48 ]


                  DNS Resolver settings:
                  dns resolver 1.JPG
                  dns resolver 2.JPG dns resolver 3.JPG

                  dns advanced 1.JPG
                  dns advanced 2.JPG
                  dns advanced 3.JPG

                  DNSBL Settings:
                  dnsbl 1.JPG
                  dnsbl 2.JPG

                  Thanks!

                  J 1 Reply Last reply Reply Quote 0
                  • D
                    Draco @jrey
                    last edited by

                    @jrey said in Update to 3.2.0_7 breaks DNSBL:

                    on the top list, the 10.10.10.1 is not even a DNS server -- that is the DNSBL web page. Don't select that in your list. (I've actually created a patch on my system to just remove that from the list for testing, so in cases where "ALL" is selected it can't possibly be used. )

                    I don't suppose you'd care to share that patch? I have Unbound set to ALL so it's grabbing the 10.10.10.1 entry as well.

                    • draco
                    J 1 Reply Last reply Reply Quote 0
                    • J
                      jrey @Draco
                      last edited by

                      @Draco

                      Of course I could.

                      but a counter question to this statement

                      I have Unbound set to ALL

                      might be:
                      Why?

                      1 Reply Last reply Reply Quote 0
                      • J
                        jrey @py
                        last edited by

                        @py

                        if you want to before starting, you can clear pfblockerng.log and error.log on the
                        firewall > pfBlockerNG > Logs (select the log to display and there is a a trash can on the right) will just make it cleaning to see new entries

                        Firewall > pfBlockerNG > DNSBL
                        Change DNSBL Mode to "Unbound Python Mode"
                        for Options that appear after doing that

                        Select (Check Enable) DNS Reply Logging, DNSBL Blocking, HSTS Mode
                        Save DNSBL Settings

                        Then DNS Resolver config
                        Enable Python Mode (if not already check)
                        Module order is Pre Validator
                        Python Module Script is pfb_unbound
                        SAVE (button at bottom)
                        (then scroll back up top page) Apply Changes

                        Then Firewall > pfBlockerNG -> Update
                        Force -> Reload -> DNSBL

                        Process start time ?
                        Process End time?
                        new errors in either of the above mentioned log files ?

                        test and report the changes you see.

                        P 1 Reply Last reply Reply Quote 0
                        • P
                          py @jrey
                          last edited by

                          @jrey Applied and saved changes to DNSBL as specified. However, in DNS Resolver, the python module script is not available until after a force reload of DNSBL. So I did the force reload and checked DNS Resolver and those settings are as specified.

                          Force Reload start time: 9:16 am.
                          end time: 9:25
                          (MUCH faster than before!)

                          No errors in error.log or pfblockerng.log.
                          There are new errors in dnsbl_parsed_error.log, examples below:

                          • 12/7/23 09:21:32,Turkey_High_Risk,denizbank-cepsubem,denizbank-cepsubem
                            12/7/23 09:21:33,Turkey_High_Risk,gettask,gettask
                            12/7/23 09:21:33,Turkey_High_Risk,gettasks,gettasks
                            12/7/23 09:21:33,Turkey_High_Risk,gettasksize,gettasksize
                            12/7/23 09:21:33,Turkey_High_Risk,getupdates,getupdates
                            12/7/23 09:21:33,Turkey_High_Risk,godpore2,godpore2
                            12/7/23 09:21:33,Turkey_High_Risk,http,http
                            12/7/23 09:21:34,Turkey_High_Risk,instances,instances
                            12/7/23 09:21:34,Turkey_High_Risk,jqdownload,jqdownload
                            12/7/23 09:21:34,Turkey_High_Risk,merkezbankasi-com,merkezbankasi-com
                            12/7/23 09:21:35,Turkey_High_Risk,mobile-activity-site,mobile-activity-site
                            12/7/23 09:21:35,Turkey_High_Risk,p27dokhp,p27dokhp
                            12/7/23 09:21:35,Turkey_High_Risk,p27dokhpz2n7n,p27dokhpz2n7n
                            12/7/23 09:21:35,Turkey_High_Risk,pm,pm*

                          dns_reply.log is now continuously logging new entries as expected.

                          DNSBL Alerts GUI and dnsbl.log now in sync.

                          Hopefully this fixes it.

                          J 1 Reply Last reply Reply Quote 0
                          • J
                            jrey @py
                            last edited by

                            @py said in Update to 3.2.0_7 breaks DNSBL:

                            the python module script is not available until after a force reload of DNSBL

                            Okay - was't sure, I've flipped back and forth on the test box, I couldn't remember when they appears.

                            new errors in dnsbl_parsed_error.log

                            I don't specifically use that list. However all those names are being rejected from the list because they are not really complete. So for example

                            godpore.0pe.kr
                            godpore2
                            godpore2.0pe.ke
                            godpore2.0pe.kr
                            

                            the second entry in that block is rejected.

                            Others are the same

                            gettask
                            gettasks
                            gettasksize
                            

                            so just names, not domain names and therefore rejected by the list parser.
                            if I added "bob" to the list it would be rejected. but "bob.com" would not.

                            Nothing wrong here. But there is a lot of bloat in that particular list.

                            Should be good.

                            P 1 Reply Last reply Reply Quote 0
                            • P
                              py @jrey
                              last edited by py

                              @jrey Yes, that made a big difference, thank you.

                              However, I rebooted to test that everything would work and unbound seems to freeze. Every line in dns_reply.log had "servfail" in it, and the DNS Resolver status page seemed frozen in that the only thing changing was TTL and all the other columns had the same values. And the DNS Resolver system log was not showing anything happening. And my network had no internet access.

                              Restarting unbound fixes it. Rebooted again to confirm, and again restarting unbound fixes it.

                              I had this problem before and it might have been one of the reasons I stopped using the python module. At that time I installed Cron to put in a script to restart unbound at one minute after a reboot, which worked as I recall. That was at least 2-3 pfsense versions ago.

                              If you think that's what I should do here I will make it so.

                              I very much appreciate all the time and effort here, it's been a great help.

                              J 1 Reply Last reply Reply Quote 0
                              • J
                                jrey @py
                                last edited by jrey

                                @py

                                I would not switch back. I would take a look for messages related to unbound restarting and find out why.

                                start with System Logs -> System -> DNS Resolver
                                (filter on process unbound)

                                edit:
                                the servfail is most likely telling you that there is no upstream server responding to requests for names that are either not available or not cached.. but could also be several other things

                                need to narrow it down.

                                Client's talk to unbound (DNSBL) - then who is unbound talking to thru 03_PP_VPN_WAN if it can't satisfy the request? That's the outbound interface you have selected.

                                P 1 Reply Last reply Reply Quote 0
                                • P
                                  py @jrey
                                  last edited by

                                  @jrey Filtering for the process 'unbound' just gives me what it's already showing. I've tried
                                  'error'
                                  'fail' and
                                  'stop'
                                  with no entries showing for those. What else can I filter for?

                                  P 1 Reply Last reply Reply Quote 0
                                  • P
                                    py @py
                                    last edited by py

                                    This post is deleted!
                                    J 1 Reply Last reply Reply Quote 0
                                    • J
                                      jrey @py
                                      last edited by

                                      @py

                                      So there is also a stop right about the middle of that sequence.
                                      Are you sure that you are not just seeing the "static" IP's being added and then it restarts.
                                      You could test this by turning off the Static DHCP "Register DHCP static mappings in the DNS Resolver" That would/should only happen at boot for the static ones.

                                      my Resolver hasn't restarted since Nov 30, so restarts are normally rare.

                                      is the restart (at the top) you ? or the system restarting it when then process is done. That said how many static addresses are being added?

                                      P 2 Replies Last reply Reply Quote 0
                                      • P
                                        py @jrey
                                        last edited by py

                                        @jrey I'll have to test that later tonight, but I note the dhcp service starting is the same time as the unbound service issuing the "control cmd: reload", then stopping and restarting:
                                        (Dec 7 14:46:27 dhcpd 22906 Server starting service.)

                                        EDIT: The DNS log will remain static and as shown until I restart the service.

                                        Those logs are pristine immediately after a reboot, so nothing in there is done by me.

                                        I have 9 static addresses.

                                        1 Reply Last reply Reply Quote 0
                                        • P
                                          py @jrey
                                          last edited by py

                                          @jrey I deleted my post with the previous logs so I'm putting the sanitized versions here:
                                          py_error.txt
                                          DNS Resolver log.txt
                                          DHCP log.txt

                                          And the latest DHCP log here:
                                          DHCP log 2.txt
                                          The only thing changed for this log was to disable:
                                          "Register DHCP static mappings in the DNS Resolver".

                                          It is also very rare for my resolver to restart.

                                          Thanks,

                                          GertjanG 1 Reply Last reply Reply Quote 0
                                          • GertjanG
                                            Gertjan @py
                                            last edited by

                                            @py said in Update to 3.2.0_7 breaks DNSBL:

                                            py_error.txt

                                            My theory : your py_error.txt file is clear enough.
                                            While updating, pfBlockerng creates the two master DNSBL files from all your selected DNSBL feeds.
                                            One of the DNSVL source files you've chosen to uses has a fckd up format, and worse, pfBlockerng didn't detect it (wonder who it was).
                                            This will choke the entire internal DNS handling for unbound, as the two master files pfb_py_data.txt and pfb_py_zone.txt have a syntax error.

                                            Easy to repair : remove all your DNSBL, do a full reload, and check if the error are gone.
                                            You probably have to purge the py_error.txt file yourself.

                                            63b0c154-47d7-4c85-9e80-4025ccc62ccf-image.png

                                            Note : this log file should always stay empty. If there are lines in this fie, consider your system unusable and react right away (by undoing your last step, or de activating DNSBL etc).

                                            @py said in Update to 3.2.0_7 breaks DNSBL:

                                            "Register DHCP static mappings in the DNS Resolver"

                                            Will, just before unbound start, create a file /var/unbound/host_entries.conf, and this file get uses by unbound also.
                                            You can keep "Register DHCP static mappings in the DNS Resolver" at any time.
                                            Small reserve : entering invalid stuff as a static host name, or something like that, can have effects also.

                                            No "help me" PM's please. Use the forum, the community will thank you.
                                            Edit : and where are the logs ??

                                            P 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.