Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Periodic since 2.2 pages load blank, certs invalid

    Scheduled Pinned Locked Moved General pfSense Questions
    126 Posts 14 Posters 46.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

      Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

      I would put this on LAN:

      pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log

      Put that above your normal pass rule.  If everything is as you say, it should log nothing.

      On pfSense 2.2 you should be able to set the dest to ! This Firewall (self).

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • D
        doktornotor Banned
        last edited by

        @Derelict:

        Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

        Pretty sure I could get these guys involved in investigating the issue here (they've also written the Knot DNS server so I'm rather convinced they are familiar with DNS  :P) – however that'd require either remote access or at least uncensored traffic captures. Not example.com -- totally useless.

        1 Reply Last reply Reply Quote 0
        • T
          Trel
          last edited by

          @doktornotor:

          @Derelict:

          If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

          Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

          Using Comcast with a modem only (not a gateway in bridged mode).  Here's the block rule.
          With these settings, if I try to look up the domain I get this scenario

          When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
          -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
          (persists until unbound service is restarted)

          I understand that an infected machine should not be on the network, but if a mere typical DNS lookup can cause this much havoc, then something is really wrong.

          restricted_dns.gif
          restricted_dns.gif_thumb

          1 Reply Last reply Reply Quote 0
          • S
            swix
            last edited by

            @Derelict:

            Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.
            Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

            It wasn't obfuscated, it really looked like that… (also with other domains, juste replace example.com by anything)

            @Derelict:

            I would put this on LAN:
            pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log
            Put that above your normal pass rule.  If everything is as you say, it should log nothing.

            Ok, thanks, will setup this.

            @doktornotor:

            Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

            I would be really happy to know the cause, it is really strange that Trel is having a similar problem with the very same target IP "195.22.26.248", especially from different countries/ISP's.  The only recent change to our infrastructure was upgrading to pfSense 2.2 at the beginning of January, otherwise nothing special.  But I'll setup some network monitoring tools later this week.

            Best regards

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              @Trel:

              When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
              -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
              (persists until unbound service is restarted)

              A search of redmine does not show that as an open issue.  Have you reported it?

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • 2
                2chemlud Banned
                last edited by

                @Trel:

                When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                (persists until unbound service is restarted)[

                [/quote]

                I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  I'm not in a position to test this at the moment.  Tonight.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • T
                    Trel
                    last edited by

                    @2chemlud:

                    @Trel:

                    When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                    -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                    (persists until unbound service is restarted)

                    I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

                    When you say interruptions, could those have been unbound not responding?
                    Someone did mention that one of the times I was unable to restart the service manually (as I was not available) it began working again after 45-50 minutes.

                    Either way though, as soon as I overrode the DNS for those sites, it's never happened again.

                    @Derelict:

                    I'm not in a position to test this at the moment.  Tonight.

                    If you're going to test, try accessing and resolving

                    api-nyc01.exip.org

                    and

                    ns3.csof.net
                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything

                      So I can test it properly, to what domains is this referring?

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        @Trel:

                        If you're going to test, try accessing and resolving

                        api-nyc01.exip.org

                        and

                        ns3.csof.net

                        What about it?

                        dig . ns

                        Pick a root server at random:

                        dig @e.root-servers.net ns3.csof.net

                        Get a list of gtld servers.  Pick one at random:

                        dig @e.gtld-servers.net ns3.csof.net

                        Pertinent info:

                        
;; AUTHORITY SECTION:
csof.net.		172800	IN	NS	ns61.domaincontrol.com.
csof.net.		172800	IN	NS	ns62.domaincontrol.com.

;; ADDITIONAL SECTION:
ns61.domaincontrol.com.	172800	IN	A	216.69.185.32
ns62.domaincontrol.com.	172800	IN	A	208.109.255.32

                        Pick one of those:

                        dig @216.69.185.32 ns3.csof.net

                        
;; QUESTION SECTION:
;ns3.csof.net.			IN	A

;; ANSWER SECTION:
ns3.csof.net.		600	IN	A	*** 195.22.26.199 ***

                        Their name servers either want that name to resolve to 195.22.26.199 or are giving bogus information or are otherwise hacked.  What, exactly, would you expect unbound to do to fix that?

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • T
                          Trel
                          last edited by

                          Those domains aren't the problem.

                          The problem is what happens AFTER looking them up.

                          I can see there's some failure to communicate here.

                          Problem 1:
                          Without DNSSEC on and with other DNS servers allowed, a few minutes after looking up those domains, looking up google.com will return something in the 195.22.x range (persisting until I restarted unbound, or possibly 45 minutes to an hour)

                          At this point I switched to DNSSEC and blocked all outgoing DNS except to the firewall.

                          Problem 2:
                          With DNSSEC enabled and only unbound able to resolve, a few minutes after looking up those domains, lookup up google.com will return nothing the result will be blank as if the domain didn't exist (persisting until I restarted unbound, or possibly 45 minutes to an hour)

                          At this point, I put DNS overrides in for those domains setting them to 0.0.0.0 so they would not be able to be looked up at all.

                          At this point, the symptoms have stopped.

                          1 Reply Last reply Reply Quote 0
                          • DerelictD
                            Derelict LAYER 8 Netgate
                            last edited by

                            Ok I just let unbound look them up.

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              Start listening to these guys.  There's something strange and it looks like it's in unbound.

                              
$ dig @192.168.223.1 www.google.com

; <<>> DiG 9.8.3-P1 <<>> @192.168.223.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 53699
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.			IN	A

;; Query time: 81 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 12:23:46 2015
;; MSG SIZE  rcvd: 32

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by

                                After bouncing unbound:

                                
$ dig @192.168.223.1 www.google.com

; <<>> DiG 9.8.3-P1 <<>> @192.168.223.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54480
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com.			IN	A

;; ANSWER SECTION:
www.google.com.		300	IN	A	216.58.216.36

;; Query time: 1333 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Feb  9 12:26:36 2015
;; MSG SIZE  rcvd: 48

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • 2
                                  2chemlud Banned
                                  last edited by

                                  @Trel:

                                  @2chemlud:

                                  @Trel:

                                  When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                                  -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                                  (persists until unbound service is restarted)

                                  I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

                                  When you say interruptions, could those have been unbound not responding?
                                  Someone did mention that one of the times I was unable to restart the service manually (as I was not available) it began working again after 45-50 minutes.

                                  ….

                                  My thread is this here:

                                  https://forum.pfsense.org/index.php?topic=88272

                                  :-)

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    Trel
                                    last edited by

                                    @Derelict:

                                    Start listening to these guys.  There's something strange and it looks like it's in unbound.

                                    I'm guessing it happened a few moments after doing a lookup on one of the domains I mentioned?

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate
                                      last edited by

                                      Yup.

                                      Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames.  I'm doing it one more time at unbound log level 5.

                                      ETA: Ironically, I had to kill unbound to post this because:

                                      $ dig forum.pfsense.org

                                      ; <<>> DiG 9.8.3-P1 <<>> forum.pfsense.org
                                      ;; global options: +cmd
                                      ;; Got answer:
                                      ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 30471
                                      ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

                                      ;; QUESTION SECTION:
                                      ;forum.pfsense.org. IN A

                                      ;; Query time: 1781 msec
                                      ;; SERVER: 192.168.223.1#53(192.168.223.1)
                                      ;; WHEN: Mon Feb  9 17:46:41 2015
                                      ;; MSG SIZE  rcvd: 35

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        kejianshi
                                        last edited by

                                        Have you guys considered setting the advanced settings that prevent excessive replies and purge things when excessive replies and poisoning is possibly happening?

                                        1 Reply Last reply Reply Quote 0
                                        • DerelictD
                                          Derelict LAYER 8 Netgate
                                          last edited by

                                          https://forum.pfsense.org/index.php?topic=88466.msg488411#msg488411

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            agreenfield1
                                            last edited by

                                            @Derelict:

                                            Yup.

                                            Unbound starts returning SERVFAIL for random domains after querying at least one or both of those hostnames.  I'm doing it one more time at unbound log level 5.

                                            This sounds like the manifestation of the issue with DNSSEC enabled.  At some point, you may want to try it again with DNSSEC disabled; you should then see all domains being resolved to a hostile IP, bad certs for https, etc. like reported in the beginning of this thread.  Not sure if this would help with the diagnostics.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.