Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Periodic since 2.2 pages load blank, certs invalid

    Scheduled Pinned Locked Moved General pfSense Questions
    126 Posts 14 Posters 46.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      I originally thought this issue was unrelated to pfsense, and posted the issue here:https://forum.pfsense.org/index.php?topic=88238.0.  But after seeing this thread, it seems like pfsense 2.2 / DNS Resolver / Unbound may be a factor?

      This has nothing to do with pfSense.  It has to do with you relying solely on google/level3 for all your DNS and someone is playing with it.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • K
        kejianshi
        last edited by

        Yep…  Now who could do that on a broad basis?

        1 Reply Last reply Reply Quote 0
        • DerelictD
          Derelict LAYER 8 Netgate
          last edited by

          It's intriguing.

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          1 Reply Last reply Reply Quote 0
          • T
            Trel
            last edited by

            Actually once I switched fully to unbound + DNSSEC only, I had a new issue.  At the same times, unbound would stop working.  The service would be running, but it wouldn't resolve anything until I restarted the service.

            I finally found a common thread for that happening.  It almost always directly followed someone doing a lookup of

            api-nyc01.exip.org

            or

            ns3.csof.net

            The IP for those are in the 195.22.x range that was mentioned earlier.

            Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

            If someone is willing to look at that, because of how it lines up, it looks like trying to access/doing a lookup on those domains will either cause the blank pages and lolcat certs, or will cause unbound to stop resolving until the service is restarted.

            It's too coincidental to ignore in this case.

            1 Reply Last reply Reply Quote 0
            • D
              doktornotor Banned
              last edited by

              @Trel:

              It almost always directly followed someone doing a lookup of

              api-nyc01.exip.org

              or

              ns3.csof.net

              The IP for those are in the 195.22.x range that was mentioned earlier.
              Almost without fail, trying to access one of those, causes unbound to stop working until I restart the service.

              Tried both, unbound still working. :) Apparently no NSA love here.  :'( ;D

              1 Reply Last reply Reply Quote 0
              • T
                Trel
                last edited by

                Now, I also had block rules in place for that range of IP.
                I wonder if that could interact in some way.

                Additionally, if you have Snort/Suricata installed, do you now have alerts mentioning the Anubis DNS Sinkhole?

                1 Reply Last reply Reply Quote 0
                • D
                  doktornotor Banned
                  last edited by

                  @Trel:

                  if you have Snort/Suricata installed

                  Noooooooooooooo!!!

                  1 Reply Last reply Reply Quote 0
                  • K
                    kejianshi
                    last edited by

                    !!!

                    1 Reply Last reply Reply Quote 0
                    • H
                      heper
                      last edited by

                      (unbound has been enabled for more then a month without issues … until an hour ago)
                      i've suddenly been experiencing the blank pages + dns redirects to buydomains.com for lots of valid domains.

                      i tried to fix it by enabling dnssec ... didn't help

                      for now i've enabled "forwarding mode" on unbound ... this seems to fix the issue.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        Unbound in resolver mode?  That makes no sense.  The deal is is that makes it pretty much impossible to affect everything.  They have to target specific name servers for specific domains (or .com, or . (root) etc..

                        What DNS servers are you handing out to your clients?  Running unbound means nothing if your clients are going to 8.8.8.8 / 8.8.4.4 for DNS.

                        Want an easy way to find out?  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged….

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • K
                          kejianshi
                          last edited by

                          After you properly set up DNS and DNSSEC, you still have to clear DNS cache on each client and also have to make sure your clients are not infected with something or running some stupid browser add-on that hijacks things.

                          1 Reply Last reply Reply Quote 0
                          • 2
                            2chemlud Banned
                            last edited by

                            @Derelict:

                            ….  Block TCP/UDP 53 on LAN to everything but your unbound and see what breaks.  :)  Or pass with logging and see what's logged....

                            Port 53 is not allowed at my network for more than a year. Doing fine with the DNS servers in the General setup and keep awful devices such as Buffallo Linkstations etc from phoning home…

                            1 Reply Last reply Reply Quote 0
                            • S
                              swix
                              last edited by

                              When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

                              swix@pc:~> host google.ch
google.ch has address 195.22.26.248                                                                                                                                                                   
google.ch mail is handled by 10 mx1.csof.net.                                                                                                                                                         
google.ch mail is handled by 10 mx2.csof.net.                          

swix@pc:~> host aaaaaafadkfjdu93jifa.ch
aaaaaafadkfjdu93jifa.ch has address 195.22.26.248                                                                                                                                                  
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx1.csof.net.                                                                                                                                        
aaaaaafadkfjdu93jifa.ch mail is handled by 10 mx2.csof.net.

                              Unbound server is set as local resolver for a small LAN, with no forwarding to remote resolvers, so everything should be resolved locally.

                              Still investigating about how this could happen, and I will update this thread as soon as I find anything.
                              Kind regards.

                              PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.

                              1 Reply Last reply Reply Quote 0
                              • P
                                Pakken
                                last edited by

                                So far happened only one time for me.
                                After enabling dnssec and disabling all the forwards to public dns servers it seems to be fixed.
                                In addition, I've created a floating rule to block every local subnet to that 195.22.0.0 range.

                                Will keep you updated.
                                To be honest the strange thing is that in a couple of years of pfsense pre-2.2 and dnsmasq this never happened.
                                The problem appeared straight after upgrading to 2.2 and dnsresolver even tho, once again, only happened one time so far to me.

                                Best regards

                                1 Reply Last reply Reply Quote 0
                                • D
                                  doktornotor Banned
                                  last edited by

                                  @swix:

                                  When it happened, all dns requests returns "195.22.26.248" as IP address (also for invalid domains):

                                  I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
                                  https://www.virustotal.com/en/ip-address/195.22.26.248/information/

                                  If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.

                                  @swix:

                                  PS: it apparently happened to last week, and I then disabled DNSSEC Support, but as it happened again it doesn't seem to be related.

                                  Disabling DNSSEC most certainly does NOT help anything. Very broken idea.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    swix
                                    last edited by

                                    @doktornotor:

                                    I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
                                    https://www.virustotal.com/en/ip-address/195.22.26.248/information/
                                    If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.
                                    ng DNSSEC most certainly does NOT help anything. Very broken idea.

                                    Thanks for the suggestion, yes, I will try to have a look on this, but the network device (VDSL Bridge Zyxel P-870M) is in bridge mode, so I have no way to connect directly to it (or only via a serial console, with a cable to be found yet).  Newest Firmware = 2009.

                                    It just happened again a few minutes ago (3rd time today).

                                    I was also trying to see if the root-servers file was tempered anyhow, but /etc/unbound/root.hints does not exist at all on the pfsense router.

                                    Log extract when problem is happening, with many requests to "ns*.csof.net" servers where it shouldn't be the case  :

                                    
Feb  9 12:55:25 pf unbound: [39509:0] info: reply from <4.85.in-addr.arpa.> 195.186.196.180#53
Feb  9 12:55:25 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN       
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.csof.net. AAAA IN  
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns4.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <ubuntu.com.>195.22.26.248#53          ########## wrong ! 
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER</ubuntu.com.></com.></com.></com.>

                                    TBC.

                                    1 Reply Last reply Reply Quote 0
                                    • T
                                      Trel
                                      last edited by

                                      I just want to clear up a few things

                                      When any DNS Server can be used (not just unbound) and DNS Sec is set to off
                                      -A DNS lookup from any computer to one of the domains cause EVERY subsequent lookup to resolve to "195.22.26.248"
                                      (persists until unbound service is restarted)

                                      When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                                      -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                                      (persists until unbound service is restarted)

                                      Thanks to a packet capture I was able to find which domains were being looked up, I then overrode the hosts and set the IP to 0.0.0.0 so they resolve but obviously can't get out

                                      When only unbound can be used and DNS Sec is set to ON, port 53 is blocked except to pfsense, AND the hosts are overrode so Unbound doesn't make any query on those domains outward either
                                      -All problems seem to cease

                                      (Also, due to the packet capture, I can say the original request is coming from an unrooted Android device requesting port 80 on a few of those sites, based on the URL, it's requesting an API for determining the outward IP)

                                      EDIT: here's the overrides I did to flat out prevent those names from being resolved.

                                      overrides.jpg
                                      overrides.jpg_thumb

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        swix
                                        last edited by

                                        Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.    And last time, it stopped by itself after about 5 minutes.

                                        When "broken", all webrequests are redirected to http://xsso.www.example.org (with the original domain name instead of example.org).

                                        GET /domain/www.example.org HTTP/1.1
Host: sso.mlwr.io
User-Agent: Mozilla/5.0 (X11; Linux x86_64) (...)
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de,en-US;q=0.8,en;q=0.6
Cookie: anbsso=a5f4221ae2729d945150c83748e2ea12 (...)

Response: HTTP/1.1 302 Moved Temporarily
Server: nginx-perl/1.2.9.7
Date: Mon, 09 Feb 2015 14:29:31 GMT
Content-Type: text/html
Transfer-Encoding: chunked Connection: keep-alive
Set-Cookie: btst=b1b2035cffe818d92d7f6604a1318beb|myip|...
Location: http://xsso.www.example.org/a5f4221ae2729d945150c83748e2ea12

                                        Just increased the logfiles size to try to see more next time it happens and added monitoring to get an alert directly.

                                        And another view with wget, with and without https, with the same "lolcat" I already saw in another thread:

                                        
om@ompc:~> wget http://www.example.org
--2015-02-09 13:55:37--  http://www.example.org                                                                                                                                                     
Resolving www.example.org (www.example.org)... 195.22.26.248                                                                                                                                           
Connecting to www.example.org (www.example.org)|195.22.26.248|:80... connected.                                                                                                                        
HTTP request sent, awaiting response... 302 Moved Temporarily                                                                                                                                      
Location: http://sso.mlwr.io/domain/www.example.org [following]                                                                                                                                      
--2015-02-09 13:55:39--  http://sso.mlwr.io/domain/www.example.org
Resolving sso.mlwr.io (sso.mlwr.io)... 195.22.26.248
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Cookie coming from sso.mlwr.io attempted to set domain to example.org
Length: unspecified [text/html]
Saving to: ‘index.html.3’

om@ompc:~> wget https://www.example.org
--2015-02-09 13:55:43--  https://www.example.org/
Resolving www.example.org (www.example.org)... 195.22.26.248
Connecting to www.example.org (www.example.org)|195.22.26.248|:443... connected.
ERROR: cannot verify www.example.org's certificate, issued by ‘/CN=lolcat’:
  Self-signed certificate encountered.
    ERROR: certificate common name ‘lolcat’ doesn't match requested host name ‘www.example.org’.
To connect to www.example.org insecurely, use `--no-check-certificate'.
                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          Trel
                                          last edited by

                                          You said you enabled DNSSEC, but question, what do you have in

                                          System -> General -> DNS servers?

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            swix
                                            last edited by

                                            PS:  installed packages on this router: arpwatch, bandwithd, cron, darkstat, mailreport, nrpe, rrd summary, openvpn client export.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.