Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Periodic since 2.2 pages load blank, certs invalid

    Scheduled Pinned Locked Moved General pfSense Questions
    126 Posts 14 Posters 46.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      swix
      last edited by

      @doktornotor:

      I'd certainly investigate the LAN for possible infection. Just look at the amount of malicious crap associated with that IP:
      https://www.virustotal.com/en/ip-address/195.22.26.248/information/
      If you have some ISP-supplied router/modem in front of the pfSense box, Google for possible well-known firmware exploits as well.
      ng DNSSEC most certainly does NOT help anything. Very broken idea.

      Thanks for the suggestion, yes, I will try to have a look on this, but the network device (VDSL Bridge Zyxel P-870M) is in bridge mode, so I have no way to connect directly to it (or only via a serial console, with a cable to be found yet).  Newest Firmware = 2009.

      It just happened again a few minutes ago (3rd time today).

      I was also trying to see if the root-servers file was tempered anyhow, but /etc/unbound/root.hints does not exist at all on the pfsense router.

      Log extract when problem is happening, with many requests to "ns*.csof.net" servers where it shouldn't be the case  :

      
Feb  9 12:55:25 pf unbound: [39509:0] info: reply from <4.85.in-addr.arpa.> 195.186.196.180#53
Feb  9 12:55:25 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN       
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns2.csof.net. AAAA IN  
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns4.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns3.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: resolving ns1.csof.net. AAAA IN
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns3.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns2.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for ns1.canonical.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <com.>54.77.72.254#53
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER
Feb  9 12:55:28 pf unbound: [39509:0] info: response for daisy.ubuntu.com. A IN
Feb  9 12:55:28 pf unbound: [39509:0] info: reply from <ubuntu.com.>195.22.26.248#53          ########## wrong ! 
Feb  9 12:55:28 pf unbound: [39509:0] info: query response was ANSWER</ubuntu.com.></com.></com.></com.>

      TBC.

      1 Reply Last reply Reply Quote 0
      • T
        Trel
        last edited by

        I just want to clear up a few things

        When any DNS Server can be used (not just unbound) and DNS Sec is set to off
        -A DNS lookup from any computer to one of the domains cause EVERY subsequent lookup to resolve to "195.22.26.248"
        (persists until unbound service is restarted)

        When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
        -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
        (persists until unbound service is restarted)

        Thanks to a packet capture I was able to find which domains were being looked up, I then overrode the hosts and set the IP to 0.0.0.0 so they resolve but obviously can't get out

        When only unbound can be used and DNS Sec is set to ON, port 53 is blocked except to pfsense, AND the hosts are overrode so Unbound doesn't make any query on those domains outward either
        -All problems seem to cease

        (Also, due to the packet capture, I can say the original request is coming from an unrooted Android device requesting port 80 on a few of those sites, based on the URL, it's requesting an API for determining the outward IP)

        EDIT: here's the overrides I did to flat out prevent those names from being resolved.

        overrides.jpg
        overrides.jpg_thumb

        1 Reply Last reply Reply Quote 0
        • S
          swix
          last edited by

          Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.    And last time, it stopped by itself after about 5 minutes.

          When "broken", all webrequests are redirected to http://xsso.www.example.org (with the original domain name instead of example.org).

          GET /domain/www.example.org HTTP/1.1
Host: sso.mlwr.io
User-Agent: Mozilla/5.0 (X11; Linux x86_64) (...)
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de,en-US;q=0.8,en;q=0.6
Cookie: anbsso=a5f4221ae2729d945150c83748e2ea12 (...)

Response: HTTP/1.1 302 Moved Temporarily
Server: nginx-perl/1.2.9.7
Date: Mon, 09 Feb 2015 14:29:31 GMT
Content-Type: text/html
Transfer-Encoding: chunked Connection: keep-alive
Set-Cookie: btst=b1b2035cffe818d92d7f6604a1318beb|myip|...
Location: http://xsso.www.example.org/a5f4221ae2729d945150c83748e2ea12

          Just increased the logfiles size to try to see more next time it happens and added monitoring to get an alert directly.

          And another view with wget, with and without https, with the same "lolcat" I already saw in another thread:

          
om@ompc:~> wget http://www.example.org
--2015-02-09 13:55:37--  http://www.example.org                                                                                                                                                     
Resolving www.example.org (www.example.org)... 195.22.26.248                                                                                                                                           
Connecting to www.example.org (www.example.org)|195.22.26.248|:80... connected.                                                                                                                        
HTTP request sent, awaiting response... 302 Moved Temporarily                                                                                                                                      
Location: http://sso.mlwr.io/domain/www.example.org [following]                                                                                                                                      
--2015-02-09 13:55:39--  http://sso.mlwr.io/domain/www.example.org
Resolving sso.mlwr.io (sso.mlwr.io)... 195.22.26.248
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Cookie coming from sso.mlwr.io attempted to set domain to example.org
Length: unspecified [text/html]
Saving to: ‘index.html.3’

om@ompc:~> wget https://www.example.org
--2015-02-09 13:55:43--  https://www.example.org/
Resolving www.example.org (www.example.org)... 195.22.26.248
Connecting to www.example.org (www.example.org)|195.22.26.248|:443... connected.
ERROR: cannot verify www.example.org's certificate, issued by ‘/CN=lolcat’:
  Self-signed certificate encountered.
    ERROR: certificate common name ‘lolcat’ doesn't match requested host name ‘www.example.org’.
To connect to www.example.org insecurely, use `--no-check-certificate'.
          1 Reply Last reply Reply Quote 0
          • T
            Trel
            last edited by

            You said you enabled DNSSEC, but question, what do you have in

            System -> General -> DNS servers?

            1 Reply Last reply Reply Quote 0
            • S
              swix
              last edited by

              PS:  installed packages on this router: arpwatch, bandwithd, cron, darkstat, mailreport, nrpe, rrd summary, openvpn client export.

              1 Reply Last reply Reply Quote 0
              • S
                swix
                last edited by

                @Trel:

                You said you enabled DNSSEC, but question, what do you have in
                System -> General -> DNS servers?

                Completely empty, so shoud unbound start with root servers directly.  I was wondering where was the hints file for unbound, but it seems to be directly in the binary file (strings unbound) :

                A.ROOT-SERVERS.NET.
198.41.0.4
B.ROOT-SERVERS.NET.
192.228.79.201
C.ROOT-SERVERS.NET.
192.33.4.12
D.ROOT-SERVERS.NET.
199.7.91.13
E.ROOT-SERVERS.NET.
192.203.230.10
F.ROOT-SERVERS.NET.
192.5.5.241
G.ROOT-SERVERS.NET.
192.112.36.4
H.ROOT-SERVERS.NET.
128.63.2.53
I.ROOT-SERVERS.NET.
192.36.148.17
J.ROOT-SERVERS.NET.
192.58.128.30
K.ROOT-SERVERS.NET.
193.0.14.129
L.ROOT-SERVERS.NET.
199.7.83.42
                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Ok.  So what are the DNS servers configured on the client?

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • S
                    swix
                    last edited by

                    @Derelict:

                    Ok.  So what are the DNS servers configured on the client?

                    Set via DHCP, simply the router's ip address:

                    
   option domain-name-servers 192.168.1.100;
                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      And you've actually verified that's the case on the client?

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • D
                        doktornotor Banned
                        last edited by

                        @swix:

                        Thanks for your update Trel, we're still searching here, but enabling DNSSEC does not stop the issue.

                        It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

                        • Block/redirect all DNS queries on LAN to pfSense
                        • Find and reimage infected crap.
                        1 Reply Last reply Reply Quote 0
                        • T
                          Trel
                          last edited by

                          @doktornotor:

                          • Find and reimage infected crap.

                          I agree, but I would like to point out the way it affects pfsense/unbound is not a good thing at all.

                          A lookup on a completely isolated network segment made unbound start giving bad resolutions to ALL network segments when other DNS servers were permitted, and when they were blocked, unbound simply stopped replying.

                          That's not really the best outcome for a single computer looking up a bad domain.

                          1 Reply Last reply Reply Quote 0
                          • S
                            swix
                            last edited by

                            @Derelict:

                            And you've actually verified that's the case on the client?

                            Yes, + also did tests with dig @192.168.1.100 and directly via the pfsense shell. "poisoned" ip in every case (after a few seconds after the beginning of a new occurence of the issue).

                            @doktornotor:

                            It does NOT stop the issue on domains that are not signed, no. Also, it will NOT prevent the DNS hijack if your clients are NOT using pfSense or another DNSSEC-enabled resolver, even if the zones are signed. It will prevent resolving domains to malicious crap for the rest.

                            Yep, I supposed that too, but sometimes there are collateral effets to such settings.

                            @doktornotor:

                            • Block/redirect all DNS queries on LAN to pfSense
                            • Find and reimage infected crap.

                            It will continue tomorrow, now it is calm again, as everybody left the office :)    But even if it is related to one malicious host on the LAN, it shouldn't be able to break the unbound resolver so easily…

                            Thanks again for all your feedbacks and until tomorrow!

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

                              You need to be looking at these queries from the root back and see where things go wrong.

                              Very intriguing.

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • D
                                doktornotor Banned
                                last edited by

                                @Derelict:

                                If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

                                Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

                                  Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

                                  I would put this on LAN:

                                  pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log

                                  Put that above your normal pass rule.  If everything is as you say, it should log nothing.

                                  On pfSense 2.2 you should be able to set the dest to ! This Firewall (self).

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    doktornotor Banned
                                    last edited by

                                    @Derelict:

                                    Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.

                                    Pretty sure I could get these guys involved in investigating the issue here (they've also written the Knot DNS server so I'm rather convinced they are familiar with DNS  :P) – however that'd require either remote access or at least uncensored traffic captures. Not example.com -- totally useless.

                                    1 Reply Last reply Reply Quote 0
                                    • T
                                      Trel
                                      last edited by

                                      @doktornotor:

                                      @Derelict:

                                      If pfSense/unbound asks the configured upstream DNS servers to resolve a query and gets something unexpected back it's not the fault of pfSense/unbound.

                                      Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

                                      Using Comcast with a modem only (not a gateway in bridged mode).  Here's the block rule.
                                      With these settings, if I try to look up the domain I get this scenario

                                      When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                                      -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                                      (persists until unbound service is restarted)

                                      I understand that an infected machine should not be on the network, but if a mere typical DNS lookup can cause this much havoc, then something is really wrong.

                                      restricted_dns.gif
                                      restricted_dns.gif_thumb

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        swix
                                        last edited by

                                        @Derelict:

                                        Also, by obfuscating everything to example.com, you are eliminating the ability of everyone reading this thread from seeing what responses they get to the same queries.
                                        Maybe someone else would get the BS responses and be in a better position to troubleshoot it than you are.

                                        It wasn't obfuscated, it really looked like that… (also with other domains, juste replace example.com by anything)

                                        @Derelict:

                                        I would put this on LAN:
                                        pass IPv4 TCP/UDP source LAN net dest ! 192.168.1.100 port 53 log
                                        Put that above your normal pass rule.  If everything is as you say, it should log nothing.

                                        Ok, thanks, will setup this.

                                        @doktornotor:

                                        Yes, exactly. Strongly suspect most of the people here are either using some hacked ISP device that hijacks the DNS traffic or the clients do not query the pfSense DNS resolver at all.

                                        I would be really happy to know the cause, it is really strange that Trel is having a similar problem with the very same target IP "195.22.26.248", especially from different countries/ISP's.  The only recent change to our infrastructure was upgrading to pfSense 2.2 at the beginning of January, otherwise nothing special.  But I'll setup some network monitoring tools later this week.

                                        Best regards

                                        1 Reply Last reply Reply Quote 0
                                        • DerelictD
                                          Derelict LAYER 8 Netgate
                                          last edited by

                                          @Trel:

                                          When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                                          -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                                          (persists until unbound service is restarted)

                                          A search of redmine does not show that as an open issue.  Have you reported it?

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          • 2
                                            2chemlud Banned
                                            last edited by

                                            @Trel:

                                            When only unbound can be used and DNS Sec is set to ON, and port 53 is blocked except to pfsense
                                            -A DNS lookup from any computer to one of the domains causes unbound to stop resolving anything, all lookups fail
                                            (persists until unbound service is restarted)[

                                            [/quote]

                                            I can not confirm this, worked fine for me in this setup (with some service interruptions, 5-7times a day)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.