Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Periodic since 2.2 pages load blank, certs invalid

    General pfSense Questions
    14
    126
    22546
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      Trel last edited by

      This has now happened three times

      The symptoms I can see are

      1. HTTP Webpages load blank
      2. HTTPs webpages give a security error
      3. Accessing pages by IP works
      4. Any IP based connection works
      5. Tracert appears valid

      When this happens, if
      1. I release and renew the IP for the WAN it works again
      2. If I reboot PFSense, it works again

      Additionally, while this is occuring
      1. I CAN access the firewall's GUI internally (correct behavior)
      1. I CAN access the firewall's GUI externally (correct behavior)

      This started happening since I upgraded to 2.2 on Saturday morning.
      No new rules have been created, no new firewall logs are showing up when this happens, nothing not usual in any of the log tabs.

      This is a physical box with PFSense installed directly.
      I just rebooted it remotely, to get everything back up and working.

      Does anyone know what's going on here, or where I can look for more info?

      EDIT: I should add, that the packages I have installed are
      1. arping
      2. Cron
      3. File Manager
      4. Notes
      5. OpenVPN Client Export Utility

      1 Reply Last reply Reply Quote 0
      • C
        cmb last edited by

        What specifically is the cert error you get on HTTPS sites in that circumstance? Short of some hacking to try to transparently proxy HTTPS, which wouldn't happen with that list of packages you have there, there wouldn't be anything within the firewall itself that'd cause a cert error.

        1 Reply Last reply Reply Quote 0
        • T
          Trel last edited by

          @cmb:

          What specifically is the cert error you get on HTTPS sites in that circumstance? Short of some hacking to try to transparently proxy HTTPS, which wouldn't happen with that list of packages you have there, there wouldn't be anything within the firewall itself that'd cause a cert error.

          I can get that information if this happens again.
          I had only had the blank page one when I was present, but I was told about a site giving the invalid cert, so I told them to try another site to see if it was their machine's clock being wrong or similar, and that's how I found out the blank page thing was happening again.

          At that point I remotely rebooted and it hasn't happened again yet.

          If I had to guess, considering the webpages didn't report an error, just loaded blank, I'd guess that it was the same for the cert and as such, they were invalid.

          1 Reply Last reply Reply Quote 0
          • T
            Trel last edited by

            It's happening again.  I can't get the cert error to occur on my machine, but I am getting the blank pages.

            Is there any debug steps I can take before I reboot?

            EDIT: I can reproduce the cert issue, any site affected shows up as a self signed cert from and to "lolcat"
            EDIT2: It suddenly resolved itself and the pages began loading again and certs were valid.

            I think I'm likely going to reinstall 2.1.5 if I can't find any reason for this.  It coincides with the upgrade and the cert part of it, especially with an actual name showing up there unless that's some default name in Firefox that I don't know about when it can't load a cert, has me seriously worried.

            1 Reply Last reply Reply Quote 0
            • C
              cmb last edited by

              Packet capture of the problem traffic would be telling.

              The whole getting a self-signed certificate with "lolcat" is a serious cause for concern, nothing on your firewall would be doing that, that suggests some kind of malware somewhere. Potentially on a client machine that's running an ARP poisoning tool and hijacking your connections on occasion.

              Check your system log for any indications of "xx:xx:xx:xx:xx:xx is using my IP 192.168.1.1" (replace 1.1 with your LAN IP), that's one place to see if you have something trying to ARP poison.

              1 Reply Last reply Reply Quote 0
              • T
                Trel last edited by

                @cmb:

                Packet capture of the problem traffic would be telling.

                The whole getting a self-signed certificate with "lolcat" is a serious cause for concern, nothing on your firewall would be doing that, that suggests some kind of malware somewhere. Potentially on a client machine that's running an ARP poisoning tool and hijacking your connections on occasion.

                Check your system log for any indications of "xx:xx:xx:xx:xx:xx is using my IP 192.168.1.1" (replace 1.1 with your LAN IP), that's one place to see if you have something trying to ARP poison.

                I'll try that when next it happens.
                I don't see anything in the system log that has any mention of the IP.

                Someone in IRC suggested that when it happens next, I run 'openssl s_client -showcerts -connect site:443' from an SSH to the firewall to verify it's not something upstream.
                Considering that I don't necessarily have to reboot PFSense to resolve it, and releasing/renewing the WAN IP fixes it as well has me worried on that.

                If it is something upstream, I'm extra worried because my connection is ISP–Modem--PFSense--Switch(s)--Computers

                1 Reply Last reply Reply Quote 0
                • C
                  cmb last edited by

                  Running openssl from the firewall itself is a good idea, that'll at least bisect the issue. It seems unlikely it'd be on your ISP's side at least if it's the same ISP you're using to hit the forum, Comcast. It's possible, and given a change of WAN IP fixes it, that makes it seem more likely it's WAN-side, as nothing LAN-side would be impacted by that (unless it's just a coincidence). Some inept small ISP with a bunch of customers on the same broadcast domain and inadequate protection against things like ARP poisoning from customer to customer, I could see as being potentially more likely.

                  1 Reply Last reply Reply Quote 0
                  • T
                    Trel last edited by

                    @cmb:

                    Running openssl from the firewall itself is a good idea, that'll at least bisect the issue. It seems unlikely it'd be on your ISP's side at least if it's the same ISP you're using to hit the forum, Comcast. It's possible, and given a change of WAN IP fixes it, that makes it seem more likely it's WAN-side, as nothing LAN-side would be impacted by that (unless it's just a coincidence). Some inept small ISP with a bunch of customers on the same broadcast domain and inadequate protection against things like ARP poisoning from customer to customer, I could see as being potentially more likely.

                    I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.
                    I can't speculate further until I can run the openssl command from pfsense when it happens again.

                    And yes, currently Comcast, is the ISP in question.

                    I have one other theory as to the lolcat cert in that it's placeholder text in Firefox in the event that a cert loads completely blank, which would make sense as non SSL pages load blank while this is happening.  I would need to look at (or ask someone familiar with) Firefox's sourcecode to know for sure.

                    But that will be answered when I run openssl from pfsense at least if I get something other than the cert I got when I ran it baseline for comparison.

                    1 Reply Last reply Reply Quote 0
                    • MikeV7896
                      MikeV7896 last edited by

                      @Trel:

                      I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.
                      …
                      And yes, currently Comcast, is the ISP in question.

                      Comcast's DHCP leases are for a few days, which is why you don't get a new address with a release/renew. From looking at the DHCP client leases file on my box, it looks like they're about 4 days (renew time is half of the lease, and there's 2 days from renew to expire). IPv6 prefix leases are 7 days, from what I was told by a Comcast network engineer in another forum.

                      The S in IOT stands for Security

                      1 Reply Last reply Reply Quote 0
                      • T
                        Trel last edited by

                        I know. I meant the issue wasn't stopping from my WAN IP changing when I release/renew.

                        1 Reply Last reply Reply Quote 0
                        • S
                          saywhat last edited by

                          We have had the exact same issue here in UK. We use BT as the ISP/

                          Same lolcat 3rd party self signed cert appearing for many sites, all DNS being redirected to 195.22.26.248 which shows as being a malicious IP in Portugal, used for lots of spammy domains.

                          Interestingly, we had Google DNS set on pfsense. When I changed this to OpenDNS the problem immediately went away, pings began to return correct IPs again etc.

                          I know Google DNS was hijacked before, so that is a possibility, but I would have thought an attack such as that would have hit the news on twitter by now.

                          1 Reply Last reply Reply Quote 0
                          • T
                            Trel last edited by

                            @saywhat:

                            We have had the exact same issue here in UK. We use BT as the ISP/

                            Same lolcat 3rd party self signed cert appearing for many sites, all DNS being redirected to 195.22.26.248 which shows as being a malicious IP in Portugal, used for lots of spammy domains.

                            Interestingly, we had Google DNS set on pfsense. When I changed this to OpenDNS the problem immediately went away, pings began to return correct IPs again etc.

                            I know Google DNS was hijacked before, so that is a possibility, but I would have thought an attack such as that would have hit the news on twitter by now.

                            Now THAT actually gives me something to go on.

                            My DNS servers are
                            4.2.2.2
                            8.8.8.8
                            4.2.2.1
                            8.8.4.4

                            The 8.8.8.8 and 8.8.4.4 being Google DNS.
                            Are those the ones you have configured?

                            EDIT: and that could also explain how releasing and renewing the WAN connection fixes it.  If it loses connection with 4.2.2.2 and goes to Google's 8.8.8.8 and that's the problem, releasing/renewing might re-establish contact with 4.2.2.2.

                            1 Reply Last reply Reply Quote 0
                            • S
                              saywhat last edited by

                              We were using 8.8.8.8 and 8.8.4.4

                              Changed them over to opendns and machines responded almost immediately

                              1 Reply Last reply Reply Quote 0
                              • T
                                Trel last edited by

                                @saywhat:

                                We were using 8.8.8.8 and 8.8.4.4

                                Changed them over to opendns and machines responded almost immediately

                                That looks like it has a good chance at being the cause then.
                                I'm going to remove those from my list and just keep the Level3 ones (4.2.2.1 and 4.2.2.2) and see if it ever happens again.

                                That might also explain why it didn't happen until right after the 2.2 upgrade if dnsmasq had a higher tolerance before falling over to the secondary DNS server than unbound does.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  Pakken last edited by

                                  God, and I thought I was the only one having this problem since I came up reading this thread.

                                  Any news about that? Same invalid cert, same google dns.
                                  Spent the last night trying to figure out what the he** could have happened.

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    Trel last edited by

                                    @Pakken:

                                    God, and I thought I was the only one having this problem since I came up reading this thread.

                                    Any news about that? Same invalid cert, same google dns.
                                    Spent the last night trying to figure out what the he** could have happened.

                                    Other than us three, I haven't found anyone who reported it anywhere but here.

                                    But it's way too coincidental that three people got the same symptoms and had the same dns.

                                    1 Reply Last reply Reply Quote 0
                                    • K
                                      kejianshi last edited by

                                      Me also - Thats main reason I turned off forwarder and turned on unbound on one of my systems.
                                      The kids were reporting same exact issues as you…

                                      Unbound with DNSSEC is technically slower than a forwarder but it seems faster in actual use and the kids report its solid.
                                      I'm also using it over the VPN for my private use.

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        doktornotor Banned last edited by

                                        NSA testing some new (broken) toys? :D

                                        1 Reply Last reply Reply Quote 0
                                        • K
                                          kejianshi last edited by

                                          I will just say I like unbound and leave it at that…    (-;

                                          Unbound + VPN = my tinfoil hat

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            Trel last edited by

                                            I just had this happen with level3 DNS (4.2.2.1 and 4.2.2.2) as the DNS servers.  I removed them leaving ONLY OpenDNS and it immediately started resolving correctly again.

                                            1 Reply Last reply Reply Quote 0
                                            • K
                                              kejianshi last edited by

                                              A lack of resolution could simply be a network error.  I was really only seeing issue with HTTPS sites.
                                              Cert errors just smell like MITM to me.

                                              1 Reply Last reply Reply Quote 0
                                              • T
                                                Trel last edited by

                                                @kejianshi:

                                                A lack of resolution could simply be a network error.  I was really only seeing issue with HTTPS sites.
                                                Cert errors just smell like MITM to me.

                                                It's not a lack of resolution.  It IS resolving to a different IP.

                                                1 Reply Last reply Reply Quote 0
                                                • K
                                                  kejianshi last edited by

                                                  I'm certain no one would use DNS resolution to effect a MITM attack.  (You are just paranoid)™

                                                  1 Reply Last reply Reply Quote 0
                                                  • C
                                                    cmb last edited by

                                                    @kejianshi:

                                                    I'm certain no one would use DNS resolution to effect a MITM attack.

                                                    That's actually pretty common, there's a variety of malware that will do just that to individual PCs, and sometimes to exploit routers and change their DNS servers so it impacts all LAN hosts. A variety of consumer-grade routers have been susceptible to such attacks.

                                                    1 Reply Last reply Reply Quote 0
                                                    • C
                                                      cmb last edited by

                                                      @Trel:

                                                      I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.

                                                      After the further details later in the thread, I think why that has an impact is because it's triggering a DNS cache flush in the DNS forwarder, so the poisoned replies are no longer there.

                                                      1 Reply Last reply Reply Quote 0
                                                      • K
                                                        kejianshi last edited by

                                                        haha - Yeah.  I know.  My sarcasm wasn't obvious enough?  I'll try harder.

                                                        1 Reply Last reply Reply Quote 0
                                                        • C
                                                          cmb last edited by

                                                          @kejianshi:

                                                          haha - Yeah.  I know.  My sarcasm wasn't obvious enough?  I'll try harder.

                                                          Oh, the sarcasm font on here must be broken, sorry. :)

                                                          1 Reply Last reply Reply Quote 0
                                                          • T
                                                            Trel last edited by

                                                            @cmb:

                                                            @Trel:

                                                            I should mention though, that when I release/renew the WAN interface, I'm not getting a new IP.  I'm getting the same one.  Breaking the connection seems to be what fixes it.

                                                            After the further details later in the thread, I think why that has an impact is because it's triggering a DNS cache flush in the DNS forwarder, so the poisoned replies are no longer there.

                                                            I actually just asked about this here: https://forum.pfsense.org/index.php?topic=87743.0

                                                            Is that a possible scenario, because if so I have a good idea of what might be doing it then.

                                                            1 Reply Last reply Reply Quote 0
                                                            • K
                                                              kejianshi last edited by

                                                              My problem was originating outside the house between the ONT and the FIOS and or google DNS servers…
                                                              Its nothing inside the network that was causing it, but hopefully its mitigated now.

                                                              1 Reply Last reply Reply Quote 0
                                                              • T
                                                                Trel last edited by

                                                                @kejianshi:

                                                                This problem is originating outside the house between the ONT and the FIOS and or google DNS servers…
                                                                Its nothing inside the network that was causing it, but hopefully its mitigated now.

                                                                If I can verify that pfsense itself is seeing the incorrect IPs for DNS lookups, there's definitely nothing internal that could be causing that at all?

                                                                1 Reply Last reply Reply Quote 0
                                                                • K
                                                                  kejianshi last edited by

                                                                  There are just too many ways to mess with DNS especially if you can't trust the network between your machine and the servers. In the end, at best you can really only make sure that the guys playing games with your network aren't common criminals because you don't own the root servers.

                                                                  1 Reply Last reply Reply Quote 0
                                                                  • T
                                                                    Trel last edited by

                                                                    @kejianshi:

                                                                    There are just too many ways to mess with DNS especially if you can't trust the network between your machine and the servers. In the end, at best you can really only make sure that the guys playing games with your network aren't common criminals because you don't own the root servers.

                                                                    But I'm asking about this specifically, I need to know if what's been happening could be due to an infected machine elsewhere on my network, or if it's definitely happening due to something from WAN and beyond.

                                                                    1 Reply Last reply Reply Quote 0
                                                                    • K
                                                                      kejianshi last edited by

                                                                      I don't think one machine on the network could be the problem (unless that machine is pfsense its self), at least in my case, or going to unbound+DNSSEC would have made no difference.

                                                                      1 Reply Last reply Reply Quote 0
                                                                      • Derelict
                                                                        Derelict LAYER 8 Netgate last edited by

                                                                        If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.

                                                                        Chattanooga, Tennessee, USA
                                                                        The pfSense Book is free of charge!
                                                                        DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                                                        1 Reply Last reply Reply Quote 0
                                                                        • T
                                                                          Trel last edited by

                                                                          @kejianshi:

                                                                          I don't think one machine on the network could be the problem (unless that machine is pfsense its self), at least in my case, or going to unbound+DNSSEC would have made no difference.

                                                                          I just double checked and it looks like I never enabled DNSSEC when I changed to unbound.  Does that change your answer at all?
                                                                          (needless to say, it's getting turned on now)

                                                                          @Derelict:

                                                                          If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.

                                                                          North East USA.

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • K
                                                                            kejianshi last edited by

                                                                            How about this - I will show you where the pfsense is sitting.  Hows that?  Check PM

                                                                            1 Reply Last reply Reply Quote 0
                                                                            • C
                                                                              cmb last edited by

                                                                              @Trel:

                                                                              @Derelict:

                                                                              If you were affected by this, what part of the world are you in?  Just curious.  Interesting incident.

                                                                              North East USA.

                                                                              Seems to be geographically diverse. At least two in the US in this thread, one in the UK.

                                                                              1 Reply Last reply Reply Quote 0
                                                                              • K
                                                                                kejianshi last edited by

                                                                                The pfsense in question is in Maryland, for me.

                                                                                1 Reply Last reply Reply Quote 0
                                                                                • T
                                                                                  Trel last edited by

                                                                                  By the way, I can confirm for sure that pfsense was seeing the bad DNS.

                                                                                  I have an alias for facebook.com
                                                                                  I saw this in the resolver log

                                                                                  filterdns: adding entry 195.22.26.248 to table Social_Test on host facebook.com
                                                                                  

                                                                                  195.22.26.248 is the bad IP.  PFSense itself saw that when it did its update on the alias resolution.

                                                                                  1 Reply Last reply Reply Quote 0
                                                                                  • K
                                                                                    kejianshi last edited by

                                                                                    https://www.virustotal.com/en/ip-address/195.22.26.248/information/

                                                                                    https://www.robtex.net/en/advisory/ip/195/22/26/248/

                                                                                    Seems like there is an associated IP block thats pretty much into everything bad.

                                                                                    1 Reply Last reply Reply Quote 0
                                                                                    • First post
                                                                                      Last post