Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Not sure what happened last night, need help. NTP???

    Scheduled Pinned Locked Moved General pfSense Questions
    12 Posts 4 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      callen
      last edited by

      At 12:42am last night I got a notification that my servers were offline from a service I use to monitor my network. Then at 5:25am I got another notification saying the servers had come back online. I was asleep this whole time.

      Also at 5:25am (same time as when the servers back online email) I got an arpwatch email notification titled: Cron root@pfSense /usr/bin/nice -n20 /etc/rc.update_bogons.sh
      The body of this email is the following:

      X-Cron-Env: <SHELL=/bin/sh>
      X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin>
      X-Cron-Env: <HOME=/root>
      X-Cron-Env: <LOGNAME=root>
      X-Cron-Env: <USER=root>

      Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid
      34374270280:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/factory-crossbuild-245-amd64/sources/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
      fetch: https://files.pfsense.org/lists/fullbogons-ipv4.txt: Authentication error
      Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid
      34374270280:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/factory-crossbuild-245-amd64/sources/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
      fetch: https://files.pfsense.org/lists/fullbogons-ipv4.txt: Authentication error
      Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid
      34374270280:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/factory-crossbuild-245-amd64/sources/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
      fetch: https://files.pfsense.org/lists/fullbogons-ipv4.txt: Authentication error
      Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid
      34374270280:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/factory-crossbuild-245-amd64/sources/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
      fetch: https://files.pfsense.org/lists/fullbogons-ipv4.txt: Authentication error
      Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid
      34374270280:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:/build/factory-crossbuild-245-amd64/sources/FreeBSD-src/crypto/openssl/ssl/s3_clnt.c:1269:
      fetch: https://files.pfsense.org/lists/fullbogons-ipv4.txt: Authentication error

      Thinking this was odd I checked the logs of my servers and there is nothing to explain being shown offline. I even had a backup running during the time and it completed with no issues.

      So I went and started looking at the PFSense logs. The only weird log entries I found were in the NTP log. Before what you see below the last log entries for the NTP log were June 28, which is when I updated PFSense to 2.4.5-RELEASE-p1

      The first entry for this morning is:

      Oct 1 01:36:48	ntpd	6346	Soliciting pool server 192.168.7.254

      ...hundreds of this log entry...about 4 each minute...and the last is:

      Oct 1 05:23:59	ntpd	6346	Soliciting pool server 192.168.7.254

      The next entry after is:

      Oct 1 05:24:33	ntpd	6346	Soliciting pool server 64.142.54.12

      Over the next 6 seconds there are 12 entries like this one soliciting a pool server with different external IP addresses.

      Then I get this entry:

      Oct 1 05:25:04	ntpd	6346	216.218.254.202 local addr <my external network IP address> -> <null>

      And another 10 of these entries between 5:25 and 7:46 with different IP addresses than 216.218.254.202.

      And the entries stop.

      BTW, I don't use 192.168.7.0/24 at all in my network.

      Everything seems to be working ok right now. I can't find any issues on my network. I am assuming the arpwatch email with the authentication errors above had to do with the NTP issues.

      Can anyone help explain to me what happened? And if I need to make any changes to stop it from happening again.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        @callen said in Not sure what happened last night, need help. NTP???:

        Certificate verification failed for /C=US/ST=MA/L=Lowell/O=Arris Group, Inc./OU=Telco CPE/CN=dsldevice.domain_not_set.invalid

        It looks like your cable connection went down and your modem started redirecting https requests to it's internal page. Which obviously threw a cert error.
        pfSense tried to update the bogons list and hit it.

        Did you see the WAN IP change to something the modem handed out? In the 192.168.7.0/24 subnet perhaps?
        You can set an IP to reject DHCP leases from the in the WAN dhcp setup. Cable modems doing that is quite common. You need to know the DHCP server address it's using though.
        https://docs.netgate.com/pfsense/en/latest/interfaces/configure-ipv4.html#dhcp

        Steve

        C 1 Reply Last reply Reply Quote 0
        • C
          callen @stephenw10
          last edited by

          @stephenw10 Thanks for replying.

          I just checked with our ISP and they are adamant we never lost service and that our Modem had service throughout this time.

          I get arp notifications on IP changes and never got one during this time. I checked the System General logs for anything with the WAN MAC address and there are no entries. Is there somewhere else I can check to be sure?

          Also, the WAN IP is set to Static IPv4. Wouldn't that mean it wouldn't accept a change?

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Yeah if it's static then you would not see an IP change. You might see an ARP warning for the gateway.

            That certificate is clearly invalid though and sure looks like something that would be on the modem. You could probably check the modem gui cert to be sure.

            It could be something further upstream.

            Either way pfSense was resolving files.pfsense.org to that. So either the https was redirected or the DNS was hijacked. If pfSense is using Unbound with DNSSec only for it's own DNS that could not happen.

            Steve

            C 1 Reply Last reply Reply Quote 0
            • C
              callen @stephenw10
              last edited by

              @stephenw10 Update with new info. After seeing your reply I checked the modem's web interface. 192.168.7.254 is the modem's IP address. Does that change your analysis of what happened?

              BTW, pfsense is using the DNS Resolver with DNSSec enabled. Only override is a domain override for our Windows devices.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                @callen said in Not sure what happened last night, need help. NTP???:

                192.168.7.254

                No that only confirms it. I would say the modem started redirecting everything to itself, including the ntp requests.
                If the cert on the modem gui looks like that string that will also confirm it.

                Modems usually only do that if they lose upstream sync. Maybe it rebooted or crashed but that wouldn't have taken hours to come back.

                Steve

                C 1 Reply Last reply Reply Quote 0
                • C
                  callen @stephenw10
                  last edited by

                  @stephenw10 Everything on that first line of the certificate verification failure lines up with the certificate of the modem. However I cannot verify lines 2 and 3. I am 99% sure it is the modem cert though.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Yeah the 2 other lines are the error caused by it.

                    Pretty conclusive your modem started redirecting all traffic to itself and that really only happens when it loses connection.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • GertjanG
                      Gertjan
                      last edited by

                      @stephenw10 said in Not sure what happened last night, need help. NTP???:

                      only happens when it loses connection.

                      Bring some people with you as a witness.
                      And rip out the 'WAN' plug of your modem for an hour or so.
                      Call your ISP again ....

                      @callen said in Not sure what happened last night, need help. NTP???:

                      I just checked with our ISP and they are adamant we never lost service

                      Now you're aware of the "quality" of that answer ;)

                      No "help me" PM's please. Use the forum, the community will thank you.
                      Edit : and where are the logs ??

                      1 Reply Last reply Reply Quote 0
                      • C
                        callen
                        last edited by

                        @stephenw10 Ok thanks. We ended up having the modem replaced yesterday afternoon just as a precaution.

                        @Gertjan yeah I am more confident now that we lost service somehow, even if it was due to an issue with the modem's DNS provider, which is not the same as I have in pfSense.

                        1 Reply Last reply Reply Quote 0
                        • AKEGECA
                          AKEGEC
                          last edited by

                          @callen , It could be that ISP reseted your modem. But modem failure happens when you have bad weather like lightning storm.

                          C 1 Reply Last reply Reply Quote 0
                          • C
                            callen @AKEGEC
                            last edited by

                            @AKEGEC I asked them that and they said the modem had been up for 20+ days. That matched up with the uptime in the GUI.

                            As for weather, at that time it was really calm and moderate. No storms in the area. (That said I will never count out squirrels as a culprit. :)

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.