Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Packages Restarting

    Scheduled Pinned Locked Moved
    General pfSense Questions
    2
    12
    989
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      igb1 is the backup WAN?

      The issue in the linked thread is if an internal interface changes state and it not set to track a WAN for IPv6. So this is not that.

      Packages should restart if a WAN changes IP. Anything that uses that IP would need to.

      What is igb1 connected to? Why is it losing link to that?

      Steve

      X 1 Reply Last reply Reply Quote 0
      • X
        Ximulate @stephenw10
        last edited by

        @stephenw10 Yes, igb1 is the backup WAN. Its connected to a Netgear LB1121 (https://www.netgear.com/support/product/lb1121).

        I have the Gateway Group set-up as a standard Failover, per the docs. So igb1 is Tier 2. Last night, I Enabled Gateway Monitoring & Gateway Action with default Monitoring IP. And, rebooted the router and the Netgear modem. Per the log, things calmed down then about 2:30am it picked up again thru about abput 4am. Since 4am I haven't seen any package restarts.

        On a whim, I just deleted two cron jobs (one was a shutdown -r) but neither were scheduled for those times.

        My primary Gateway (igb0) is a wISP. The bandwidth gets saturated at times, eventually leading to the failover. That was happening a lot the last several days (holiday weekend, so the wISP gets hammered by its customers).

        Would frequent switching between gateways cause this? I've tried bumping up probe times (& related settings). Not really sure how else to deal with the GG switching during these episodes. I've also wondered if the Monitoring IP might be an issue; perhaps 1.1.1.1, 8.8.8.8, 9.9.9.9, etc are throttling the probes.

        Other thing is, I didn't notice this until recently & wonder if its a V2.7 issue.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          I wouldn't expect to see igb1 actually lose link whatever the gateway is doing. Does the Netgear modem device also log a loss of link?

          X 1 Reply Last reply Reply Quote 0
          • X
            Ximulate @stephenw10
            last edited by Ximulate

            I don't see a way to read logs from the Netgear LTE modem.

            The router logs show this activity on until about 4am again, then calm. Issue picked up again this morning, I suspect about the time I turned on the Android TV. This morning I decided to disconnect the Neatgear LTE modem. So far, no issues.

            I'm now wondering if the Primary WAN Latency Threshold & Packet Loss Threshold are too low, currently:
            Latency: 350/500
            Packet Loss: 15/35
            So, AndroidTV (or some other devices) starts streaming, my wISP bandwidth get saturated. However, doesn't make sense that this is happening overnight until 4am.

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              It also shouldn't cause the NIC to actually lose link. That should only happen if the NIC has it's setting changed or the other end of the link (the modem) drops it.

              X 1 Reply Last reply Reply Quote 0
              • X
                Ximulate @stephenw10
                last edited by Ximulate

                @stephenw10 Thanks. Since disconnecting the Netgear LTE modem, the system logs have been quiter. So maybe the modem or some other hardware issue (that I will narrow down.) To further complicate things, in reviewing last nights system log (below) its appears to me that maybe the wISP is losing connection or Quad9 is having issues. Reviewing logs from Sep 1, I see the same behaviour: GW_Primary alarm at 2:36am with various log e tries until about 04:00. I think this has been ongoing for some time, and I have tried different Monitor IPs during that time so I don't think its Quad9

                Sep 7 03:49:24 	php-fpm 	87590 	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 03:49:23 	php-fpm 	1107 	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use GW_Primary.
Sep 7 03:49:21 	check_reload_status 	412 	Reloading filter
Sep 7 03:49:21 	check_reload_status 	412 	Restarting OpenVPN tunnels/interfaces
Sep 7 03:49:21 	check_reload_status 	412 	Restarting IPsec tunnels
Sep 7 03:49:21 	check_reload_status 	412 	updating dyndns GW_Primary
Sep 7 03:49:21 	rc.gateway_alarm 	95468 	>>> Gateway alarm: GW_Primary (Addr:9.9.9.9 Alarm:0 RTT:23.544ms RTTsd:6.640ms Loss:0%)
Sep 7 03:49:10 	php-cgi 	95611 	notify_monitor.php: Message sent to dev.note@domain.org OK
Sep 7 03:48:59 	php-fpm 	18413 	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 03:48:58 	php-fpm 	22 	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use GW_Primary.
Sep 7 03:48:58 	php-fpm 	22 	9.9.9.9|38.XX.XX.Y7|GW_Primary|29.876ms|7.593ms|0.0%|online|none
Sep 7 03:48:58 	php-fpm 	22 	/rc.openvpn: MONITOR: GW_Primary is available now, adding to routing group GWG_Failover
Sep 7 03:48:56 	check_reload_status 	412 	Reloading filter
Sep 7 03:48:56 	check_reload_status 	412 	Restarting OpenVPN tunnels/interfaces
Sep 7 03:48:56 	check_reload_status 	412 	Restarting IPsec tunnels
Sep 7 03:48:56 	check_reload_status 	412 	updating dyndns GW_Primary
Sep 7 03:48:56 	rc.gateway_alarm 	63076 	>>> Gateway alarm: GW_Primary (Addr:9.9.9.9 Alarm:1 RTT:29.821ms RTTsd:7.671ms Loss:0%)
Sep 7 03:47:44 	php-fpm 	383 	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 03:47:43 	php-fpm 	1107 	/rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 03:47:43 	php-fpm 	3255 	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use GW_Primary.
Sep 7 03:47:41 	check_reload_status 	412 	Reloading filter
Sep 7 03:47:41 	check_reload_status 	412 	Restarting OpenVPN tunnels/interfaces
Sep 7 03:47:41 	check_reload_status 	412 	Restarting IPsec tunnels
Sep 7 03:47:41 	check_reload_status 	412 	updating dyndns GW_Primary
Sep 7 03:47:41 	rc.gateway_alarm 	30240 	>>> Gateway alarm: GW_Primary (Addr:9.9.9.9 Alarm:1 RTT:4219.342ms RTTsd:3736.108ms Loss:75%)
Sep 7 03:45:01 	php-cgi 	82400 	rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 03:30:01 	php-cgi 	57273 	rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 03:15:02 	php-cgi 	83850 	rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 03:00:02 	php-cgi 	53983 	rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 02:45:01 	php-cgi 	21419 	rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 02:37:03 	php-cgi 	63727 	notify_monitor.php: Could not send the message to dev.note@domain.org -- Error: Failed to connect to ssl://smtp.dreamhost.com:465 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo for smtp.dreamhost.com failed: Address family for hostname not supported (code: -1, response: )]
Sep 7 02:36:28 	php-fpm 	22 	/rc.dyndns.update: phpDynDNS (): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Sep 7 02:36:27 	php-fpm 	87590 	/rc.filter_configure_sync: An error occurred while trying to find the interface got 38.XX.XX.Y6 . The rule has not been added.
Sep 7 02:36:26 	php-fpm 	8672 	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use GW_Primary.
Sep 7 02:36:26 	php-fpm 	8672 	9.9.9.9|38.XX.XX.Y7|GW_Primary|18.317ms|1.492ms|45%|down|highloss
Sep 7 02:36:26 	php-fpm 	8672 	/rc.openvpn: MONITOR: GW_Primary has packet loss, omitting from routing group GWG_Failover
Sep 7 02:36:25 	check_reload_status 	412 	Reloading filter
Sep 7 02:36:25 	check_reload_status 	412 	Restarting OpenVPN tunnels/interfaces
Sep 7 02:36:25 	check_reload_status 	412 	Restarting IPsec tunnels
Sep 7 02:36:25 	check_reload_status 	412 	updating dyndns GW_Primary
Sep 7 02:36:25 	rc.gateway_alarm 	29639 	>>> Gateway alarm: GW_Primary (Addr:9.9.9.9 Alarm:1 RTT:18.495ms RTTsd:1.552ms Loss:40%)
                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Yeah sounds like maybe that modem has an issue. I would not expect it to lose link on the Ethernet like that. Does it really not log anything? I assume it has some sort of UI you have access to?

                  WISPs can be hard to tune the gateway monitoring for. You do need that and gateway action enabled though if you need to failover.

                  X 1 Reply Last reply Reply Quote 0
                  • X
                    Ximulate @stephenw10
                    last edited by

                    @stephenw10 The Netgear modem does have a simple UI, but no obvious way to view logs. I don't see anything in its manual either. It may require Netgear support to get involved.

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Does it show uptime? Any way to know if it rebooted?

                      X 1 Reply Last reply Reply Quote 0
                      • X
                        Ximulate @stephenw10
                        last edited by

                        @stephenw10

                        No, I don't see uptime or any other way to tell if its rebooted. I've tried different cables. I was going to try to swap-out the wall wart & put a switch between the router and the modem. Haven't done that yet. In the meantime, I'm corrosponding with the ISP about the nightly outages.

                        X 1 Reply Last reply Reply Quote 1
                        • X
                          Ximulate @Ximulate
                          last edited by

                          @stephenw10 Factory reset modem seems to have cleared up the link issue. ISP claims nightly outage is scheduled service

                          1 Reply Last reply Reply Quote 1
                          • X Ximulate referenced this topic on
                          • First post
                            Last post