Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    WAN repeatedly going down every few days again

    Scheduled Pinned Locked Moved
    General pfSense Questions
    3
    14
    238
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pp-ng @stephenw10
      last edited by

      @stephenw10 It's the logs from Status > System Logs > Gateway

      So when or if it happens again, what things should I look for or do?

      Is there more debug logging I can enable to help narrow it down if it happens again?

      1 Reply Last reply Reply Quote 0
      • P
        pp-ng @stephenw10
        last edited by

        @stephenw10 Also - to get my WAN back to 'online' I went into Interfaces > WAN and just clicked Save and then Activate. I know it runs several scripts or whatever in the background, so not sure which one got me back online, but that did it.

        JeGrJ 1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Right so check in the main System > General tab in the logs at that time. It will show interfaces losing link, processes restarting, IP address changes etc.

          P 1 Reply Last reply Reply Quote 0
          • P
            pp-ng @stephenw10
            last edited by

            @stephenw10 Here you go

            Sep 1 09:31:00 sshguard 32697 Now monitoring attacks.
            Sep 1 09:31:00 sshguard 44278 Exiting on signal.
            Sep 1 09:25:00 php 80793 /usr/local/sbin/acbupload.php: End of configuration backup to https://acb.netgate.com/save (success).
            Sep 1 09:24:11 php-fpm 4605 /rc.start_packages: Restarting/Starting all packages.
            Sep 1 09:24:11 php-fpm 4605 /rc.start_packages: Restarting/Starting all packages.
            Sep 1 09:24:10 check_reload_status 630 Starting packages
            Sep 1 09:24:10 php-fpm 21640 /interfaces.php: Creating rrd update script
            Sep 1 09:24:10 check_reload_status 630 Reloading filter
            Sep 1 09:24:10 check_reload_status 630 Starting packages
            Sep 1 09:24:10 check_reload_status 630 Reloading filter
            Sep 1 09:24:10 php-fpm 4605 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> xxx.xxx.xxx.xxx - Restarting packages.
            Sep 1 09:24:10 php-fpm 21640 /interfaces.php: Removing static route for monitor 1.1.1.1 and adding a new route through xxx.xxx.xxx.xxx
            Sep 1 09:24:10 php-fpm 21640 /interfaces.php: Resyncing OpenVPN instances for interface WAN.
            Sep 1 09:24:08 php-fpm 4605 /rc.newwanip: Creating rrd update script
            Sep 1 09:24:08 php-fpm 4605 /rc.newwanip: Resyncing OpenVPN instances for interface WAN.
            Sep 1 09:24:08 check_reload_status 630 updating dyndns wan
            Sep 1 09:24:08 check_reload_status 630 Restarting IPsec tunnels
            Sep 1 09:24:08 php-fpm 4605 /rc.newwanip: IP Address has changed, killing states on former IP Address 0.0.0.0.
            Sep 1 09:24:08 php-fpm 21640 /interfaces.php: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
            Sep 1 09:24:08 php-fpm 4605 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
            Sep 1 09:24:07 php-fpm 4605 /rc.newwanip: Removing static route for monitor 1.1.1.1 and adding a new route through xxx.xxx.xxx.xxx
            Sep 1 09:24:07 php-fpm 4605 /rc.newwanip: rc.newwanip: on (IP address: xxx.xxx.xxx.xxx) (interface: WAN[wan]) (real interface: igb0).
            Sep 1 09:24:07 php-fpm 4605 /rc.newwanip: rc.newwanip: Info: starting on igb0.
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: Starting rtsold process on wan(igb0)
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: Starting DHCP6 client for interfaces igb0 in DHCP6 without RA mode
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: Accept router advertisements on interface igb0
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: calling interface_dhcpv6_configure.
            Sep 1 09:24:06 check_reload_status 630 rc.newwanip starting igb0
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: Error starting gateway monitor for WAN_DHCP
            Sep 1 09:24:06 php-fpm 21640 /interfaces.php: The command '/usr/local/bin/dpinger -S -r 0 -i WAN_DHCP -B xxx.xxx.xxx.xxx -p /var/run/dpinger_WAN_DHCP~xxx.xxx.xxx.xxx~1.1.1.1.pid -u /var/run/dpinger_WAN_DHCP~xxx.xxx.xxx.xxx~1.1.1.1.sock -C "/etc/rc.gateway_alarm" -d 1 -s 500 -l 2000 -t 60000 -A 1000 -D 500 -L 20 1.1.1.1 >/dev/null' returned exit code '1', the output was ''
            Sep 1 09:24:03 php 1125 /usr/local/sbin/acbupload.php: New alert found: An error occurred while uploading the encrypted Netgate pfSense Plus configuration to https://acb.netgate.com/save ( Unable to resolve acb.netgate.com )
            Sep 1 09:24:03 php 1125 /usr/local/sbin/acbupload.php: An error occurred while uploading the encrypted Netgate pfSense Plus configuration to https://acb.netgate.com/save ( Unable to resolve acb.netgate.com ) - Unable to resolve acb.netgate.com
            Sep 1 09:24:01 php-fpm 35200 /interfaces.php: Beginning configuration backup to https://acb.netgate.com/save
            Sep 1 09:24:01 check_reload_status 630 Syncing firewall
            Sep 1 09:24:01 php-fpm 35200 /interfaces.php: Configuration Change: admin@xxx.xxx.xxx.xxx (Local Database): Interfaces settings changed
            Sep 1 09:23:52 php-fpm 21640 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
            Sep 1 09:23:51 check_reload_status 630 Updating all dyndns
            Sep 1 09:23:51 check_reload_status 630 Reloading filter
            Sep 1 09:23:51 check_reload_status 630 Reloading filter
            Sep 1 09:23:51 check_reload_status 630 Restarting OpenVPN tunnels/interfaces
            Sep 1 09:23:51 check_reload_status 630 Restarting IPsec tunnels
            Sep 1 09:23:51 check_reload_status 630 updating dyndns WAN_DHCP
            Sep 1 09:23:51 rc.gateway_alarm 8515 >>> Gateway alarm: WAN_DHCP (Addr:1.1.1.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%)
            Sep 1 09:23:51 php-fpm 33576 /system_gateways.php: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
            Sep 1 09:23:49 php-fpm 33576 /system_gateways.php: Removing static route for monitor 1.1.1.1 and adding a new route through xxx.xxx.xxx.xxx
            Sep 1 09:23:47 php-fpm 33576 /system_gateways_edit.php: Beginning configuration backup to https://acb.netgate.com/save
            Sep 1 09:23:47 check_reload_status 630 Syncing firewall
            Sep 1 09:23:47 php-fpm 33576 /system_gateways_edit.php: Configuration Change: admin@xxx.xxx.xxx.xxx (Local Database): Gateway settings changed
            Sep 1 09:23:36 php-fpm 33576 /index.php: Successful login for user 'admin' from: xxx.xxx.xxx.xxx (Local Database)
            Sep 1 09:22:14 php-fpm 35200 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
            Sep 1 09:22:13 check_reload_status 630 Reloading filter
            Sep 1 09:22:13 check_reload_status 630 Restarting OpenVPN tunnels/interfaces
            Sep 1 09:22:13 check_reload_status 630 Restarting IPsec tunnels
            Sep 1 09:22:13 check_reload_status 630 updating dyndns WAN_DHCP
            Sep 1 09:22:13 rc.gateway_alarm 91060 >>> Gateway alarm: WAN_DHCP (Addr:1.1.1.1 Alarm:1 RTT:4.395ms RTTsd:.763ms Loss:21%)

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Ah, OK so a lot of those changes in the gateway log are because you logged in and resaved the gateway settings.

              The initial packet loss itself doesn't trigger anything more than the alarm and the reload that brings.

              Packet loss on the WAN is usually actually just packet loss. So some upstream routing or connection issue.

              P 1 Reply Last reply Reply Quote 0
              • P
                pp-ng @stephenw10
                last edited by

                @stephenw10 So do we know what the issue could be or what should I do next time it happens?

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  The logs seem to show the WAN just started dropping packets.

                  Did you login and resave that in order to bring the WAN back up? Does it not come back up if you do not?

                  P 1 Reply Last reply Reply Quote 0
                  • P
                    pp-ng @stephenw10
                    last edited by

                    @stephenw10 to get it back up I went into Interfaces > WAN and saved.

                    Prior to that I might have gone into System > Routing and also saved, I cannot recall.

                    Why do these show up? I don't have any VPNs configured? Or is that just something that happens

                    Sep 1 09:23:51 check_reload_status 630 Restarting OpenVPN tunnels/interfaces
                    Sep 1 09:23:51 check_reload_status 630 Restarting IPsec tunnels

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Yes those lines are shown when the script runs but don't actually do anything if you don't have tunnels configured.

                      1 Reply Last reply Reply Quote 0
                      • JeGrJ
                        JeGr LAYER 8 Moderator @pp-ng
                        last edited by

                        @pp-ng said in WAN repeatedly going down every few days again:

                        @stephenw10 Also - to get my WAN back to 'online' I went into Interfaces > WAN and just clicked Save and then Activate. I know it runs several scripts or whatever in the background, so not sure which one got me back online, but that did it.

                        Sounds to me more like a configuration problem on your WAN or your ISP/upstream provider. That somehow smells like you have DHCP on WAN and your box looses its connection because the DHCP address expired or your provider doesn't "know" it anymore. Or you get a quasi static IP from your ISP and configured it as static IP but your ISP needs to hand it out via DHCP.

                        We had some of that use cases in support here and most of them had that exact problem. Saving WAN config brought them online and after a few hours or days the connection dropped 'cause packages wouldn't go out/in anymore. Switching the WAN e.g. from static to DHCP or configuring it the way your ISP needs/wants it could solve that. Or check the ISP modem or %device% depending on your internet. That a manual "save & apply" from WAN brings you back seems to indicate that a manual performed DHCP restart seems to work, so I'd have a look at the way you get your IP from your ISP.

                        Cheers :)

                        Don't forget to upvote 👍 those who kindly offered their time and brainpower to help you!

                        If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post