SG-1000 Lost WAN Access

[borsaid]

We have an SG-1000 deployed in at a remote satellite office. This morning the two person office lost all WAN connectivity. Other than IPSec to the main office, DHCP, DNS, there are no other major services running on the SG-1000.

We confirmed with the internet provider that everything was good on their end. Able to remotely reboot the modem, and still didn't have internet access.

The ultimate resolution was a manual power cycle of the SG-1000.

Now I'm trying to discover what may have gone wrong. Here are the log entries surrounding the time in which we went down:

May 15 10:20:05 check_reload_status updating dyndns WANGW
May 15 10:20:05 check_reload_status Restarting ipsec tunnels
May 15 10:20:05 check_reload_status Restarting OpenVPN tunnels/interfaces
May 15 10:20:05 check_reload_status Reloading filter
May 15 10:20:21 php-fpm 47637 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
May 15 10:20:21 check_reload_status Reloading filter
May 15 10:54:16 check_reload_status Linkup starting cpsw0
May 15 10:54:16 kernel cpsw0: link state changed to DOWN
May 15 10:54:17 php-fpm 29789 /rc.linkup: Hotplug event detected for WAN(wan) static IP (xx.xx.xx.xxx )
May 15 10:54:17 check_reload_status Reloading filter
May 15 10:54:18 kernel arpresolve: can't allocate llinfo for yy.yy.yy.yyy(WAN Gateway IP) on cpsw0
May 15 10:54:23 kernel arpresolve: can't allocate llinfo for yy.yy.yy.yyy on cpsw0
May 15 10:54:24 kernel arpresolve: can't allocate llinfo for yy.yy.yy.yyy on cpsw0
May 15 10:54:35 kernel arpresolve: can't allocate llinfo for yy.yy.yy.yyy on cpsw0
May 15 10:54:36 check_reload_status Linkup starting cpsw0
May 15 10:54:36 kernel cpsw0: link state changed to UP
May 15 10:54:37 php-fpm 60729 /rc.linkup: Hotplug event detected for WAN(wan) static IP (xx.xx.xx.xxx )
May 15 10:54:37 check_reload_status rc.newwanip starting cpsw0
May 15 10:54:37 check_reload_status Reloading filter
May 15 10:54:38 php-fpm 29789 /rc.newwanip: rc.newwanip: Info: starting on cpsw0.
May 15 10:54:38 php-fpm 29789 /rc.newwanip: rc.newwanip: on (IP address: xx.xx.xx.xxx) (interface: WAN[wan]) (real interface: cpsw0).
May 15 10:54:38 php-fpm 29789 /rc.newwanip: IP has changed, killing states on former IP 0.0.0.0.
May 15 10:58:29 php-fpm 29789 /rc.newwanip: ROUTING: setting default route to yy.yy.yy.yyy
May 15 10:58:33 check_reload_status updating dyndns WANGW
May 15 10:58:33 check_reload_status Restarting ipsec tunnels
May 15 10:58:33 check_reload_status Restarting OpenVPN tunnels/interfaces
May 15 10:58:33 check_reload_status Reloading filter
May 15 10:58:49 php-fpm 47637 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
May 15 10:58:49 check_reload_status Reloading filter
May 15 10:59:26 php-fpm 29789 /rc.newwanip: Forcefully reloading IPsec
May 15 11:02:30 syslogd kernel boot file is /boot/kernel/kernel
May 15 11:02:30 kernel Copyright 1992-2016 The FreeBSD Project.

We believe the above log includes when the issue started, shows when the cable modem was rebooted, and when we finally forcefully power cycled the SG-1000.