No internet after power outage
-
Hi
After power failure on a site, my pfsense does not route any traffic to internet.
WAN receives an IP adress via DHCP, and when I look at the pfsense dashboard, my Gateway WAN_DHCP is green (and my OpenVPN tunnel gateway is red because it cannot communicate with internet). I have configured the WAN_DHCP as default the default IPv4 Gateway and have no gateway set in the firewall rules.I have compared the logs from a sucessful manual reboot with the failing one and the order of the events for receiving the WAN IP and setting the default gateway seem to differ.
Successful:
Jun 19 18:06:25 php-fpm 408 /rc.newwanip: rc.newwanip: on (IP address: 83.233.101.5) (interface: WAN[wan]) (real interface: mvneta2).
...
Jun 19 18:06:27 php 451 rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.Failing:
Aug 13 09:36:59 php 472 rc.bootup: Gateway, NONE AVAILABLE
...
Aug 13 09:37:52 php-fpm 428 /rc.newwanip: rc.newwanip: on (IP address: 31.208.129.240) (interface: WAN[wan]) (real interface: mvneta2).Is it possible that the routing it not correctly setup if there is no IP adress on the WAN interface when rc.bootup tries to setup the default interface? is there a way to avoid the problem?
Disconnecting and reconnecting the WAN ehternet cable triggers pfsense to reevaluate and after that everything works fine. But the site is remote and since it often faces power outages this is a real problem for us. I have an OpenVPN tunnel to the main site for remote management but that of course requires that the client site gets its internet connection working....
More from the logs;
Successful:
Jun 19 18:06:24 kernel tun1: changing name to 'ovpnc1' Jun 19 18:06:24 kernel e6000sw0port1: link state changed to UP Jun 19 18:06:24 kernel e6000sw0port3: link state changed to UP Jun 19 18:06:24 check_reload_status 447 Linkup starting $e6000sw0port1 Jun 19 18:06:24 check_reload_status 447 Linkup starting $e6000sw0port3 Jun 19 18:06:24 php 451 rc.bootup: The command '/sbin/route -q delete 192.168.254.2' returned exit code '1', the output was 'route: route has not been found' Jun 19 18:06:24 kernel ovpnc1: link state changed to UP Jun 19 18:06:24 check_reload_status 447 rc.newwanip starting ovpnc1 Jun 19 18:06:24 kernel pflog0: promiscuous mode enabled Jun 19 18:06:24 php-fpm 408 /rc.newwanip: rc.newwanip: Info: starting on mvneta2. Jun 19 18:06:25 php-fpm 408 /rc.newwanip: rc.newwanip: on (IP address: 83.233.101.5) (interface: WAN[wan]) (real interface: mvneta2). Jun 19 18:06:25 check_reload_status 447 Linkup starting $e6000sw0port2 Jun 19 18:06:25 kernel e6000sw0port2: link state changed to UP Jun 19 18:06:25 php-fpm 407 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1. Jun 19 18:06:25 php-fpm 407 /rc.newwanip: rc.newwanip: on (IP address: 192.168.254.2) (interface: GWFORVPN[opt2]) (real interface: ovpnc1). Jun 19 18:06:25 kernel done. Jun 19 18:06:27 rc.gateway_alarm 65582 >>> Gateway alarm: GWFORVPN_VPNV4 (Addr:192.168.254.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Jun 19 18:06:27 check_reload_status 447 updating dyndns GWFORVPN_VPNV4 Jun 19 18:06:27 check_reload_status 447 Restarting ipsec tunnels Jun 19 18:06:27 check_reload_status 447 Restarting OpenVPN tunnels/interfaces Jun 19 18:06:27 check_reload_status 447 Reloading filter Jun 19 18:06:27 php 451 rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default. Jun 19 18:06:28 php 451 rc.bootup: Gateway, none 'available' for inet6, use the first one configured. '' Jun 19 18:06:28 kernel done. Jun 19 18:06:47 rc.gateway_alarm 2132 >>> Gateway alarm: GWFORVPN_VPNV4 (Addr:192.168.254.1 Alarm:0 RTT:2.414ms RTTsd:.827ms Loss:11%) Jun 19 18:06:47 check_reload_status 447 updating dyndns GWFORVPN_VPNV4 Jun 19 18:06:47 check_reload_status 447 Restarting ipsec tunnels Jun 19 18:06:47 check_reload_status 447 Restarting OpenVPN tunnels/interfaces Jun 19 18:06:47 check_reload_status 447 Reloading filter Jun 19 18:06:55 php 451 rc.bootup: sync unbound done. Jun 19 18:06:55 kernel done. Jun 19 18:06:56 kernel done. Jun 19 18:06:57 kernel done. Jun 19 18:06:57 php 451 rc.bootup: NTPD is starting up. Jun 19 18:06:57 kernel done. Jun 19 18:06:57 check_reload_status 447 Updating all dyndns Jun 19 18:06:57 kernel done. Jun 19 18:06:58 kernel .... Jun 19 18:06:59 kernel .done. Jun 19 18:06:59 php 451 rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).' Jun 19 18:07:03 php 451 rc.bootup: Creating rrd update script Jun 19 18:07:03 kernel done. Jun 19 18:07:04 kernel done. Jun 19 18:07:04 root 3650 /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one Jun 19 18:07:06 syslogd exiting on signal 15 Jun 19 18:07:06 syslogd kernel boot file is /boot/kernel/kernel Jun 19 18:07:06 kernel done. Jun 19 18:07:06 kernel done. Jun 19 18:07:07 php-fpm 408 /rc.start_packages: Restarting/Starting all packages. Jun 19 18:07:07 root 41146 Bootup complete Jun 19 18:07:08 login 46224 login on ttyu0 as root Jun 19 18:07:10 php 22163 notify_monitor.php: Message sent to XXXXXXXXXXXXXXXXXXXXX OK
Failing:
Aug 13 09:36:56 kernel tun1: changing name to 'ovpnc1' Aug 13 09:36:56 php 472 rc.bootup: The command '/sbin/route -q delete 192.168.254.2' returned exit code '1', the output was 'route: route has not been found' Aug 13 09:36:56 kernel ovpnc1: link state changed to UP Aug 13 09:36:56 check_reload_status 468 rc.newwanip starting ovpnc1 Aug 13 09:36:56 kernel pflog0: promiscuous mode enabled Aug 13 09:36:57 php-fpm 429 /rc.newwanip: rc.newwanip: Info: starting on ovpnc1. Aug 13 09:36:57 php-fpm 429 /rc.newwanip: rc.newwanip: on (IP address: 192.168.254.2) (interface: GWFORVPN[opt2]) (real interface: ovpnc1). Aug 13 09:36:57 kernel .done. Aug 13 09:36:59 rc.gateway_alarm 64344 >>> Gateway alarm: GWFORVPN_VPNV4 (Addr:192.168.254.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Aug 13 09:36:59 check_reload_status 468 updating dyndns GWFORVPN_VPNV4 Aug 13 09:36:59 check_reload_status 468 Restarting IPsec tunnels Aug 13 09:36:59 check_reload_status 468 Restarting OpenVPN tunnels/interfaces Aug 13 09:36:59 check_reload_status 468 Reloading filter Aug 13 09:36:59 php 472 rc.bootup: Gateway, NONE AVAILABLE Aug 13 09:36:59 php 472 rc.bootup: Gateway, none 'available' for inet6, use the first one configured. '' Aug 13 09:36:59 kernel done. Aug 13 09:37:21 php 472 rc.bootup: sync unbound done. Aug 13 09:37:21 kernel done. Aug 13 09:37:21 kernel done. Aug 13 09:37:51 php 472 rc.bootup: The command '/usr/bin/timeout -k 45 30 /usr/local/sbin/ntpd -g -q -c /dev/null 216.239.35.0 216.239.35.4 216.239.35.8 216.239.35.12' returned exit code '124', the output was '13 Aug 09:37:21 ntpd[150]: ntpd 4.2.8p15@1.3728-o Thu Jan 13 01:10:15 UTC 2022 (1): Starting 13 Aug 09:37:21 ntpd[150]: Command line: /usr/local/sbin/ntpd -g -q -c /dev/null 216.239.35.0 216.239.35.4 216.239.35.8 216.239.35.12 XXXXXXXXXXXXXXXXXXX (cut) Aug 13 09:37:51 kernel done. Aug 13 09:37:51 php 472 rc.bootup: NTPD is starting up. Aug 13 09:37:51 check_reload_status 468 rc.newwanip starting mvneta2 Aug 13 09:37:51 kernel done. Aug 13 09:37:52 php-fpm 428 /rc.newwanip: rc.newwanip: Info: starting on mvneta2. Aug 13 09:37:52 php-fpm 428 /rc.newwanip: rc.newwanip: on (IP address: 31.208.129.240) (interface: WAN[wan]) (real interface: mvneta2). Aug 13 09:37:52 kernel done. Aug 13 09:37:53 check_reload_status 468 Updating all dyndns Aug 13 09:37:53 kernel done. Aug 13 09:37:54 rc.gateway_alarm 21762 >>> Gateway alarm: GWFORVPN_VPNV4 (Addr:192.168.254.1 Alarm:1 RTT:0.000ms RTTsd:0.000ms Loss:100%) Aug 13 09:37:54 check_reload_status 468 updating dyndns GWFORVPN_VPNV4 Aug 13 09:37:54 check_reload_status 468 Restarting IPsec tunnels Aug 13 09:37:54 check_reload_status 468 Restarting OpenVPN tunnels/interfaces Aug 13 09:37:54 check_reload_status 468 Reloading filter Aug 13 09:37:54 kernel .. Aug 13 09:37:55 kernel .. Aug 13 09:37:56 php 472 rc.bootup: The command '/usr/local/sbin/strongswanrc stop' returned exit code '1', the output was 'strongswan not running? (check /var/run/daemon-charon.pid).' Aug 13 09:37:56 kernel .done. Aug 13 09:38:00 php 472 rc.bootup: Creating rrd update script Aug 13 09:38:01 kernel done. Aug 13 09:38:01 kernel done. Aug 13 09:38:02 root 86084 /etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one Aug 13 09:38:04 syslogd exiting on signal 15 Aug 13 09:38:04 syslogd kernel boot file is /boot/kernel/kernel Aug 13 09:38:04 kernel done. Aug 13 09:38:04 kernel done. Aug 13 09:38:05 php 2992 notify_monitor.php: Could not send the message to XXXXXXXXXXXXXXXXX -- Error: Failed to connect to smtp.office365.com:587 [SMTP: Failed to connect socket: Network is unreachable (code: -1, response: )]
BR
/Marcus