Unreliable WAN_DHCP and WAN_DHCP6 Gateways after upgrading to CE 2.9.0
-
I upgraded to CD 2.9.0 yesterday and I had to switch back to 2.8.1 because it became increasingly unstable. I wasn't even able to get it to shutdown properly, so I halted it and restored the 2.8.1 checkpoint.
I pasted log messages below. I could have posted many more, but there is a limit on the length.
My impression is that the beta test period for 2.9.0 was extremely short. In the many years I've used pfSense, I don't recall such a short beta period, without even a release candidate.
Aug 23 09:29:26 php-fpm 33864 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:29:25 rtsold 45595 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:29:24 check_reload_status 503 Reloading filter Aug 23 09:29:24 check_reload_status 503 updating dyndns wan Aug 23 09:29:23 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:29:23 php-fpm 94842 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:29:23 php-fpm 815 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:29:23 php-fpm 815 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:29:23 php-fpm 815 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:29:21 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:29:21 php-fpm 94842 NOTICE HOTPLUG: Configuring interface wan Aug 23 09:29:21 php-fpm 94842 NOTICE DEVD Ethernet attached event for wan Aug 23 09:29:21 php-fpm 94842 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:29:20 kernel hn0: link state changed to UP Aug 23 09:29:20 check_reload_status 503 Linkup starting hn0 Aug 23 09:29:19 php-fpm 59885 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:29:19 php-fpm 65823 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:29:19 php-fpm 65823 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:29:19 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:29:19 php-fpm 94842 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:29:19 php-fpm 94842 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:29:18 check_reload_status 503 Reloading filter Aug 23 09:29:18 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:29:18 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:29:18 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:29:18 check_reload_status 503 Reloading filter Aug 23 09:29:18 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:29:18 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:29:18 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:29:16 php-fpm 59885 NOTICE DEVD Ethernet detached event for wan Aug 23 09:29:16 php-fpm 59885 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:29:15 php-fpm 59885 WARNING Could not check for system updates: DNS servers not available Aug 23 09:29:15 kernel hn0: link state changed to DOWN Aug 23 09:29:15 kernel hn0: network changed, change 1 Aug 23 09:29:15 check_reload_status 503 Linkup starting hn0 Aug 23 09:29:00 php-fpm 32041 WARNING Could not check for system updates: DNS servers not available Aug 23 09:28:54 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:54 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:28:54 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:54 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:28:54 php-fpm 94842 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:28:54 php-fpm 815 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:28:53 check_reload_status 503 Reloading filter Aug 23 09:28:53 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:28:53 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:28:53 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:28:53 check_reload_status 503 Reloading filter Aug 23 09:28:53 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:28:53 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:28:53 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:28:53 php-fpm 66209 WARNING Could not check for system updates: DNS servers not available Aug 23 09:28:30 check_reload_status 503 Reloading filter Aug 23 09:28:30 check_reload_status 503 Starting packages Aug 23 09:28:30 php-fpm 66209 NOTICE pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 216.232.118.152 - Restarting packages. Aug 23 09:28:27 php-fpm 59885 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:28:27 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:28:26 php-fpm 66209 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787502506] unbound[16382:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787502506] unbound[16382:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787502506] unbound[16382:0] error: bind: address already in use [1787502506] unbound[16382:0] fatal error: could not open ports' Aug 23 09:28:26 check_reload_status 503 Reloading filter Aug 23 09:28:26 check_reload_status 503 updating dyndns wan Aug 23 09:28:25 rtsold 1684 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:28:24 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:28:24 php-fpm 94842 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:24 php-fpm 66209 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:28:24 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:24 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:28:23 kernel cannot forward src fe80:6::1:b3ff:fedd:9f24, dst 2001:569:5b86:b500:215:5dff:fe5c:f601, nxt 58, rcvif hn1, outif hn0 Aug 23 09:28:22 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:28:22 kernel cannot forward src fe80:6::1:b3ff:fedd:9f24, dst 2001:569:5b86:b500:215:5dff:fe5c:f601, nxt 58, rcvif hn1, outif hn0 Aug 23 09:28:22 php-fpm 94842 NOTICE HOTPLUG: Configuring interface wan Aug 23 09:28:22 php-fpm 94842 NOTICE DEVD Ethernet attached event for wan Aug 23 09:28:22 php-fpm 94842 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:28:21 kernel cannot forward src fe80:6::1:b3ff:fedd:9f24, dst 2001:569:5b86:b500:215:5dff:fe5c:f601, nxt 58, rcvif hn1, outif hn0 Aug 23 09:28:21 kernel hn0: link state changed to UP Aug 23 09:28:21 check_reload_status 503 Linkup starting hn0 Aug 23 09:28:20 check_reload_status 503 Reloading filter Aug 23 09:28:20 php-fpm 70922 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:20 php-fpm 70922 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:28:20 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:28:20 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:28:20 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:28:20 php-fpm 94842 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:28:19 check_reload_status 503 Reloading filter Aug 23 09:28:19 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:28:19 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:28:19 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:28:19 check_reload_status 503 Reloading filter Aug 23 09:28:19 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:28:19 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:28:19 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:28:17 php-fpm 32041 NOTICE DEVD Ethernet detached event for wan Aug 23 09:28:17 php-fpm 32041 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:28:16 kernel hn0: link state changed to DOWN Aug 23 09:28:16 kernel hn0: network changed, change 1 Aug 23 09:28:16 check_reload_status 503 Linkup starting hn0 Aug 23 09:27:53 php-fpm 815 WARNING Could not check for system updates: DNS servers not available Aug 23 09:27:50 php-fpm 70922 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:50 php-fpm 70922 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:50 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:50 php-fpm 93800 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:50 php-fpm 94842 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:50 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:49 check_reload_status 503 Reloading filter Aug 23 09:27:49 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:49 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:49 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:27:49 check_reload_status 503 Reloading filter Aug 23 09:27:49 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:49 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:49 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:27:44 check_reload_status 503 Reloading filter Aug 23 09:27:44 check_reload_status 503 Starting packages Aug 23 09:27:44 php-fpm 33864 NOTICE pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 216.232.118.152 - Restarting packages. Aug 23 09:27:41 php-fpm 94842 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:27:41 php-fpm 33864 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:41 check_reload_status 503 Starting packages Aug 23 09:27:41 check_reload_status 503 Reloading filter Aug 23 09:27:40 rtsold 75309 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:27:40 php-fpm 94842 NOTICE The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_hn0.pid -A /var/etc/rtsold_hn0_script.sh -R /usr/bin/true hn0' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Aug 23 09:27:40 php-fpm 33864 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:27:40 php-fpm 33864 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:38 php-fpm 32041 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:38 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:27:37 php-fpm 815 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:37 check_reload_status 503 updating dyndns wan Aug 23 09:27:37 check_reload_status 503 Reloading filter Aug 23 09:27:37 php-fpm 815 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:27:36 php-fpm 93800 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787502456] unbound[9092:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787502456] unbound[9092:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787502456] unbound[9092:0] error: bind: address already in use [1787502456] unbound[9092:0] fatal error: could not open ports' Aug 23 09:27:36 check_reload_status 503 updating dyndns wan Aug 23 09:27:36 rtsold 79414 <call_script> wait(): Resource temporarily unavailable Aug 23 09:27:36 php-fpm 94842 NOTICE The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.hn0.pid hn0 > /tmp/hn0_output 2> /tmp/hn0_error_output' returned exit code '15', the output was '' Aug 23 09:27:36 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:36 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:36 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:36 php-fpm 66209 NOTICE The command '/usr/sbin/rtsold -1 -p /var/run/rtsold_hn0.pid -A /var/etc/rtsold_hn0_script.sh -R /usr/bin/true hn0' returned exit code '1', the output was 'rtsold: failed to open pidfile: File exists' Aug 23 09:27:36 rtsold 83994 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:27:35 check_reload_status 503 Configuring interface wan Aug 23 09:27:35 php-fpm 59885 WARNING rc.newwanip: Failed to update wan IP, restarting... Aug 23 09:27:35 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:35 php-fpm 93800 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:35 php-fpm 93800 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:34 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:27:34 check_reload_status 503 Configuring interface wan Aug 23 09:27:34 php-fpm 94842 WARNING rc.newwanip: Failed to update wan IP, restarting... Aug 23 09:27:33 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:27:32 php-fpm 93800 NOTICE The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf -p /var/run/dhclient.hn0.pid hn0 > /tmp/hn0_output 2> /tmp/hn0_error_output' returned exit code '15', the output was '' Aug 23 09:27:32 php-fpm 66209 NOTICE HOTPLUG: Configuring interface wan Aug 23 09:27:32 php-fpm 66209 NOTICE DEVD Ethernet attached event for wan Aug 23 09:27:32 php-fpm 66209 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:27:32 kernel hn0: link state changed to UP Aug 23 09:27:31 check_reload_status 503 Linkup starting hn0 Aug 23 09:27:30 check_reload_status 503 Reloading filter Aug 23 09:27:30 php-fpm 32041 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:30 php-fpm 32041 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:30 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:30 php-fpm 33864 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:30 php-fpm 33864 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:30 php-fpm 815 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:29 check_reload_status 503 Reloading filter Aug 23 09:27:29 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:29 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:29 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:27:29 check_reload_status 503 Reloading filter Aug 23 09:27:29 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:29 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:29 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:27:27 php-fpm 70922 NOTICE DEVD Ethernet detached event for wan Aug 23 09:27:27 php-fpm 70922 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:27:27 kernel hn0: link state changed to DOWN Aug 23 09:27:27 kernel hn0: network changed, change 1 Aug 23 09:27:26 check_reload_status 503 Linkup starting hn0 Aug 23 09:27:26 check_reload_status 503 Syncing firewall Aug 23 09:27:26 php-fpm 93800 NOTICE [Config] Configuration Change: admin@10.28.92.20 (Local Database): Interfaces settings changed Aug 23 09:27:15 php-fpm 815 WARNING Could not check for system updates: DNS servers not available Aug 23 09:27:05 php-fpm 33864 WARNING Could not check for system updates: DNS servers not available Aug 23 09:27:02 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:02 php-fpm 93800 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:01 php-fpm 93800 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:01 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:27:01 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:27:01 php-fpm 59885 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:27:00 check_reload_status 503 Reloading filter Aug 23 09:27:00 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:00 check_reload_status 503 Reloading filter Aug 23 09:27:00 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:27:00 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:00 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:27:00 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:27:00 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:26:57 check_reload_status 503 Reloading filter Aug 23 09:26:57 check_reload_status 503 Starting packages Aug 23 09:26:57 php-fpm 94842 NOTICE pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 216.232.118.152 - Restarting packages. Aug 23 09:26:55 php-fpm 92723 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:26:54 check_reload_status 503 Reloading filter Aug 23 09:26:54 check_reload_status 503 updating dyndns wan Aug 23 09:26:54 php-fpm 33864 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787502414] unbound[30752:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787502414] unbound[30752:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787502414] unbound[30752:0] error: bind: address already in use [1787502414] unbound[30752:0] fatal error: could not open ports' Aug 23 09:26:54 rtsold 28818 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:26:54 php-fpm 94842 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:26:52 php-fpm 93800 WARNING Could not check for system updates: DNS servers not available Aug 23 09:26:52 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:52 php-fpm 33864 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:51 php-fpm 94842 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:26:51 php-fpm 94842 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:51 php-fpm 94842 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:50 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:26:49 php-fpm 33864 NOTICE HOTPLUG: Configuring interface wan Aug 23 09:26:49 php-fpm 33864 NOTICE DEVD Ethernet attached event for wan Aug 23 09:26:49 php-fpm 33864 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:26:49 php-fpm 815 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:49 php-fpm 815 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:48 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:48 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:48 kernel hn0: link state changed to UP Aug 23 09:26:48 check_reload_status 503 Linkup starting hn0 Aug 23 09:26:47 check_reload_status 503 Reloading filter Aug 23 09:26:47 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:26:47 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:47 check_reload_status 503 Reloading filter Aug 23 09:26:47 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:26:47 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:47 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:26:46 php-fpm 59885 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:26:46 php-fpm 815 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:46 php-fpm 815 NOTICE Default gateway setting Interface WAN_DHCP Gateway as default. Aug 23 09:26:46 php-fpm 815 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:46 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:46 php-fpm 59885 NOTICE Default gateway setting Interface WAN_DHCP Gateway as default. Aug 23 09:26:46 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:46 php-fpm 92723 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:26:45 check_reload_status 503 Reloading filter Aug 23 09:26:45 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:26:45 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:45 check_reload_status 503 Reloading filter Aug 23 09:26:45 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:26:45 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:26:45 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:45 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:26:44 php-fpm 92723 WARNING Could not check for system updates: DNS servers not available Aug 23 09:26:44 php-fpm 93800 NOTICE DEVD Ethernet detached event for wan Aug 23 09:26:44 php-fpm 93800 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:26:43 kernel hn0: link state changed to DOWN Aug 23 09:26:43 kernel hn0: network changed, change 1 Aug 23 09:26:43 check_reload_status 503 Linkup starting hn0 Aug 23 09:26:42 check_reload_status 503 Starting packages Aug 23 09:26:42 check_reload_status 503 Reloading filter Aug 23 09:26:39 php-fpm 70922 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787502399] unbound[72:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787502399] unbound[72:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787502399] unbound[72:0] error: bind: address already in use [1787502399] unbound[72:0] fatal error: could not open ports' Aug 23 09:26:29 php-fpm 66209 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:26:29 php-fpm 93800 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:26:29 php-fpm 66209 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:26:28 check_reload_status 503 Reloading filter Aug 23 09:26:28 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:26:28 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:26:28 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:25:55 check_reload_status 503 Reloading filter Aug 23 09:25:55 check_reload_status 503 Starting packages Aug 23 09:25:55 php-fpm 59885 NOTICE pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 216.232.118.152 - Restarting packages. Aug 23 09:25:53 php-fpm 93800 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:25:52 php-fpm 59885 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:25:51 rtsold 52360 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:25:51 check_reload_status 503 updating dyndns wan Aug 23 09:25:50 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:25:50 php-fpm 70922 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:50 php-fpm 59885 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:25:50 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:50 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:25:48 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:25:47 check_reload_status 503 Reloading filter Aug 23 09:25:47 check_reload_status 503 Starting packages Aug 23 09:25:47 php-fpm 55435 NOTICE pfSense package system has detected an IP change or dynamic WAN reconnection - 0.0.0.0 -> 216.232.118.152 - Restarting packages. Aug 23 09:25:45 php-fpm 815 NOTICE rc.newwanipv6: No IPv6 address found for interface WAN [wan]. Aug 23 09:25:44 check_reload_status 503 Reloading filter Aug 23 09:25:44 check_reload_status 503 updating dyndns wan Aug 23 09:25:44 php-fpm 55435 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:25:44 php-fpm 93800 NOTICE The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1787502344] unbound[68305:0] warning: setsockopt(..., SO_SNDBUF, ...) was not granted: No buffer space available [1787502344] unbound[68305:0] warning: so-sndbuf 4194304 was not granted. Got 57344. To fix: start with root permissions(linux) or sysctl bigger net.core.wmem_max(linux) or kern.ipc.maxsockbuf(bsd) values. or set so-sndbuf: 0 (use system value). [1787502344] unbound[68305:0] error: bind: address already in use [1787502344] unbound[68305:0] fatal error: could not open ports' Aug 23 09:25:44 rtsold 69607 NOTICE [DHCP Client] Received RA specifying route fe80::6aab:9ff:fe9b:4801 for interface wan(hn0) Aug 23 09:25:43 check_reload_status 503 Syncing firewall Aug 23 09:25:43 php-fpm 92723 NOTICE [Config] Configuration Change: admin@10.28.92.20 (Local Database): Interfaces settings changed Aug 23 09:25:42 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:25:42 php-fpm 93800 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:42 php-fpm 55435 NOTICE [Filter] IP Address has changed, killing states on former IP Address 0.0.0.0. Aug 23 09:25:42 php-fpm 55435 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:42 php-fpm 55435 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:25:40 check_reload_status 503 rc.newwanip starting hn0 Aug 23 09:25:40 php-fpm 93800 NOTICE HOTPLUG: Configuring interface wan Aug 23 09:25:40 php-fpm 93800 NOTICE DEVD Ethernet attached event for wan Aug 23 09:25:40 php-fpm 93800 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:25:39 kernel hn0: link state changed to UP Aug 23 09:25:39 check_reload_status 503 Linkup starting hn0 Aug 23 09:25:39 php-fpm 66209 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:25:39 kernel Limiting ICMPv6 destination unreachable output from 114 to 93 packets/sec Aug 23 09:25:38 php-fpm 59885 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:38 php-fpm 59885 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:25:38 php-fpm 55435 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:38 php-fpm 55435 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' Aug 23 09:25:38 php-fpm 93800 ERROR [phpDynDNS] () Could not determine the request IP address (using "wan", "hn0"): gateway not online Aug 23 09:25:37 kernel Limiting ICMPv6 destination unreachable output from 126 to 92 packets/sec Aug 23 09:25:37 check_reload_status 503 Reloading filter Aug 23 09:25:37 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:25:37 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:25:37 check_reload_status 503 updating dyndns WAN_DHCP Aug 23 09:25:37 check_reload_status 503 Reloading filter Aug 23 09:25:37 check_reload_status 503 Restarting OpenVPN tunnels/interfaces Aug 23 09:25:37 check_reload_status 503 Restarting IPsec tunnels Aug 23 09:25:37 check_reload_status 503 updating dyndns WAN_DHCP6 Aug 23 09:25:37 php-fpm 55435 WARNING Could not check for system updates: DNS servers not available Aug 23 09:25:35 php-fpm 66209 NOTICE DEVD Ethernet detached event for wan Aug 23 09:25:35 php-fpm 66209 NOTICE Hotplug event detected for WAN (wan) dynamic IP address (4: dhcp, 6: dhcp6) Aug 23 09:25:34 kernel hn0: link state changed to DOWN Aug 23 09:25:34 kernel hn0: network changed, change 1 Aug 23 09:25:34 check_reload_status 503 Linkup starting hn0 Aug 23 09:25:30 php-fpm 59885 WARNING Could not check for system updates: DNS servers not available Aug 23 09:25:23 php-fpm 15258 WARNING Could not check for system updates: DNS servers not available Aug 23 09:25:15 php-fpm 66209 WARNING Could not check for system updates: DNS servers not available Aug 23 09:25:11 php-fpm 93800 NOTICE No default gateway available for inet6, use the first one configured. 'WAN_DHCP6' Aug 23 09:25:11 php-fpm 93800 NOTICE No default gateway available for inet, use the first one configured. 'WAN_DHCP' -
This post is deleted! -
Are you actually making config changes before the system has fully come up?
[ . . . ] Aug 23 09:27:26 php-fpm 93800 NOTICE [Config] Configuration Change: admin@10.28.92.20 (Local Database): Interfaces settings changed [ . . . ] Aug 23 09:25:43 php-fpm 92723 NOTICE [Config] Configuration Change: admin@10.28.92.20 (Local Database): Interfaces settings changed [ . . . ] -
@tinfoilmatt No
-
Since I reverted to 2.8.1, everything is perfectly stable.
-
Any idea what those log entries are about then? This is a Hyper-V guest?
-
@tinfoilmatt Yes, it's a hyper-v VM. I've been running pfSense on hyper-v for ages.
I have no idea what's causing the messages. I tried to post a longer log but it was character limited so I kept shortening it until I could post. Looking through the log (2000 entries), the gateways went up and down many times over the past 24 hours.
I updated yesterday morning. It was initially very unstable, but it settled down enough that the gateways were up most of the time. This morning, it was much worse again. I tried to reboot it, but it wouldn't even shutdown, so I killed it and reverted to 2.8.1. I've never experienced a pfSense upgrade that went this badly.
-
You might try again at some point and let it do its thing for longer than you were. I can't imagine what'd be generating those notices except for an actual user config change.
-
@tinfoilmatt The only explanation was that the log captured a save and apply of the WAN to get it to reconnect. The gateways were going up and down for a long time over the past 24 hours. If it didn't fix itself over that long it wasn't going to.
-
Is there a way to post a longer portion of the log? I have almost 24 hours of log messages.
-
Probably not necessary. Log from a cleaner upgrade attempt would be better.
-
@tinfoilmatt The upgrade proceeded without any obvious errors and it didn't work. Why would it be any "cleaner" the second time than it was the first time? I've upgraded pfsense countless times. It's not like I was f*cking with it.
-
You're really only left with trying again or staying on 2.8.1 at this point. Log of the former would be helpful, before any user intervention.
-
What is hn0 connected to there? It is shown as changing link state 15 times in that 5min period. That is probably the root cause here. Though it still shouldn't become unresponsive like that.
-
S stephenw10 moved this topic from Problems Installing or Upgrading pfSense Software
-
@stephenw10 Thank you for your reply.
pfSense is installed on a hyper-v 2022 server. hn0 is the WAN interface and it's connected to a virtual switch, which is connected to an Intel I-350 NIC. (The I-350 is connected to a physical switch, which is connected to the ISP ONT.) My service is gigabit fibre optic and my ISP supports up to 5 DHCP addresses and DHCPv6 prefixes.
This server has been operational since 2023 and has been reliable the entire time. (It replaced a previous server that was operational for years before that.)
There is another instance of pfSense running on this server, connected to the same virtual switch, but it only has a virtual LAN. It was running the 2.9.0 beta version before I upgraded it to 2.9.0 release version without any apparent issues. I had previously updated this instance to 2.9.0 beta from 2.8.1 release when the beta became available. I didn't observe any problems with it, although I barely paid attention to it, since it gave all appearances of working nominally.
There is also an instance of OPNsense running on this server, with a similar configuration as the second pfSense instance. It has also been working nominally.
I've been running this configuration for years and I've tested many beta, release candidates, and release versions of pfSense and OPNsense. I don't recall ever having to back out of an installation under circumstances like this.
(Since I have 2X pfSense and 1X OPNsense, unless stated otherwise, I'm talking about the main instance of pfSense, which is for my LAN. It typically has around 30 connected devices.)
Before I attempted to upgrade to the 2.9.0 release version, the 2.8.1 release version had been running without any issues and it was running the 2.8.0 release version before that.
I didn't see any errors during the upgrade, but it started showing problems immediately after it rebooted. The WAN_DHCP and WAN_DHCP6 gateways were both going up and down repeatedly. The only things I did to stabilize it was to to try Interfaces / WAN / Save / Apply Settings. That didn't work, so I also tried Status / Interface / Release (with Relinquish Lease enabled) / Renew. That also didn't work, so I rebooted it.
After rebooting, it somewhat stabilized so I left it running. I noticed that the gateways both periodically dropped and restored during the day. The next morning, the gateways were more down than up, so I tried to reboot again. It wouldn't even shutdown, so I reverted to the 2.8.1 checkpoint that I saved before upgrading. That worked nominally.
Looking at things more closely today, I noticed in the log that during the time 2.9.0 was running on the main pfSense, the gateways went up and down many times, even during the night when there was little activitity. I noticed in the log that the other pfSense was also having the same problems. With the main pfSense reverted to 2.8.1, the other pfSense is running 2.9.0 nominally. I could not tell from the log of the OPNsense instance was also having problems because the log wrapped.
At this point, there appears to be a problem with pfSense 2.9.0, perhaps with FreeBSD 16, specifically related to running on hyper-v.
I didn't create a checkpoint of the main pfSense running 2.9.0, so I would have to upgrade the 2.8.1 instance to try it again.
I'm open to further testing. Let me know if you have any suggestions or questions.
-
Is there any difference between the two VMs that ere running 2.9.0?
Since hn0 is connected to a virtual switch and not apass-through NIC you really wouldn't ever expect it to lose link. The only thing I'm aware of that can cause that is Snort or Suricata in in-line mode. Are you running either?
-
@stephenw10 Thank you for your reply.
The VMs are identical except for the following:
The main pfSense has VMQ, IPsec offloading and SR-IOV enabled on both the WAN and LAN interfaces. The other does not. (The LAN interface is also an Intel I-350.)
The other pfSense isn't connected to the LAN switch. It's connected to private virtual switch.
The main pfSense has SNORT enabled, as well as a few firewall rules and DHCP reservations. The other pfSense is basically out of the box and it has only one client, a Windows 10 VM.
When I upgraded the main pfSense, SNORT was running. Should I have uninstalled it beforehand?
-
Ah so that does seem like a significant difference. Whatever happened at upgrade wouldn't be ongoing. I doubt it would have made any difference disabling snort there.
But running the NICs with SR-IOV and using Snort may well be causing the re-links. Is snort in in-line mode?
Can you test with Snort in legacy mode or not running at all?
-
@stephenw10 said in Unreliable WAN_DHCP and WAN_DHCP6 Gateways after upgrading to CE 2.9.0:
Ah so that does seem like a significant difference. Whatever happened at upgrade wouldn't be ongoing. I doubt it would have made any difference disabling snort there.
But running the NICs with SR-IOV and using Snort may well be causing the re-links. Is snort in in-line mode?
Can you test with Snort in legacy mode or not running at all?
The only shared NIC is for the WAN switch. Hardware accelerations are only enabled for the main pfSense, not any other pfSense or OPNsense. Everything was working properly until I upgraded the main pfSense.
One thing I could try is shutting down the other pfSense and OPNsense, then upgrading the main pfSense.
How do I tell what mode Snort is running in? Other than enabling the offered rule sets, all of the settings are default.
If it would help, I could remove snort and try again.
-
It's a setting on each interface it's running on if you have enabled blocking mode:
In In-line mode when Snort restarts the link will bounce because it intercepts all traffic.
Privacy Policy · Cookie Policy