2.5.1-RC Routing Issue on Reboot
-
EDIT: I think this is actually a routing issue. Please see later posts.
Upgraded from a working 2.5 instance to 2.5.1-RC.
After the upgrade, my OpenVPN "client" gateway shows as down. The VPN status is Up.
The manually configured IPv4 monitor IP is the VPN providers DNS server.
Manually pinging the IP from the interface works.
Packet Capture show my manual ping attempts.Packet capture shows NO traffic being generated by the gateway monitor.
I've not attempted any changes in config to resolve it in case you want to see some specific post upgrade configs.
-
OK that is so weird. It fixed itself.
Looking in the logs shows that for some reason at 09:58 (about an hour after upgrade) it suddenly realised that it was trying to route that monitor down the wrong interface:
It's the 149.112.112.112 address @ Mar 18 09:58:19
Mar 18 09:01:35 php 17808 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Mar 18 09:58:04 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.IPv6 with address 2403:5800:7600:b00:4262:31ff:fe14:862f. Mar 18 09:58:04 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.IPv6 with address fe80::1:1. Mar 18 09:58:04 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.30.IPv6 with address 2403:5800:7600:b30:4262:31ff:fe14:862f. Mar 18 09:58:04 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.30.IPv6 with address fe80::1:1. Mar 18 09:58:04 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.66.IPv6 with address 2403:5800:7600:b66:4262:31ff:fe14:862f. Mar 18 09:58:04 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.66.IPv6 with address fe80::1:1. Mar 18 09:58:05 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.IPv6 with address fe80::1:1. Mar 18 09:58:05 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.IPv6 with address 2403:5800:7600:b00:4262:31ff:fe14:862f. Mar 18 09:58:05 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.30.IPv6 with address fe80::1:1. Mar 18 09:58:05 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.30.IPv6 with address 2403:5800:7600:b30:4262:31ff:fe14:862f. Mar 18 09:58:05 avahi-daemon 65923 Leaving mDNS multicast group on interface igb1.66.IPv6 with address fe80::1:1. Mar 18 09:58:05 avahi-daemon 65923 Joining mDNS multicast group on interface igb1.66.IPv6 with address 2403:5800:7600:b66:4262:31ff:fe14:862f. Mar 18 09:58:05 php-fpm 684 /rc.newwanipv6: rc.newwanipv6: Info: starting on igb0. Mar 18 09:58:05 php-fpm 684 /rc.newwanipv6: rc.newwanipv6: on (IP address: 2403:5800:7001:b:f4f7:9e2e:66f2:31ef) (interface: wan) (real interface: igb0). Mar 18 09:58:07 dhcpleases 11960 Could not deliver signal HUP to process 83895: No such process. Mar 18 09:58:12 dhcpleases 14161 Could not deliver signal HUP to process 13739: No such process. Mar 18 09:58:17 dhcpleases 5908 Could not deliver signal HUP to process 15742: No such process. Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: Removing static route for monitor 2403:5800:100:1::142 and adding a new route through dynamic Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: route_add_or_change: Invalid gateway dynamic and/or network interface Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: Removing static route for monitor 149.112.112.112 and adding a new route through 10.8.0.5 Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: Removing static route for monitor 9.9.9.9 and adding a new route through 10.70.216.246 Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: Removing static route for monitor 1.0.0.1 and adding a new route through 10.5.0.2 Mar 18 09:58:19 php-fpm 684 /rc.newwanipv6: Removing static route for monitor 1.1.1.1 and adding a new route through 10.13.118.217 Mar 18 09:58:20 check_reload_status 723 Reloading filter -
Something kicked off DHCP at that time which restarted multiple times which triggered the routing fix.
Mar 18 09:53:41 dhcpd 22460 DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:53:41 dhcpd 22460 DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:56:37 dhcp6c 1883 Sending Rebind Mar 18 09:58:04 dhcp6c 1883 remove an address 2403:5800:7600:b00:4262:31ff:fe14:862f/64 on igb1 Mar 18 09:58:04 dhcp6c 1883 remove an address 2403:5800:7600:b30:4262:31ff:fe14:862f/64 on igb1.30 Mar 18 09:58:04 dhcp6c 1883 remove an address 2403:5800:7600:b66:4262:31ff:fe14:862f/64 on igb1.66 Mar 18 09:58:04 dhcp6c 1883 Sending Renew Mar 18 09:58:04 dhcp6c 1883 dhcp6c Received INFO Mar 18 09:58:04 dhcp6c 1883 add an address 2403:5800:7001:b:f4f7:9e2e:66f2:31ef/128 on igb0 Mar 18 09:58:04 dhcp6c 1883 Sending Solicit Mar 18 09:58:05 dhcp6c 1883 Sending Request Mar 18 09:58:05 dhcp6c 1883 dhcp6c Received REQUEST Mar 18 09:58:05 dhcp6c 1883 add an address 2403:5800:7600:b00:4262:31ff:fe14:862f/64 on igb1 Mar 18 09:58:05 dhcp6c 1883 add an address 2403:5800:7600:b30:4262:31ff:fe14:862f/64 on igb1.30 Mar 18 09:58:05 dhcp6c 1883 add an address 2403:5800:7600:b66:4262:31ff:fe14:862f/64 on igb1.66 Mar 18 09:58:07 dhcpleases 11960 Sending HUP signal to dns daemon(83895) Mar 18 09:58:07 dhcpleases 11960 Could not deliver signal HUP to process 83895: No such process. Mar 18 09:58:09 dhcpd 21316 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:09 dhcpd 21316 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:09 dhcpd 21316 All rights reserved. Mar 18 09:58:09 dhcpd 21316 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:09 dhcpd 21316 Config file: /etc/dhcpdv6.conf Mar 18 09:58:09 dhcpd 21316 Database file: /var/db/dhcpd6.leases Mar 18 09:58:09 dhcpd 21316 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:09 dhcpd 21316 PID file: /var/run/dhcpdv6.pid Mar 18 09:58:09 dhcpd 21316 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:09 dhcpd 21316 All rights reserved. Mar 18 09:58:09 dhcpd 21316 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:09 dhcpd 21316 Wrote 7 NA, 0 TA, 0 PD leases to lease file. Mar 18 09:58:09 dhcpd 21316 Bound to *:547 Mar 18 09:58:09 dhcpd 21316 Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:09 dhcpd 21316 Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:09 dhcpd 21316 Listening on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:09 dhcpd 21316 Sending on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:09 dhcpd 21316 Server starting service. Mar 18 09:58:12 dhcpleases 14161 Sending HUP signal to dns daemon(13739) Mar 18 09:58:12 dhcpleases 14161 Could not deliver signal HUP to process 13739: No such process. Mar 18 09:58:14 dhcpd 21481 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:14 dhcpd 21481 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:14 dhcpd 21481 All rights reserved. Mar 18 09:58:14 dhcpd 21481 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:14 dhcpd 21481 Config file: /etc/dhcpdv6.conf Mar 18 09:58:14 dhcpd 21481 Database file: /var/db/dhcpd6.leases Mar 18 09:58:14 dhcpd 21481 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:14 dhcpd 21481 PID file: /var/run/dhcpdv6.pid Mar 18 09:58:14 dhcpd 21481 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:14 dhcpd 21481 All rights reserved. Mar 18 09:58:14 dhcpd 21481 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:14 dhcpd 21481 Wrote 7 NA, 0 TA, 0 PD leases to lease file. Mar 18 09:58:14 dhcpd 21481 Bound to *:547 Mar 18 09:58:14 dhcpd 21481 Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:14 dhcpd 21481 Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:14 dhcpd 21481 Listening on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:14 dhcpd 21481 Sending on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:14 dhcpd 21481 Server starting service. Mar 18 09:58:17 dhcpleases 5908 Sending HUP signal to dns daemon(15742) Mar 18 09:58:17 dhcpleases 5908 Could not deliver signal HUP to process 15742: No such process. Mar 18 09:58:18 dhcpd 13560 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:18 dhcpd 13560 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:18 dhcpd 13560 All rights reserved. Mar 18 09:58:18 dhcpd 13560 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:18 dhcpd 13560 Config file: /etc/dhcpdv6.conf Mar 18 09:58:18 dhcpd 13560 Database file: /var/db/dhcpd6.leases Mar 18 09:58:18 dhcpd 13560 Internet Systems Consortium DHCP Server 4.4.2 Mar 18 09:58:18 dhcpd 13560 PID file: /var/run/dhcpdv6.pid Mar 18 09:58:18 dhcpd 13560 Copyright 2004-2020 Internet Systems Consortium. Mar 18 09:58:18 dhcpd 13560 All rights reserved. Mar 18 09:58:18 dhcpd 13560 For info, please visit https://www.isc.org/software/dhcp/ Mar 18 09:58:18 dhcpd 13560 Wrote 7 NA, 0 TA, 0 PD leases to lease file. Mar 18 09:58:18 dhcpd 13560 Bound to *:547 Mar 18 09:58:18 dhcpd 13560 Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:18 dhcpd 13560 Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64 Mar 18 09:58:18 dhcpd 13560 Listening on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:18 dhcpd 13560 Sending on Socket/6/igb1/2403:5800:7600:b00::/64 Mar 18 09:58:18 dhcpd 13560 Server starting service. Mar 18 09:59:36 dhcpd 22460 reuse_lease: lease age 14712 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188 Mar 18 09:59:36 dhcpd 22460 DHCPDISCOVER from 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:36 dhcpd 22460 DHCPOFFER on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:36 dhcpd 22460 reuse_lease: lease age 14712 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188 Mar 18 09:59:36 dhcpd 22460 DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:36 dhcpd 22460 DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:53 dhcpd 22460 reuse_lease: lease age 14729 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188 Mar 18 09:59:53 dhcpd 22460 DHCPDISCOVER from 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:53 dhcpd 22460 DHCPOFFER on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:53 dhcpd 22460 reuse_lease: lease age 14729 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188 Mar 18 09:59:53 dhcpd 22460 DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1 Mar 18 09:59:53 dhcpd 22460 DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1 Mar 18 10:00:06 dhcpd 22460 reuse_lease: lease age 14742 (secs) under 25% threshold, reply w -
@griffo OK so this morning I upgrade to the latest build.
On reboot, the same behaviour occurred, this time multiple gateways were showing down and routing was all screwed up. It was trying to route all my traffic down a WireGuard tunnel that was showing offline.
I do policy based routing for certain vlans, not the default LAN. My default gateways are set to WAN_DHCP and WAN_DHCP6. You can see in the route table that's not what is reflected.
As a note, when I did a release on the WAN interface in order to fix, it dropped my SSH sessions and the web-page timed out. After a minute I was able to reconnect.
Routes-on-reboot.txt
Routes-after-release-renew.txt
syslog.txt -
@griffo This continues to be an issue on every reboot.
Routing is up the wazoo. Release / Renew the WAN, and it sorts itself out. -
Problem went away with the removal of Wireguard.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.