Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.5.1-RC Routing Issue on Reboot

    Scheduled Pinned Locked Moved
    21.02.2/2.5.1 Snapshots (Retired)
    1
    6
    743
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      Griffo
      last edited by Griffo

      EDIT: I think this is actually a routing issue. Please see later posts.

      Upgraded from a working 2.5 instance to 2.5.1-RC.

      After the upgrade, my OpenVPN "client" gateway shows as down. The VPN status is Up.

      The manually configured IPv4 monitor IP is the VPN providers DNS server.
      Manually pinging the IP from the interface works.
      Packet Capture show my manual ping attempts.

      Packet capture shows NO traffic being generated by the gateway monitor.

      I've not attempted any changes in config to resolve it in case you want to see some specific post upgrade configs.

      G 1 Reply Last reply Reply Quote 0
      • G
        Griffo @Griffo
        last edited by Griffo

        OK that is so weird. It fixed itself.

        Looking in the logs shows that for some reason at 09:58 (about an hour after upgrade) it suddenly realised that it was trying to route that monitor down the wrong interface:

        It's the 149.112.112.112 address @ Mar 18 09:58:19

        Mar 18 09:01:35	php	17808	[pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Mar 18 09:58:04	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.IPv6 with address 2403:5800:7600:b00:4262:31ff:fe14:862f.
Mar 18 09:58:04	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.IPv6 with address fe80::1:1.
Mar 18 09:58:04	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.30.IPv6 with address 2403:5800:7600:b30:4262:31ff:fe14:862f.
Mar 18 09:58:04	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.30.IPv6 with address fe80::1:1.
Mar 18 09:58:04	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.66.IPv6 with address 2403:5800:7600:b66:4262:31ff:fe14:862f.
Mar 18 09:58:04	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.66.IPv6 with address fe80::1:1.
Mar 18 09:58:05	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.IPv6 with address fe80::1:1.
Mar 18 09:58:05	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.IPv6 with address 2403:5800:7600:b00:4262:31ff:fe14:862f.
Mar 18 09:58:05	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.30.IPv6 with address fe80::1:1.
Mar 18 09:58:05	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.30.IPv6 with address 2403:5800:7600:b30:4262:31ff:fe14:862f.
Mar 18 09:58:05	avahi-daemon	65923	Leaving mDNS multicast group on interface igb1.66.IPv6 with address fe80::1:1.
Mar 18 09:58:05	avahi-daemon	65923	Joining mDNS multicast group on interface igb1.66.IPv6 with address 2403:5800:7600:b66:4262:31ff:fe14:862f.
Mar 18 09:58:05	php-fpm	684	/rc.newwanipv6: rc.newwanipv6: Info: starting on igb0.
Mar 18 09:58:05	php-fpm	684	/rc.newwanipv6: rc.newwanipv6: on (IP address: 2403:5800:7001:b:f4f7:9e2e:66f2:31ef) (interface: wan) (real interface: igb0).
Mar 18 09:58:07	dhcpleases	11960	Could not deliver signal HUP to process 83895: No such process.
Mar 18 09:58:12	dhcpleases	14161	Could not deliver signal HUP to process 13739: No such process.
Mar 18 09:58:17	dhcpleases	5908	Could not deliver signal HUP to process 15742: No such process.
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: Removing static route for monitor 2403:5800:100:1::142 and adding a new route through dynamic
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: route_add_or_change: Invalid gateway dynamic and/or network interface
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: Removing static route for monitor 149.112.112.112 and adding a new route through 10.8.0.5
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: Removing static route for monitor 9.9.9.9 and adding a new route through 10.70.216.246
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: Removing static route for monitor 1.0.0.1 and adding a new route through 10.5.0.2
Mar 18 09:58:19	php-fpm	684	/rc.newwanipv6: Removing static route for monitor 1.1.1.1 and adding a new route through 10.13.118.217
Mar 18 09:58:20	check_reload_status	723	Reloading filter
        G 1 Reply Last reply Reply Quote 0
        • G
          Griffo @Griffo
          last edited by Griffo

          Something kicked off DHCP at that time which restarted multiple times which triggered the routing fix.

          Mar 18 09:53:41	dhcpd	22460	DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:53:41	dhcpd	22460	DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:56:37	dhcp6c	1883	Sending Rebind
Mar 18 09:58:04	dhcp6c	1883	remove an address 2403:5800:7600:b00:4262:31ff:fe14:862f/64 on igb1
Mar 18 09:58:04	dhcp6c	1883	remove an address 2403:5800:7600:b30:4262:31ff:fe14:862f/64 on igb1.30
Mar 18 09:58:04	dhcp6c	1883	remove an address 2403:5800:7600:b66:4262:31ff:fe14:862f/64 on igb1.66
Mar 18 09:58:04	dhcp6c	1883	Sending Renew
Mar 18 09:58:04	dhcp6c	1883	dhcp6c Received INFO
Mar 18 09:58:04	dhcp6c	1883	add an address 2403:5800:7001:b:f4f7:9e2e:66f2:31ef/128 on igb0
Mar 18 09:58:04	dhcp6c	1883	Sending Solicit
Mar 18 09:58:05	dhcp6c	1883	Sending Request
Mar 18 09:58:05	dhcp6c	1883	dhcp6c Received REQUEST
Mar 18 09:58:05	dhcp6c	1883	add an address 2403:5800:7600:b00:4262:31ff:fe14:862f/64 on igb1
Mar 18 09:58:05	dhcp6c	1883	add an address 2403:5800:7600:b30:4262:31ff:fe14:862f/64 on igb1.30
Mar 18 09:58:05	dhcp6c	1883	add an address 2403:5800:7600:b66:4262:31ff:fe14:862f/64 on igb1.66
Mar 18 09:58:07	dhcpleases	11960	Sending HUP signal to dns daemon(83895)
Mar 18 09:58:07	dhcpleases	11960	Could not deliver signal HUP to process 83895: No such process.
Mar 18 09:58:09	dhcpd	21316	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:09	dhcpd	21316	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:09	dhcpd	21316	All rights reserved.
Mar 18 09:58:09	dhcpd	21316	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:09	dhcpd	21316	Config file: /etc/dhcpdv6.conf
Mar 18 09:58:09	dhcpd	21316	Database file: /var/db/dhcpd6.leases
Mar 18 09:58:09	dhcpd	21316	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:09	dhcpd	21316	PID file: /var/run/dhcpdv6.pid
Mar 18 09:58:09	dhcpd	21316	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:09	dhcpd	21316	All rights reserved.
Mar 18 09:58:09	dhcpd	21316	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:09	dhcpd	21316	Wrote 7 NA, 0 TA, 0 PD leases to lease file.
Mar 18 09:58:09	dhcpd	21316	Bound to *:547
Mar 18 09:58:09	dhcpd	21316	Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:09	dhcpd	21316	Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:09	dhcpd	21316	Listening on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:09	dhcpd	21316	Sending on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:09	dhcpd	21316	Server starting service.
Mar 18 09:58:12	dhcpleases	14161	Sending HUP signal to dns daemon(13739)
Mar 18 09:58:12	dhcpleases	14161	Could not deliver signal HUP to process 13739: No such process.
Mar 18 09:58:14	dhcpd	21481	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:14	dhcpd	21481	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:14	dhcpd	21481	All rights reserved.
Mar 18 09:58:14	dhcpd	21481	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:14	dhcpd	21481	Config file: /etc/dhcpdv6.conf
Mar 18 09:58:14	dhcpd	21481	Database file: /var/db/dhcpd6.leases
Mar 18 09:58:14	dhcpd	21481	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:14	dhcpd	21481	PID file: /var/run/dhcpdv6.pid
Mar 18 09:58:14	dhcpd	21481	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:14	dhcpd	21481	All rights reserved.
Mar 18 09:58:14	dhcpd	21481	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:14	dhcpd	21481	Wrote 7 NA, 0 TA, 0 PD leases to lease file.
Mar 18 09:58:14	dhcpd	21481	Bound to *:547
Mar 18 09:58:14	dhcpd	21481	Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:14	dhcpd	21481	Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:14	dhcpd	21481	Listening on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:14	dhcpd	21481	Sending on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:14	dhcpd	21481	Server starting service.
Mar 18 09:58:17	dhcpleases	5908	Sending HUP signal to dns daemon(15742)
Mar 18 09:58:17	dhcpleases	5908	Could not deliver signal HUP to process 15742: No such process.
Mar 18 09:58:18	dhcpd	13560	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:18	dhcpd	13560	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:18	dhcpd	13560	All rights reserved.
Mar 18 09:58:18	dhcpd	13560	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:18	dhcpd	13560	Config file: /etc/dhcpdv6.conf
Mar 18 09:58:18	dhcpd	13560	Database file: /var/db/dhcpd6.leases
Mar 18 09:58:18	dhcpd	13560	Internet Systems Consortium DHCP Server 4.4.2
Mar 18 09:58:18	dhcpd	13560	PID file: /var/run/dhcpdv6.pid
Mar 18 09:58:18	dhcpd	13560	Copyright 2004-2020 Internet Systems Consortium.
Mar 18 09:58:18	dhcpd	13560	All rights reserved.
Mar 18 09:58:18	dhcpd	13560	For info, please visit https://www.isc.org/software/dhcp/
Mar 18 09:58:18	dhcpd	13560	Wrote 7 NA, 0 TA, 0 PD leases to lease file.
Mar 18 09:58:18	dhcpd	13560	Bound to *:547
Mar 18 09:58:18	dhcpd	13560	Listening on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:18	dhcpd	13560	Sending on Socket/6/igb1.66/2403:5800:7600:b66::/64
Mar 18 09:58:18	dhcpd	13560	Listening on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:18	dhcpd	13560	Sending on Socket/6/igb1/2403:5800:7600:b00::/64
Mar 18 09:58:18	dhcpd	13560	Server starting service.
Mar 18 09:59:36	dhcpd	22460	reuse_lease: lease age 14712 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188
Mar 18 09:59:36	dhcpd	22460	DHCPDISCOVER from 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:36	dhcpd	22460	DHCPOFFER on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:36	dhcpd	22460	reuse_lease: lease age 14712 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188
Mar 18 09:59:36	dhcpd	22460	DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:36	dhcpd	22460	DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:53	dhcpd	22460	reuse_lease: lease age 14729 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188
Mar 18 09:59:53	dhcpd	22460	DHCPDISCOVER from 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:53	dhcpd	22460	DHCPOFFER on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:53	dhcpd	22460	reuse_lease: lease age 14729 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.1.188
Mar 18 09:59:53	dhcpd	22460	DHCPREQUEST for 192.168.1.188 (192.168.1.1) from 10:4f:a8:d4:6d:fd via igb1
Mar 18 09:59:53	dhcpd	22460	DHCPACK on 192.168.1.188 to 10:4f:a8:d4:6d:fd via igb1
Mar 18 10:00:06	dhcpd	22460	reuse_lease: lease age 14742 (secs) under 25% threshold, reply w
          G 1 Reply Last reply Reply Quote 0
          • G
            Griffo @Griffo
            last edited by Griffo

            @griffo OK so this morning I upgrade to the latest build.

            On reboot, the same behaviour occurred, this time multiple gateways were showing down and routing was all screwed up. It was trying to route all my traffic down a WireGuard tunnel that was showing offline.

            I do policy based routing for certain vlans, not the default LAN. My default gateways are set to WAN_DHCP and WAN_DHCP6. You can see in the route table that's not what is reflected.

            As a note, when I did a release on the WAN interface in order to fix, it dropped my SSH sessions and the web-page timed out. After a minute I was able to reconnect.

            Routes-on-reboot.txt
            Routes-after-release-renew.txt
            syslog.txt

            G 1 Reply Last reply Reply Quote 0
            • G
              Griffo @Griffo
              last edited by

              @griffo This continues to be an issue on every reboot.
              Routing is up the wazoo. Release / Renew the WAN, and it sorts itself out.

              G 1 Reply Last reply Reply Quote 0
              • G
                Griffo @Griffo
                last edited by

                Problem went away with the removal of Wireguard.

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post