VTI gateways not adding static routes in 24.03

LarryFahnoe

@stephenw10 After quite a bit more testing, I have narrowed the missing static route problem down to the non-dynamic <gateway_item> shown above. The real puzzler is that rolling back to 23.09.1 (BE right before the upgrade), I only have the two dynamic <gateway_items>. <staticroutes> are the same.

23.09.1 pre-upgrade:

        <staticroutes>
                <route>
                        <network>192.168.3.0/24</network>
                        <gateway>MPLS_ALEX_VTIV4</gateway>
                        <descr><![CDATA[Alex LAN]]></descr>
                </route>
        </staticroutes>

        <gateways>
                <gateway_item>
                        <interface>opt3</interface>
                        <gateway>dynamic</gateway>
                        <name>MPLS_ALEX_VTIV4</name>
                        <weight>1</weight>
                        <ipprotocol>inet</ipprotocol>
                        <descr><![CDATA[Interface MPLS_ALEX_VTIV4 Gateway]]></descr>
                        <monitor_disable></monitor_disable>
                        <gw_down_kill_states></gw_down_kill_states>
                </gateway_item>
                <gateway_item>
                        <interface>wan</interface>
                        <gateway>dynamic</gateway>
                        <name>WAN_DHCP</name>
                        <weight>1</weight>
                        <ipprotocol>inet</ipprotocol>
                        <interval>1000</interval>
                        <descr><![CDATA[Via Quantum Fiber C5500XK]]></descr>
                        <gw_down_kill_states></gw_down_kill_states>
                </gateway_item>
                <defaultgw4>WAN_DHCP</defaultgw4>
                <defaultgw6></defaultgw6>
        </gateways>

To clean up the errant <gateway_item> required tearing down and rebuilding much of the config:

Delete static route to 192.168.3.0/24 via MPLS_ALEX_VTIV4
Delete MPLS_ALEX_VTIV4 interface assignment
Disable IPsec P1 and P2
Delete gateway MPLS_ALEX_VTIV4
[ Gateway was grayed out (Gateway inactive, interface is missing) before attempting to delete and remains in this state after attempting to delete ]
Delete the IPsec P2
Delete the gateway MPLS_ALEX_VTIV4
[ Gateway is deleted and deleted from config.xml ]
Recreate IPsec P2
Enable IPsec P1 and P2
Add interface ipsec1
Enable interface OPT3 (skip renaming to MPLS_ALEX_VTIV4)
OPT3_VTIV4 gateway is created automatically
Add static route to 192.168.3.0/24 via OPT3_VTIV4
Add the OPT3 rules for site to site traffic and gateway monitoring
Reboot

Did this on both of my systems and they are both rebooting cleanly with the IPsec VTI coming up and passing traffic immediately.

I will add this update to the redmine, but it still does not explain where the non-dynamic <gateway_item> came from, and I'm not sure it addresses the problem that @OhYeah-0 is seeing.

--Larry

stephenw10

Nice troubleshooting.

Hmm, so the bug is potentially that an additional gateway is created at upgrade.

I'll try to find any other instances. I'd expect to see quite a few if so.

LarryFahnoe

@stephenw10 said in VTI gateways in 24.03:

Hmm, so the bug is potentially that an additional gateway is created at upgrade.

I'm quite curious now as to the root cause & will look forward to hearing if you uncover more. Will also be interesting to see what @OhYeah-0 finds.

--Larry

OhYeah 0

@LarryFahnoe said in VTI gateways in 24.03:

Will also be interesting to see what @OhYeah-0 finds.

Well this doesn't move us closer to a solution.. I have only 2 gateways defined in the config file.

<gateways>
		<gateway_item>
			<interface>wan</interface>
			<gateway>xxx.xx.xxx.xx</gateway>
			<name>WANGW</name>
			<weight>1</weight>
			<descr><![CDATA[WAN Gateway]]></descr>
			<defaultgw></defaultgw>
		</gateway_item>
		<gateway_item>
			<interface>opt5</interface>
			<gateway>dynamic</gateway>
			<name>IPSEC_SWE_GW</name>
			<weight>1</weight>
			<ipprotocol>inet</ipprotocol>
			<descr><![CDATA[Test description]]></descr>
			<monitor_disable></monitor_disable>
			<action_disable></action_disable>
			<gw_down_kill_states></gw_down_kill_states>
		</gateway_item>

The problem is that I cannot remember if I performed the upgrade before I created the IPSEC tunnel or not.

OhYeah 0

Tried something a bit more drastic.

Deleted everything: static routes, gateway, disabled interface, deleted assignment, deleted P2, deleted P1.
Restart.
Switch global states back to "floating" and IPSEC filter mode back to "on IPSEC tab".
Restart.
Add everything back in the same order as standard (but different names just to make sure something doesn't clash with cached or old entries).
Restart.

Same status. P1 comes up, routes are not added to the routing table.

LarryFahnoe

@OhYeah-0 And when you rebuilt, you did so with 0.0.0.0/0 correct? The rationale for that was a mixed environment if I understood.

Would it be possible to do a test using a private /30 transit network?

--Larry

OhYeah 0

@LarryFahnoe said in VTI gateways in 24.03:

And when you rebuilt, you did so with 0.0.0.0/0 correct? The rationale for that was a mixed environment if I understood.

Yep. With that client we have a hub-and-spoke topology with different vendor platforms (also mix of virtual and physical instances). The solution had been working flawlessly until the 24.03 update.

stephenw10

Ok so some of the connected clients are using policy mode and require a P2 that carries all destination traffic?

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

Ok so some of the connected clients are using policy mode and require a P2 that carries all destination traffic?

All endpoints are connected via the same method (0.0.0.0/0 local/remote and static routes).

I know that while it's possible to mix policy and route based IPSEC; it's really not a good idea. You lose all the benefits and there's another source of potential problems.

stephenw10

Right, I agree with that. So these are all route mode devices the tunnels are connected to? In which case why are you using 0/0 for the P2s?

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

So these are all route mode devices the tunnels are connected to? In which case why are you using 0/0 for the P2s?

Yes, all the spokes are connected to the hub via 0/0. Except for end-user remote access VPN which is a separate virtual network and then routed to the hub via parent router LAN/IPSEC (Fortinet because it offers 365/Entra integration).

As to why use 0/0 for P2s... tried it out with pfsense and a couple of ISPs/partners and found out it works incredibly well across multiple platforms.

If that mode of VPN setup is suddenly not supported anymore, I would like to hear the reasoning behind this change. At the moment it sounds more like a bug. :)

stephenw10

Hmm, curious. The only time I've ever seen that is when one side of the tunnel is using policy mode. Otherwise having a local interface defined as 0/0 could potentially break routing entirely.

However I'm not aware of any specific change in 24.03 that would prevent it if it worked in 23.09. It's unlikely a setup like that was ever tested though. Let me see what I can find....

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

However I'm not aware of any specific change in 24.03 that would prevent it if it worked in 23.09. It's unlikely a setup like that was ever tested though.

I can provide also some logs/data from routers that are running 23.09, if it would help to figure out what actually changed.

Nikkeli

I'm also having problems with static routes not being loaded on boot.
However they get loaded after editing and saving routes (without changes), after which the tunnel works as intended.

I have IPsec VTI with local/remote networks set to "address".
Issue appeared after upgrade from 23.09.1 with no changes to configuration between upgrades.

I can post more information if needed.

OhYeah 0

Maybe it's also a good idea to change the title of the topic to include the phrase "static routes"?

LarryFahnoe

@Nikkeli Your situation sounds a lot like mine.

Might be interesting to take a peek at your /cf/conf/config.xml and compare it to what I showed above in https://forum.netgate.com/post/1170175

Do you have a spurious <gateway_item> with a <gateway> containing an address rather than "dynamic"?

I have on my "spare time list" (ahem!) to roll back to 23.09.1, then do the upgrade again and document how the config changes. I suspect there is a bug in the upgrade process.

@stephenw10 I'd vote for adding "static routes" to the title of this thread if possible.

--Larry

Nikkeli

@LarryFahnoe
I actually don't have this problem, the configuration seems fine. Below is the configuration for the only (vti) gateway listed.

<gateway_item>
<interface>opt10</interface>
<gateway></gateway>
<name>IPSEC_VT13_VT10_VTIV4</name>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<descr><![CDATA[Interface IPSEC_VT13_VT10_VTIV4 Gateway]]></descr>
<gw_down_kill_states></gw_down_kill_states>
</gateway_item>

stephenw10

So no additional gateways? No disabled gateways?

Nikkeli

@stephenw10
The only other gateway is WAN gateway. No gateways are disabled.

stephenw10

Hmm, any errors in the routing or system logs at boot?