VTI gateways not adding static routes in 24.03

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

So these are all route mode devices the tunnels are connected to? In which case why are you using 0/0 for the P2s?

Yes, all the spokes are connected to the hub via 0/0. Except for end-user remote access VPN which is a separate virtual network and then routed to the hub via parent router LAN/IPSEC (Fortinet because it offers 365/Entra integration).

As to why use 0/0 for P2s... tried it out with pfsense and a couple of ISPs/partners and found out it works incredibly well across multiple platforms.

If that mode of VPN setup is suddenly not supported anymore, I would like to hear the reasoning behind this change. At the moment it sounds more like a bug. :)

[stephenw10]

Hmm, curious. The only time I've ever seen that is when one side of the tunnel is using policy mode. Otherwise having a local interface defined as 0/0 could potentially break routing entirely.

However I'm not aware of any specific change in 24.03 that would prevent it if it worked in 23.09. It's unlikely a setup like that was ever tested though. Let me see what I can find....

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

However I'm not aware of any specific change in 24.03 that would prevent it if it worked in 23.09. It's unlikely a setup like that was ever tested though.

I can provide also some logs/data from routers that are running 23.09, if it would help to figure out what actually changed.

[Nikkeli]

I'm also having problems with static routes not being loaded on boot.
However they get loaded after editing and saving routes (without changes), after which the tunnel works as intended.

I have IPsec VTI with local/remote networks set to "address".
Issue appeared after upgrade from 23.09.1 with no changes to configuration between upgrades.

I can post more information if needed.

[OhYeah 0]

Maybe it's also a good idea to change the title of the topic to include the phrase "static routes"?

[LarryFahnoe]

@Nikkeli Your situation sounds a lot like mine.

Might be interesting to take a peek at your /cf/conf/config.xml and compare it to what I showed above in https://forum.netgate.com/post/1170175

Do you have a spurious <gateway_item> with a <gateway> containing an address rather than "dynamic"?

I have on my "spare time list" (ahem!) to roll back to 23.09.1, then do the upgrade again and document how the config changes. I suspect there is a bug in the upgrade process.

@stephenw10 I'd vote for adding "static routes" to the title of this thread if possible.

--Larry

[Nikkeli]

@LarryFahnoe
I actually don't have this problem, the configuration seems fine. Below is the configuration for the only (vti) gateway listed.

<gateway_item>
<interface>opt10</interface>
<gateway></gateway>
<name>IPSEC_VT13_VT10_VTIV4</name>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<descr><![CDATA[Interface IPSEC_VT13_VT10_VTIV4 Gateway]]></descr>
<gw_down_kill_states></gw_down_kill_states>
</gateway_item>

[stephenw10]

So no additional gateways? No disabled gateways?

[Nikkeli]

@stephenw10
The only other gateway is WAN gateway. No gateways are disabled.

[stephenw10]

Hmm, any errors in the routing or system logs at boot?

[Nikkeli]

@stephenw10
On System/General I can actually see some errors/warnings that seem to be relevant. On other logs I could not find anything relevant.
IPsec logging has too much log noise but I can turn that down aswell and reboot, if you think it could help.

Here is System/General logging after booting, with the relevant lines.

May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: Gateway, NONE AVAILABLE
May 24 10:11:27 	syslogd 		kernel boot file is /boot/kernel/kernel
May 24 10:11:27 	syslogd 		exiting on signal 15
May 24 10:11:26 	kernel 		done.
May 24 10:11:26 	php-cgi 	685 	rc.bootup: Creating rrd update script
May 24 10:11:24 	kernel 		.done.
May 24 10:11:24 	check_reload_status 	650 	Restarting IPsec tunnels
May 24 10:11:24 	kernel 		...
May 24 10:11:15 	kernel 		done.
May 24 10:11:15 	check_reload_status 	650 	Updating all dyndns
May 24 10:11:14 	kernel 		done.
May 24 10:11:14 	php-cgi 	685 	rc.bootup: NTPD is starting up.
May 24 10:11:08 	kernel 		done.
May 24 10:11:08 	kernel 		done.
May 24 10:11:08 	php-cgi 	685 	rc.bootup: sync unbound done.
May 24 10:11:07 	kernel 		done.
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: Gateway, NONE AVAILABLE
May 24 10:11:07 	php-cgi 	685 	rc.bootup: Default gateway setting as default.

[OhYeah 0]

Rebooted device, went through the logs to see if I catch something that might be relevant (Netgate 4100).

May 24 13:53:46	php-cgi	678	rc.bootup: The command '/sbin/ifconfig 'ipsec1' inet '0.0.0.0/0' '0.0.0.0'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): Destination address required'
May 24 13:53:46	php-cgi	678	rc.bootup: Gateway, NONE AVAILABLE
May 24 13:53:46	php-cgi	678	rc.bootup: Gateway, NONE AVAILABLE
May 24 13:53:46	kernel		route: message indicates error: Invalid argument

[stephenw10]

Ah, there we go. Yup that's pretty much what I'd expect when trying to use 0/0. It tries to apply it to the interfaces and fails because it's invalid there.

The interesting thing is how that ever worked in 23.09.

[OhYeah 0]

And these are similar messages from a Netgate 4100 running 23.09:

May 24 19:26:59	php-cgi	466	rc.bootup: The command '/sbin/ifconfig 'ipsec2' inet '0.0.0.0/0' '0.0.0.0/0'' returned exit code '1', the output was 'ifconfig: 0.0.0.0/0: bad value'
May 24 19:26:59	php-cgi	466	rc.bootup: Gateway, NONE AVAILABLE

The message is very slightly different, so I assume it must be meaningful in some way.

I also got offered 24.03_1 on the same device but no release notes yet?

[stephenw10]

Hmm, interesting. Presumably you don't see the route errors in 23.09?:

May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1

The patch 1 update is a no-op for amd64 devices. It applies only to aarch64. It won't change anything here.

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

Hmm, interesting. Presumably you don't see the route errors in 23.09?

Nope, didn't see any..

[OhYeah 0]

This post is deleted!

[OhYeah 0]

Any news from devs regarding this issue? Well actually two issues I guess.

[stephenw10]

Not yet. In all honesty it's pretty low priority because VTI / Static routes are working as intended in 24.03. Using 0/0 for both ends of the tunnel subnet was never a supported setup.

It is curious that is changed though.

The issue with disabled gateways causing a problem is a bigger issue since that happens in the expected config. Updates there should be shown on the bug report: https://redmine.pfsense.org/issues/15449

[OhYeah 0]

@stephenw10 said in VTI gateways not adding static routes in 24.03:

In all honesty it's pretty low priority because VTI / Static routes are working as intended in 24.03. Using 0/0 for both ends of the tunnel subnet was never a supported setup.

Like I said, this was the only setup that worked across multiple platforms and it worked exceptionally well... until 24.03 that is. I really hope this gets sorted out, otherwise it's a massive headache for us.

Any chances these two issues are related somehow since they occurred at the same time?