VTI gateways not adding static routes in 24.03

[LarryFahnoe]

@Nikkeli Your situation sounds a lot like mine.

Might be interesting to take a peek at your /cf/conf/config.xml and compare it to what I showed above in https://forum.netgate.com/post/1170175

Do you have a spurious <gateway_item> with a <gateway> containing an address rather than "dynamic"?

I have on my "spare time list" (ahem!) to roll back to 23.09.1, then do the upgrade again and document how the config changes. I suspect there is a bug in the upgrade process.

@stephenw10 I'd vote for adding "static routes" to the title of this thread if possible.

--Larry

[Nikkeli]

@LarryFahnoe
I actually don't have this problem, the configuration seems fine. Below is the configuration for the only (vti) gateway listed.

<gateway_item>
<interface>opt10</interface>
<gateway></gateway>
<name>IPSEC_VT13_VT10_VTIV4</name>
<weight>1</weight>
<ipprotocol>inet</ipprotocol>
<descr><![CDATA[Interface IPSEC_VT13_VT10_VTIV4 Gateway]]></descr>
<gw_down_kill_states></gw_down_kill_states>
</gateway_item>

[stephenw10]

So no additional gateways? No disabled gateways?

[Nikkeli]

@stephenw10
The only other gateway is WAN gateway. No gateways are disabled.

[stephenw10]

Hmm, any errors in the routing or system logs at boot?

[Nikkeli]

@stephenw10
On System/General I can actually see some errors/warnings that seem to be relevant. On other logs I could not find anything relevant.
IPsec logging has too much log noise but I can turn that down aswell and reboot, if you think it could help.

Here is System/General logging after booting, with the relevant lines.

May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:27 	php-cgi 	685 	rc.bootup: Gateway, NONE AVAILABLE
May 24 10:11:27 	syslogd 		kernel boot file is /boot/kernel/kernel
May 24 10:11:27 	syslogd 		exiting on signal 15
May 24 10:11:26 	kernel 		done.
May 24 10:11:26 	php-cgi 	685 	rc.bootup: Creating rrd update script
May 24 10:11:24 	kernel 		.done.
May 24 10:11:24 	check_reload_status 	650 	Restarting IPsec tunnels
May 24 10:11:24 	kernel 		...
May 24 10:11:15 	kernel 		done.
May 24 10:11:15 	check_reload_status 	650 	Updating all dyndns
May 24 10:11:14 	kernel 		done.
May 24 10:11:14 	php-cgi 	685 	rc.bootup: NTPD is starting up.
May 24 10:11:08 	kernel 		done.
May 24 10:11:08 	kernel 		done.
May 24 10:11:08 	php-cgi 	685 	rc.bootup: sync unbound done.
May 24 10:11:07 	kernel 		done.
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: Gateway, NONE AVAILABLE
May 24 10:11:07 	php-cgi 	685 	rc.bootup: Default gateway setting as default.

[OhYeah 0]

Rebooted device, went through the logs to see if I catch something that might be relevant (Netgate 4100).

May 24 13:53:46	php-cgi	678	rc.bootup: The command '/sbin/ifconfig 'ipsec1' inet '0.0.0.0/0' '0.0.0.0'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCAIFADDR): Destination address required'
May 24 13:53:46	php-cgi	678	rc.bootup: Gateway, NONE AVAILABLE
May 24 13:53:46	php-cgi	678	rc.bootup: Gateway, NONE AVAILABLE
May 24 13:53:46	kernel		route: message indicates error: Invalid argument

[stephenw10]

Ah, there we go. Yup that's pretty much what I'd expect when trying to use 0/0. It tries to apply it to the interfaces and fails because it's invalid there.

The interesting thing is how that ever worked in 23.09.

[OhYeah 0]

And these are similar messages from a Netgate 4100 running 23.09:

May 24 19:26:59	php-cgi	466	rc.bootup: The command '/sbin/ifconfig 'ipsec2' inet '0.0.0.0/0' '0.0.0.0/0'' returned exit code '1', the output was 'ifconfig: 0.0.0.0/0: bad value'
May 24 19:26:59	php-cgi	466	rc.bootup: Gateway, NONE AVAILABLE

The message is very slightly different, so I assume it must be meaningful in some way.

I also got offered 24.03_1 on the same device but no release notes yet?

[stephenw10]

Hmm, interesting. Presumably you don't see the route errors in 23.09?:

May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1
May 24 10:11:07 	php-cgi 	685 	rc.bootup: route_add_or_change: Invalid gateway and/or network interface ipsec1

The patch 1 update is a no-op for amd64 devices. It applies only to aarch64. It won't change anything here.

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

Hmm, interesting. Presumably you don't see the route errors in 23.09?

Nope, didn't see any..

[OhYeah 0]

This post is deleted!

[OhYeah 0]

Any news from devs regarding this issue? Well actually two issues I guess.

[stephenw10]

Not yet. In all honesty it's pretty low priority because VTI / Static routes are working as intended in 24.03. Using 0/0 for both ends of the tunnel subnet was never a supported setup.

It is curious that is changed though.

The issue with disabled gateways causing a problem is a bigger issue since that happens in the expected config. Updates there should be shown on the bug report: https://redmine.pfsense.org/issues/15449

[OhYeah 0]

@stephenw10 said in VTI gateways not adding static routes in 24.03:

In all honesty it's pretty low priority because VTI / Static routes are working as intended in 24.03. Using 0/0 for both ends of the tunnel subnet was never a supported setup.

Like I said, this was the only setup that worked across multiple platforms and it worked exceptionally well... until 24.03 that is. I really hope this gets sorted out, otherwise it's a massive headache for us.

Any chances these two issues are related somehow since they occurred at the same time?

[marcosm]

I've added a patch to the redmine that should fix the issue:
https://redmine.pfsense.org/issues/15449

Note that while it's valid for the routing to work for an interface regardless of its IP, the strongswan docs seem to indicate that a point-to-point link with specific local/remote addresses is expected. The IPsec P2 configuration in pfSense uses the local and remote fields to build the interface, and "0.0.0.0/0,::/0" is added on top as part of the traffic selectors. We do not recommend nor support using 0/0 as the interface address.

[LarryFahnoe]

@marcosm I just applied the patch to my 2 4200 systems and then rebooted. The static route was added at boot and traffic passes as expected without having to wait for rc.newwanip to trigger the route to get loaded about 15 minutes after the reboot. Many thanks!!

--Larry

[Nikkeli]

@marcosm
Thanks, this fixed the static routes not being applied from boot for me aswell. I did not have 0/0 address in IPSec, just VTI IPsec and static routes.

[OhYeah 0]

@marcosm said in VTI gateways not adding static routes in 24.03:

Note that while it's valid for the routing to work for an interface regardless of its IP, the strongswan docs seem to indicate that a point-to-point link with specific local/remote addresses is expected.

I applied the patch on a non-production Netgate 4100 and sadly I have to say it did not fix the problem with my 0/0 setup. I was tinkering around with various config settings but so far no luck.

Any idea why it was working before in earlier versions?

[marcosm]

@OhYeah-0 I haven't looked into that specifically, but my guess is it's related to the error shown on https://forum.netgate.com/post/1170859