VTI gateways not adding static routes in 24.03

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

However you read it pfSense cannot use that to create a dynamic gateway. It needs a single IP address.

It works. I've tried it with physical routers from Netgate, Fortinet and Juniper on the other side as well as virtual pfsense instances.

So far every tunnel setup has been 100% solid and reliable, no issues whatsoever.

[stephenw10]

Yes, I'm not saying it can't work, you can route via an interface, but the gateway will always show as dynamic or 0.0.0.0 and not an actual gateway IP.

Most users would not have seen a change in the reported gateway behaviour because in both 23.09.1 and 24.03 a single IP defined in the P2 will be shown as the gateway address.

[OhYeah 0]

Upgraded a client's Netgate 4100 model to 24.03 and it broke the hub-and-spoke topology. None of the combination of global states and IPSEC filtering mode settings worked. I will investigate further tomorrow and let you know if I come up with a working solution.

[LarryFahnoe]

@OhYeah-0 did you notice if your static routes were getting loaded properly? On mine, while the IPsec VTI comes up, the static routes are not getting loaded so traffic to the remote network(s) doesn't flow.

--Larry

[OhYeah 0]

@LarryFahnoe Good catch. It seems that this is indeed the issue: the required routes are not loaded.

Devs, any comments? Is this a bug or are we missing something here?

[LarryFahnoe]

@OhYeah-0 Interesting, good to know there is another related case. I attempted to document what I was seeing in https://redmine.pfsense.org/issues/15449 In my case it initially appeared that the cause was deleting an old disabled gateway, but after many tests I do not believe this to be the root cause. On one system following Lev's suggestion in note #8 I've gotten it to the point where the static route does load at boot, on another those steps did not work and I have to wait for about 15 minutes for rc.newwanip to trigger something that causes the route to get loaded. There is more detail in a support ticket, but I cannot see the ticket anymore. My configuration was working reliably in 23.09.1, but not 24.03.

--Larry

[stephenw10]

So you resaved the static routes and it created them as expected?

Are they not present after boot for both of you?

Do you both have a disabled gateway?

[LarryFahnoe]

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

Running 24.03, the static route is created and loaded as normal. By loaded I mean shows up in the routing table and traffic passes as expected.

Are they not present after boot for both of you?

(speaking for myself) Correct.

Do you both have a disabled gateway?

No, I have gotten rid of the disabled gateway & I no longer think this has anything to do with the issue.

As background, both of my systems were initially configured with static private IPv4 addresses behind CPE routers, hence WANGW gateways with static addresses. Later on I either switched providers or reconfigured the CPE device to become a bridge and now both are IPv4 with dynamic addresses, hence WAN_DHCP gateways. I had left the WANGW gateways in place but disabled in case I wanted to revert, but once I upgraded to 24.03 I had no plans to revert, so deleted the disabled gateway. Observing that deleting the gateway and rebooting resulted in a situation where the tunnel no longer passed traffic, I initially felt it was due to deleting the gateway.

Now I observe that if I roll back to 23.09.1, delete the gateway, reboot and verify that the tunnel is functioning and then upgrade to 24.03, the problem with the broken tunnel (which is really due to the missing static route) shows up.

--Larry

[LarryFahnoe]

At the risk of muddying the waters and showing my own ignorance of the /etc/rc* mechanics of pfSense, I'll also share that I have seen two different behaviors with the static route.

1. With the static route defined in the config, it is seemingly never is loaded after a reboot.

2. With the static route defined in the config, it is loaded about 15 minutes after the reboot. It appears that rc.newwanip triggers the the route being loaded, but the WAN address did not change.

I'm happy to provide whatever evidence or data is necessary to help diagnose this bug. I'm a seasoned system admin, just not as well versed in pfSense or FreeBSD.

--Larry

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

I'm not sure what you mean by "resaved the static routes", can you clarify?

The static routes defined for IPSEC tunnels have not loaded, the Netgate 4100 device has been running now close to 24hrs.

I do not have a disabled gateway.

[LarryFahnoe]

@OhYeah-0 Stephen's question about resaving the route is related to the steps I was asked to try by Lev (in the redmine above)

I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

--Larry

[OhYeah 0]

This post is deleted!

[stephenw10]

@OhYeah-0 said in VTI gateways in 24.03:

I'm not sure what you mean by "resaved the static routes", can you clarify?

I mean if you edit the static route and resave it (without changing anything) does the route then appear?

So is the static route prevented entirely or just at boot.

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

I mean if you edit the static route and resave it (without changing anything) does the route then appear?

Nope...

[stephenw10]

Is there anything you can do to make the static routes return?

Do you see any errors logged when you resaved the static route? In the System or Routing logs?

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

Do you see any errors logged when you resaved the static route? In the System or Routing logs?

I did see this bit in the "general" section of system logs after I resaved the static routes. These log entries repeated for every static route.

May 19 13:32:14 php-fpm 54069 /system_routes_edit.php: Configuration Change: admin@xxx.xxx.xx.xx (Local Database): Saved static route configuration.
May 19 13:32:14 check_reload_status 646 Syncing firewall
May 19 13:32:16 php-fpm 594 /system_routes.php: Gateway, NONE AVAILABLE
May 19 13:32:16 check_reload_status 646 Reloading filter

PS. Obscured my IP address.

[OhYeah 0]

@LarryFahnoe said in VTI gateways in 24.03:

I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

No no, I was actually relieved to find out that someone else had ran into the same issue.

PS. When you resave the static route do you get the same messaged in system logs/general?

[LarryFahnoe]

@OhYeah-0 Yes, same as the messages you show.

May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Configuration Change: fahnoe@192.168.5.67 (Local Database): Saved static route configuration.
May 19 16:02:36 pfs-m check_reload_status[645]: Syncing firewall
May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 19 16:02:40 pfs-m php-fpm[594]: /system_routes.php: Gateway, NONE AVAILABLE
May 19 16:02:40 pfs-m check_reload_status[645]: Reloading filter

[stephenw10]

No errors? Nothing in the Routing log?

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

No errors? Nothing in the Routing log?

Nothing else apart from the "gateway not available" one.

I booted the device back into 23.09 until a fix is found.