VTI gateways not adding static routes in 24.03

LarryFahnoe

@OhYeah-0 did you notice if your static routes were getting loaded properly? On mine, while the IPsec VTI comes up, the static routes are not getting loaded so traffic to the remote network(s) doesn't flow.

--Larry

OhYeah 0

@LarryFahnoe Good catch. It seems that this is indeed the issue: the required routes are not loaded.

Devs, any comments? Is this a bug or are we missing something here?

LarryFahnoe

@OhYeah-0 Interesting, good to know there is another related case. I attempted to document what I was seeing in https://redmine.pfsense.org/issues/15449 In my case it initially appeared that the cause was deleting an old disabled gateway, but after many tests I do not believe this to be the root cause. On one system following Lev's suggestion in note #8 I've gotten it to the point where the static route does load at boot, on another those steps did not work and I have to wait for about 15 minutes for rc.newwanip to trigger something that causes the route to get loaded. There is more detail in a support ticket, but I cannot see the ticket anymore. My configuration was working reliably in 23.09.1, but not 24.03.

--Larry

stephenw10

So you resaved the static routes and it created them as expected?

Are they not present after boot for both of you?

Do you both have a disabled gateway?

LarryFahnoe

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

Running 24.03, the static route is created and loaded as normal. By loaded I mean shows up in the routing table and traffic passes as expected.

Are they not present after boot for both of you?

(speaking for myself) Correct.

Do you both have a disabled gateway?

No, I have gotten rid of the disabled gateway & I no longer think this has anything to do with the issue.

As background, both of my systems were initially configured with static private IPv4 addresses behind CPE routers, hence WANGW gateways with static addresses. Later on I either switched providers or reconfigured the CPE device to become a bridge and now both are IPv4 with dynamic addresses, hence WAN_DHCP gateways. I had left the WANGW gateways in place but disabled in case I wanted to revert, but once I upgraded to 24.03 I had no plans to revert, so deleted the disabled gateway. Observing that deleting the gateway and rebooting resulted in a situation where the tunnel no longer passed traffic, I initially felt it was due to deleting the gateway.

Now I observe that if I roll back to 23.09.1, delete the gateway, reboot and verify that the tunnel is functioning and then upgrade to 24.03, the problem with the broken tunnel (which is really due to the missing static route) shows up.

--Larry

LarryFahnoe

At the risk of muddying the waters and showing my own ignorance of the /etc/rc* mechanics of pfSense, I'll also share that I have seen two different behaviors with the static route.

With the static route defined in the config, it is seemingly never is loaded after a reboot.
With the static route defined in the config, it is loaded about 15 minutes after the reboot. It appears that rc.newwanip triggers the the route being loaded, but the WAN address did not change.

I'm happy to provide whatever evidence or data is necessary to help diagnose this bug. I'm a seasoned system admin, just not as well versed in pfSense or FreeBSD.

--Larry

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

I'm not sure what you mean by "resaved the static routes", can you clarify?

The static routes defined for IPSEC tunnels have not loaded, the Netgate 4100 device has been running now close to 24hrs.

I do not have a disabled gateway.

LarryFahnoe

@OhYeah-0 Stephen's question about resaving the route is related to the steps I was asked to try by Lev (in the redmine above)

I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

--Larry

OhYeah 0

This post is deleted!

stephenw10

@OhYeah-0 said in VTI gateways in 24.03:

I'm not sure what you mean by "resaved the static routes", can you clarify?

I mean if you edit the static route and resave it (without changing anything) does the route then appear?

So is the static route prevented entirely or just at boot.

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

I mean if you edit the static route and resave it (without changing anything) does the route then appear?

Nope...

stephenw10

Is there anything you can do to make the static routes return?

Do you see any errors logged when you resaved the static route? In the System or Routing logs?

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

Do you see any errors logged when you resaved the static route? In the System or Routing logs?

I did see this bit in the "general" section of system logs after I resaved the static routes. These log entries repeated for every static route.

May 19 13:32:14 php-fpm 54069 /system_routes_edit.php: Configuration Change: admin@xxx.xxx.xx.xx (Local Database): Saved static route configuration.
May 19 13:32:14 check_reload_status 646 Syncing firewall
May 19 13:32:16 php-fpm 594 /system_routes.php: Gateway, NONE AVAILABLE
May 19 13:32:16 check_reload_status 646 Reloading filter

PS. Obscured my IP address.

OhYeah 0

@LarryFahnoe said in VTI gateways in 24.03:

I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

No no, I was actually relieved to find out that someone else had ran into the same issue.

PS. When you resave the static route do you get the same messaged in system logs/general?

LarryFahnoe

@OhYeah-0 Yes, same as the messages you show.

May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Configuration Change: fahnoe@192.168.5.67 (Local Database): Saved static route configuration.
May 19 16:02:36 pfs-m check_reload_status[645]: Syncing firewall
May 19 16:02:36 pfs-m php-fpm[67932]: /system_routes_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 19 16:02:40 pfs-m php-fpm[594]: /system_routes.php: Gateway, NONE AVAILABLE
May 19 16:02:40 pfs-m check_reload_status[645]: Reloading filter

stephenw10

No errors? Nothing in the Routing log?

OhYeah 0

@stephenw10 said in VTI gateways in 24.03:

No errors? Nothing in the Routing log?

Nothing else apart from the "gateway not available" one.

I booted the device back into 23.09 until a fix is found.

LarryFahnoe

@stephenw10 said in VTI gateways in 24.03:

No errors? Nothing in the Routing log?

No. This is in part why I opened the redmine and am trying to provide information. I believe my config to be quite simple: just a pair of 4200s with an IPsec VTI between them & and static routes to the LANs on either side, so I would have expected that others would be seeing the same thing. It sounds like @OhYeah-0 has a somewhat more complex config but is seeing a similar issue. That such a simple config (as mine is) that was working properly prior to the upgrade to 24.03 spells BUG to me.

Earlier I'd asked on the support thread about enabling debugging but got crickets. I see debug is set to false in /etc/inc/globals.inc and am tempted to turn that on. Is there a better or supported way to do that via the GUI somewhere? If so, I haven't found it.

--Larry

OhYeah 0

I just remember that I installed another new Netgate 4100 for a new client and that device isn't actively being used, so I can use it for testing. It was immediately updated to 24.03 and it is showing exactly the same behavior.

I tried deleting the existing static route and re-create it, it is still not appearing in the routes table. No error messages in system logs -> routing.

My gut feeling is that the core reason of the bug is pfsense not considering 0.0.0.0/0 routing valid and thus not applying the static routes to the routes table.

LarryFahnoe

@OhYeah-0 As mentioned above, mine are using a /30 transit network rather than the 0.0.0.0/0 config you have, but we seem to be seeing the same thing: the static route doesn't load. My curious gut says: is there a timing issue where the tunnel hasn't come up yet which makes the static route seem invalid? Seems like the logs are not telling us the whole story though.

--Larry