VTI gateways not adding static routes in 24.03

[OhYeah 0]

I have minor question for devs:

since the update to 24.03, routed IPSEC tunnel gateways show the IP address as "0.0.0.0" while in the previous versions is simply showed as "dynamic". Is this intended behavior?

[stephenw10]

When the tunnel is UP?

Where are you seeing that?

I'd expect it to show the actual gateway IP there.

Steve

[OhYeah 0]

@stephenw10 In the GUI, under System -> Routing -> Gateways. What is also strange that under gateway monitor IP it doesn't show anything. In 23.09 they showed both as "dynamic".

I use IPSEC in VTI mode and set local and remote network types to "network" and 0.0.0.0/0 and use static routes.

[stephenw10]

Hmm, you would normally set the local and remote networks to single IP addresses and then route traffic via that:
https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/routed-vti.html#ipsec-configuration

[LarryFahnoe]

For what it's worth, I use a /30 transit network per the reference Stephen cited and under System / Routing my gateways show their remote network addresses. This behavior has been consistent in 23.09.1 and 24.03. Seems to me that the only time I see "dynamic" is if the tunnel is down.

--Larry

[OhYeah 0]

@stephenw10

I can't remember the specifics, but I ran into all sorts of problems/issues when trying to set up the tunnels following those instructions.

Through some trial and error and consulting with other people here is the system that I use:

Set up P1 as standard
Set up P2 as VTI and set local/remote network types to "network" and 0.0.0.0/0
Assign the VTI connection to a virtual adapter and enable interface
Assign a gateway to the the newly created virtual adapter (these always show IP as "dynamic", or at least did before)
Set up static routes accordingly

This system has been running without any issues with several clients, the most complex of which has multiple sites running both virtual pfsense (CE), physical Netgate routers and Fortinet routers. We've also tried the same system with Junipers on the other side, worked flawlessly.

[stephenw10]

Hmm, well I would expect to need a manually assigned gateway to see a real IP there then. pfSense can't create one for the remote side when it's defined as the entire internet!

I'm surprised it doesn't break routing for other traffic. You should only need a /0 P2 like that in policy mode IPSec here it defines what can be carried.

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

pfSense can't create one for the remote side when it's defined as the entire internet!

I would rephrase that in a different way. 0.0.0.0/0 just means "take any path that's allowed by the routing table".

[stephenw10]

However you read it pfSense cannot use that to create a dynamic gateway. It needs a single IP address.

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

However you read it pfSense cannot use that to create a dynamic gateway. It needs a single IP address.

It works. I've tried it with physical routers from Netgate, Fortinet and Juniper on the other side as well as virtual pfsense instances.

So far every tunnel setup has been 100% solid and reliable, no issues whatsoever.

[stephenw10]

Yes, I'm not saying it can't work, you can route via an interface, but the gateway will always show as dynamic or 0.0.0.0 and not an actual gateway IP.

Most users would not have seen a change in the reported gateway behaviour because in both 23.09.1 and 24.03 a single IP defined in the P2 will be shown as the gateway address.

[OhYeah 0]

Upgraded a client's Netgate 4100 model to 24.03 and it broke the hub-and-spoke topology. None of the combination of global states and IPSEC filtering mode settings worked. I will investigate further tomorrow and let you know if I come up with a working solution.

[LarryFahnoe]

@OhYeah-0 did you notice if your static routes were getting loaded properly? On mine, while the IPsec VTI comes up, the static routes are not getting loaded so traffic to the remote network(s) doesn't flow.

--Larry

[OhYeah 0]

@LarryFahnoe Good catch. It seems that this is indeed the issue: the required routes are not loaded.

Devs, any comments? Is this a bug or are we missing something here?

[LarryFahnoe]

@OhYeah-0 Interesting, good to know there is another related case. I attempted to document what I was seeing in https://redmine.pfsense.org/issues/15449 In my case it initially appeared that the cause was deleting an old disabled gateway, but after many tests I do not believe this to be the root cause. On one system following Lev's suggestion in note #8 I've gotten it to the point where the static route does load at boot, on another those steps did not work and I have to wait for about 15 minutes for rc.newwanip to trigger something that causes the route to get loaded. There is more detail in a support ticket, but I cannot see the ticket anymore. My configuration was working reliably in 23.09.1, but not 24.03.

--Larry

[stephenw10]

So you resaved the static routes and it created them as expected?

Are they not present after boot for both of you?

Do you both have a disabled gateway?

[LarryFahnoe]

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

Running 24.03, the static route is created and loaded as normal. By loaded I mean shows up in the routing table and traffic passes as expected.

Are they not present after boot for both of you?

(speaking for myself) Correct.

Do you both have a disabled gateway?

No, I have gotten rid of the disabled gateway & I no longer think this has anything to do with the issue.

As background, both of my systems were initially configured with static private IPv4 addresses behind CPE routers, hence WANGW gateways with static addresses. Later on I either switched providers or reconfigured the CPE device to become a bridge and now both are IPv4 with dynamic addresses, hence WAN_DHCP gateways. I had left the WANGW gateways in place but disabled in case I wanted to revert, but once I upgraded to 24.03 I had no plans to revert, so deleted the disabled gateway. Observing that deleting the gateway and rebooting resulted in a situation where the tunnel no longer passed traffic, I initially felt it was due to deleting the gateway.

Now I observe that if I roll back to 23.09.1, delete the gateway, reboot and verify that the tunnel is functioning and then upgrade to 24.03, the problem with the broken tunnel (which is really due to the missing static route) shows up.

--Larry

[LarryFahnoe]

At the risk of muddying the waters and showing my own ignorance of the /etc/rc* mechanics of pfSense, I'll also share that I have seen two different behaviors with the static route.

1. With the static route defined in the config, it is seemingly never is loaded after a reboot.

2. With the static route defined in the config, it is loaded about 15 minutes after the reboot. It appears that rc.newwanip triggers the the route being loaded, but the WAN address did not change.

I'm happy to provide whatever evidence or data is necessary to help diagnose this bug. I'm a seasoned system admin, just not as well versed in pfSense or FreeBSD.

--Larry

[OhYeah 0]

@stephenw10 said in VTI gateways in 24.03:

So you resaved the static routes and it created them as expected?

I'm not sure what you mean by "resaved the static routes", can you clarify?

The static routes defined for IPSEC tunnels have not loaded, the Netgate 4100 device has been running now close to 24hrs.

I do not have a disabled gateway.

[LarryFahnoe]

@OhYeah-0 Stephen's question about resaving the route is related to the steps I was asked to try by Lev (in the redmine above)

I'm not meaning to hijack your thread, but it would appear we're both stumbling over the same (or related) bug: the static route for a remote network across an IPsec VTI is not being loaded.

--Larry