vti routed ipsec tunnel interface is down permanently

nzkiwi68

2.4.4_p3
We have some 50+ traditional P1/P2 ipec tunnels.

I want to setup multi WAN failover between 2 sites which both have HA firewalls and WAN1 and WAN2. To do this I will use the excellent FRR package, vti ipsec tunnel interfaces and OSFP.

I have another site running all this perfectly.

One end setup fine, the problem end, the ipsec vti interface is down.
It doesn't appear at all in the local arp, you can't ping the local vti IP address, OSFP or course doesn't work and when I dug more, under interfaces it's down.

> Status > Interfaces;
HO_SY1_WAN Interface (opt5, ipsec52000)
Status
down

I wonder if it's the unusually high ispec number of 52000?

I've tried:

rebooting the firewall(s)
deleting the p2 vti and making it again
deleting p1 and the p2 vti
making an a new p1/vti combo (it became ipsec53000 and was also permanently down too)
Exporting the config and restoring the config in the hope that might fix it (it didn't)

nzkiwi68

Ok.

Having another look tonight. Same deal. Create a vti interface and it's down, no matter what.

This looks like a bug / config corruption but I have no idea what to do from here.

nzkiwi68

And, of course I suppose, the router never appears in the rotuing table and of course you can't ping the vti interface either.

Diagnostics > Routes
Your vti interface subnet (in my case 172.31.254.2 /30) doesn't appear in the route table.

Anyone have any ideas what I can look at?
I've eposted the config, it looks clean and then restored it, but that hasn't helped.

stephenw10

Curious. I assume the tunnel itself is up? And the tunnel and interface both show up at the other end?

Do both ends have the existing 50 P1s? Or just the end giving trouble?

Steve

nzkiwi68

@stephenw10
P1 comes up, other end is all good, you can ping the local end of the vti interface, it's up, but the problem end, the vti interface is d-d-d-d-d-d-down.

Doesn't really matter if p1 completes or not, the vti interface is an interface and it should be up, after all, it's a local interface.

That's what's got me stumped.

stephenw10

The IPSec connection number is unusually high as you say which is why I asked if those tunnels exist at both ends.

Is the number much lower at the working end?

Steve

nzkiwi68

ipsec1000 - very normal looking

I must say, I haven't raised a case, but, it's happened too many times on the problem firewall, with normal p1/p2 ipsec tunnels (it did it again last night after a reboot)

One or so tunnels (that was working fine) will suddenly have p2 (0) - the p2 is completely gone burgers!

I really wonder if it's ipsec config bit / interface numbers is broken.

stephenw10

Hmm, like in the IPSec status or in the actual config? If the P2s are somehow mysteriously vanishing from the config that opens up pretty much anything! I've never sent that happen though.

Steve

nzkiwi68

With some assistance from pfsense support, and my mate Brett (thanks Brett!!!!)

It has been identified that FreeBSD has a limited on interfaces and will not accept interfaces numbered above 32767, so an interface number of 52000 is impossible.

I have created a bug report:
[https://redmine.pfsense.org/issues/9592](link url)

Brett has written a pull request that basically drops the ipsec vti interface creation padding from 000 to a single 0 and thereby changes the maximum number of vti interfaces from 32 to 3276.

[https://github.com/pfsense/pfsense/pull/4071](link url)

Looks like this will get fixed in time.