Zebra Routes Missing in System Route Table - v2.5

helloadam

Hello,

I am running into a strange issue where Zebra sees the OSFP routes but does not update the system route table.

The errors in the pfSense GUI I am seeing are the following:

Jun 15 08:09:22	zebra	91899	warning: connected_add_ipv6 called for interface ipsec1000 with peer flag set, but no peer address supplied
Jun 15 08:09:22	zebra	91899	Can't lookup mtu by ioctl(SIOCGIFMTU)
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	ospfd	92709	[EC 100663299] can't setsockopt IP_DROP_MEMBERSHIP (fd 15, addr 10.12.255.1, ifindex 11, AllSPFRouters): Can't assign requested address

A few details about my setup:

This is a hub and spoke IPsec VTI configuration using OSPF
The error above only exists in one of my spoke. 1 Hub, 3 spokes
I don't believe this is a firewall issue but rather something with the FRR package, pfSense UI or maybe hardware related
This is IPv4 only network, not sure why error is saying anything about IPv6
I have destroyed and re-created the Phase 1 and Phase 2 entries on the offending spoke as well as any entries on its corresponding hub.
PowerCycled hub and spoke multiple times
If go to Status -> Services and perform hail mary of restarting the following services, system route table gets updated and everything works (yahoo!!) but then sometime later the routes disappear.
The services I restart are: ipsec + FRR zebra, FTT watchff, FRR staticd, FRR ospfd

Zebra Routes from Spoke:

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

K>* 0.0.0.0/0 [0/0] via $SPOKE-PUBLIC-GW-XXX.67.238.1, em0, 1d20h45m
O   10.10.10.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.12.70.0/24 [110/10] is directly connected, em1, weight 1, 02:17:52  # Spoke local network
C>* 10.12.70.0/24 [0/1] is directly connected, em1, 02:17:52  # Spoke local  network
O   10.12.70.53/32 [110/10] is directly connected, em1, weight 1, 02:17:51  # Spoke local network
C>* 10.12.70.53/32 [0/1] is directly connected, em1, 02:17:51 # Spoke local  network
O   10.12.71.0/24 [110/100] is directly connected, em2, weight 1, 1d20h45m  # Spoke local network
C>* 10.12.71.0/24 [0/1] is directly connected, em2, 1d20h45m  # Spoke local network
C>* 10.12.255.0/30 [0/1] is directly connected, ipsec1000, 02:17:28  # VTI Point-to-Point network
O   10.42.12.0/24 [110/30] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.83.50.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.83.83.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.183.30.0/24 [110/30] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
K>* $HUB-PUBLIC-ADDRESS-XXX.123.246.10/32 [0/0] via $SPOKE-PUBLIC-GW-XXX.67.238.1, em0, 02:17:29
C>* $SPOKE-PUBLIC-SUBNET-XXX.67.238.0/23 [0/1] is directly connected, em0, 1d20h45m

helloadam

As an update, I have done some more troubleshooting on the issue:

Switching to static routes over the VTI tunnel works. Using regular tunnel IPv4 also works Its only when we use FRR via OSPF (have not tested BGP) that traffic does not flow between hub and spoke.
- Topology is 1 Hub (virtual) with 3 spokes (2 virtual, 1 physical pfSense). Its the physical pfSense spoke that is having issue
Enable IPsec MSS Clamping with different values, 1400, 1350, 1200, etc. on both hub and spoke and no issue. Also adjusted the VTI MTU value as well with no luck
Both sides are using AES-NI CPU Crypto. Enable/Disabling this has no effect
Both sides are using IPsec Asynchronous Cryptography. Enable/Disabling this has no effect
Tried different P2 encryption options but no luck. Currently using
- P1: AES128-GCM (128 bits) AES-XCBC via 14 (2048) DH Group
- P2: ESP AES128-GCM (128 bits) PFS Group: 14 (2048). NO Hash algorithms

It appears another user on Reddit is facing similar issues: https://www.reddit.com/r/PFSENSE/comments/mzab6v/251_and_ipsec_vti/

Any ideas why FRR and OSPF is not sending traffic over the network? What troubleshooting steps can I take to debug this further?