Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Zebra Routes Missing in System Route Table - v2.5

    Scheduled Pinned Locked Moved
    FRR
    frr ospf route
    1
    2
    768
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      helloadam
      last edited by helloadam

      Hello,

      I am running into a strange issue where Zebra sees the OSFP routes but does not update the system route table.

      The errors in the pfSense GUI I am seeing are the following:

      Jun 15 08:09:22	zebra	91899	warning: connected_add_ipv6 called for interface ipsec1000 with peer flag set, but no peer address supplied
Jun 15 08:09:22	zebra	91899	Can't lookup mtu by ioctl(SIOCGIFMTU)
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	zebra	91899	[EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
Jun 15 08:09:22	ospfd	92709	[EC 100663299] can't setsockopt IP_DROP_MEMBERSHIP (fd 15, addr 10.12.255.1, ifindex 11, AllSPFRouters): Can't assign requested address

      A few details about my setup:

      • This is a hub and spoke IPsec VTI configuration using OSPF

      • The error above only exists in one of my spoke. 1 Hub, 3 spokes

      • I don't believe this is a firewall issue but rather something with the FRR package, pfSense UI or maybe hardware related

      • This is IPv4 only network, not sure why error is saying anything about IPv6

      • I have destroyed and re-created the Phase 1 and Phase 2 entries on the offending spoke as well as any entries on its corresponding hub.

      • PowerCycled hub and spoke multiple times

      • If go to Status -> Services and perform hail mary of restarting the following services, system route table gets updated and everything works (yahoo!!) but then sometime later the routes disappear.

      • The services I restart are: ipsec + FRR zebra, FTT watchff, FRR staticd, FRR ospfd

      Zebra Routes from Spoke:

      Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

K>* 0.0.0.0/0 [0/0] via $SPOKE-PUBLIC-GW-XXX.67.238.1, em0, 1d20h45m
O   10.10.10.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.12.70.0/24 [110/10] is directly connected, em1, weight 1, 02:17:52  # Spoke local network
C>* 10.12.70.0/24 [0/1] is directly connected, em1, 02:17:52  # Spoke local  network
O   10.12.70.53/32 [110/10] is directly connected, em1, weight 1, 02:17:51  # Spoke local network
C>* 10.12.70.53/32 [0/1] is directly connected, em1, 02:17:51 # Spoke local  network
O   10.12.71.0/24 [110/100] is directly connected, em2, weight 1, 1d20h45m  # Spoke local network
C>* 10.12.71.0/24 [0/1] is directly connected, em2, 1d20h45m  # Spoke local network
C>* 10.12.255.0/30 [0/1] is directly connected, ipsec1000, 02:17:28  # VTI Point-to-Point network
O   10.42.12.0/24 [110/30] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.83.50.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.83.83.0/24 [110/20] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
O   10.183.30.0/24 [110/30] via 10.12.255.2, ipsec1000 inactive onlink, weight 1, 02:17:11  # From Hub, missing in system route table
K>* $HUB-PUBLIC-ADDRESS-XXX.123.246.10/32 [0/0] via $SPOKE-PUBLIC-GW-XXX.67.238.1, em0, 02:17:29
C>* $SPOKE-PUBLIC-SUBNET-XXX.67.238.0/23 [0/1] is directly connected, em0, 1d20h45m
      1 Reply Last reply Reply Quote 0
      • H
        helloadam
        last edited by

        As an update, I have done some more troubleshooting on the issue:

        • Switching to static routes over the VTI tunnel works. Using regular tunnel IPv4 also works Its only when we use FRR via OSPF (have not tested BGP) that traffic does not flow between hub and spoke.

          • Topology is 1 Hub (virtual) with 3 spokes (2 virtual, 1 physical pfSense). Its the physical pfSense spoke that is having issue

        • Enable IPsec MSS Clamping with different values, 1400, 1350, 1200, etc. on both hub and spoke and no issue. Also adjusted the VTI MTU value as well with no luck

        • Both sides are using AES-NI CPU Crypto. Enable/Disabling this has no effect

        • Both sides are using IPsec Asynchronous Cryptography. Enable/Disabling this has no effect

        • Tried different P2 encryption options but no luck. Currently using

          • P1: AES128-GCM (128 bits) AES-XCBC via 14 (2048) DH Group
          • P2: ESP AES128-GCM (128 bits) PFS Group: 14 (2048). NO Hash algorithms

        It appears another user on Reddit is facing similar issues: https://www.reddit.com/r/PFSENSE/comments/mzab6v/251_and_ipsec_vti/

        Any ideas why FRR and OSPF is not sending traffic over the network? What troubleshooting steps can I take to debug this further?

        1 Reply Last reply Reply Quote 0
        • First post
          Last post