FRR/OSPF won't distribute static routes from FRR/Zebra


  • LAYER 8 Moderator

    Hi,

    we set up FRR/OSPF between two sites. Both sites are connected via an MPLS link and have a static-key-OVPN tunnel (as suggested for FRR) over their WAN link as a failback.

    As FRR pushed not only routes from the static routes section from pfSense GUI but also Host /32 routes configured by pfSense itself (e.g. OVPN peer, DNS via WAN1/2 etc.) and as denying those was ignored, we disabled Kernel routing table pushing and only selected "FRR static routes" which - as @jimp showed in the hangout - should push routes configured under FRR/zebra. But although we put those routes in there, the aren't pushed to the OSPF peers.

    Any thoughts as to why?

    Greets Jens


  • Rebel Alliance Netgate Administrator

    What status are the other routers showing in status -> frr - OSPF Neighbors


  • LAYER 8 Moderator

    @chrismacmahon

    One side is

    full/DR
    full/DRother
    

    the other side is

    full/Backup
    full/DRother
    

    DR/Backup state is on the MPLS link which is OK, DRother is on the ovpns/c interfaces.


  • Rebel Alliance Developer Netgate

    Can you share the contents of /var/etc/frr/ospfd.conf and /var/etc/frr/zebra.conf? I haven't tried doing that specifically in a while, but it worked last time I did. You do have to enable redistribution of static routes in addition to making the routes in Zebra.


  • LAYER 8 Moderator

    @jimp said in FRR/OSPF won't distribute static routes from FRR/Zebra:

    I haven't tried doing that specifically in a while, but it worked last time I did.

    Me too, that's why I'm completely baffled it won't propagate.

    On another thought: Do we have to remove the static routes from "System / Routing" for the static routes of FRR/Zebra to work (so to not define them two times in different locations)?

    You do have to enable redistribution of static routes in addition to making the routes in Zebra

    Aye, that's what I've done:

    744ad3b4-2f00-40db-a9c1-58f094452e13-image.png
    Routes set up in FRR/Zebra/General config

    a7e5c7f9-5e9e-455e-bd6a-8081333978e1-image.png
    Routes from Zebra to propagate as per your Hangout (and my previous setup test) in FRR/OSPF settings

    Really baffled.


  • LAYER 8 Moderator

    @jimp

    That's from the remote side (e.g. the screenshot above with 192.168.10.0/24 is one of those local nets to push to the other side)

    ##################### DO NOT EDIT THIS FILE! ######################
    ###################################################################
    # This file was created by an automatic configuration generator.  #
    # The contents of this file will be overwritten without warning!  #
    ###################################################################
    password xxx
    log syslog
    
    # Null Routes
    
    # Static Routes
    ip route 192.168.10.0/24 10.99.9.49
    ip route 172.17.41.0/24 10.99.9.49
    ip route 172.17.47.0/24 10.99.9.49
    ip route 172.17.61.0/24 10.99.9.49
    ip route 192.168.71.0/24 10.99.9.49
    
    # Accept Filters
    ip prefix-list ACCEPTFILTER deny 10.99.9.16/28
    ip prefix-list ACCEPTFILTER deny 10.99.9.19/32
    ip prefix-list ACCEPTFILTER deny 10.20.20.0/30
    ip prefix-list ACCEPTFILTER deny 10.20.20.2/32
    ip prefix-list ACCEPTFILTER deny 172.30.31.0/24
    ip prefix-list ACCEPTFILTER deny 172.30.31.201/32
    ip prefix-list ACCEPTFILTER deny 10.99.9.32/28
    ip prefix-list ACCEPTFILTER deny 10.99.9.33/32
    ip prefix-list ACCEPTFILTER deny 10.99.9.48/28
    ip prefix-list ACCEPTFILTER deny 10.99.9.50/32
    ip prefix-list ACCEPTFILTER permit any
    route-map ACCEPTFILTER permit 10
     match ip address prefix-list ACCEPTFILTER
    ip protocol ospf route-map ACCEPTFILTER
    
    
    ##################### DO NOT EDIT THIS FILE! ######################
    ###################################################################
    # This file was created by an automatic configuration generator.  #
    # The contents of this file will be overwritten without warning!  #
    ###################################################################
    password xxx
    log syslog
    interface igb3
      ip ospf cost 10
    interface ovpnc1
      ip ospf cost 100
    interface igb1
    interface igb0
    interface igb2
    
    router ospf
      ospf router-id 10.99.9.19
      redistribute static route-map DNR
      passive-interface igb1
      passive-interface igb0
      passive-interface igb2
      network 10.99.9.16/28 area 0.0.0.0
      network 10.20.20.0/30 area 0.0.0.0
      network 172.30.31.0/24 area 0.0.0.0
      network 10.99.9.32/28 area 0.0.0.0
      network 10.99.9.48/28 area 0.0.0.0
    access-list dnr-list permit any
    route-map DNR permit 10
      match ip address dnr-list
    

  • Rebel Alliance Developer Netgate

    Is this on 2.5.0 or 2.4.4-p3? I found an issue on 2.5.0 that would prevent static routes in Zebra from working, but I don't have a 2.4.4-p3 setup handy to check right now (and that bug wouldn't be possible there, as it would only affect FRR 6+).

    That said, once I got the FRR static route to show in the table, it happily propagated across OSPF to the neighbor.

    If you go into vtysh do your static routes show up in show ip route? If so, do they have K> or S>?

    If you already have static routes in System > Routing you could just use the option to redistribute kernel routes instead of static routes.


  • LAYER 8 Moderator

    @jimp said in FRR/OSPF won't distribute static routes from FRR/Zebra:

    Is this on 2.5.0 or 2.4.4-p3? I found an issue on 2.5.0 that would prevent static routes in Zebra from working, but I don't have a 2.4.4-p3 setup handy to check right now (and that bug wouldn't be possible there, as it would only affect FRR 6+).

    Nope, that ist a 2.4.4-p2 stable/production setup. No dev-version anywhere near it ;)

    If you already have static routes in System > Routing you could just use the option to redistribute kernel routes instead of static routes.

    That was my first intention, but that also triggered distribution of IPs like the WAN IP of the peer as they have an OVPN tunnel between them and that /32 IP is then distributed to the other side (as it's a static host route). Filtering out that entry didn't work either, so I thought about just dropping the hassle with all the unnecessary host routes from the kernel table and instead just configure the 4-5 static routes and be done.

    If you go into vtysh do your static routes show up in show ip route? If so, do they have K> or S>?

    Which one on which side? I assume you mean the static routes on the same side. Yes they do show.

    S 172.16.45.0/24 [1/0] via 10.99.9.6, igb11, 3d18h20m
    K>* 172.16.45.0/24 [0/0] via 10.99.9.6, igb11, 3d18h20m

    ATM they show up as both S and K. As kernel routes they get pushed (but as described above, a whole lot more, too that I don't want and filtering doesn't seem to work either), as statics they don't :/

    All 5 routes defined in the FRR/Zebra/General section show up as "S" routes.


  • Rebel Alliance Developer Netgate

    I'd use one method or the other then.

    Either

    1. Take the routes out of System > Routing and only put them in Zebra, then redistribute static
    2. Leave the routes in System > Routing and remove them from Zebra, then redistribute kernel. If there are routes you don't want to redistribute then in the OSPF main settings add them to the networks list at the bottom with Disable Redistribution checked. You have to match the route exactly, so be sure that it matches what is shown in the table.

  • LAYER 8 Moderator

    @jimp said in FRR/OSPF won't distribute static routes from FRR/Zebra:

    Take the routes out of System > Routing and only put them in Zebra, then redistribute static

    That's what I did, but they won't distribute to the other side. So I had to re-enter them for the current routing to work before I get FRR to work properly.

    Leave the routes in System > Routing and remove them from Zebra, then redistribute kernel. If there are routes you don't want to redistribute then in the OSPF main settings add them to the networks list at the bottom with Disable Redistribution checked. You have to match the route exactly, so be sure that it matches what is shown in the table.

    Did that first, too. Problem was:

    • Static Host Routes from e.g. OpenVPN peer is getting pushed to the other side - which is bad, as it is one of the WAN VIPs there. So I did exactly that - added it with /32 to the OSPF main settings. But it didn't get "deleted" from the pushed kernel routing table.
    • There were conflicts as both sides use e.g. 1.1.1.1 as Host Route for checking a specific WAN gateway so both have them as static host route. Entering it into OSPF main settings as disabled didn't delete it from the published routes either.

    So either I'm missing some switch I never had to play with before, I'm kinda bummed as to where to go now. Kernel routes would be easiest as static routes are "where they are suspected" but the removal of unnecessary routes is not working correctly, even if I try to match the route exactly it will be published nonetheless.


  • Rebel Alliance Developer Netgate

    Curious. In my lab the only OSPF pair I have setup uses a zebra static route to distribute a route to neighbors and it's working fine there. Nothing too special, it's just in the zebra static routes (not in System > Routing), and I have Redistribute Static set.


  • LAYER 8 Moderator

    As do I in the screens above. Yet nothing happens on the remote site. It's completely bonkers...



  • I found if the route is for an interface in pfSense that has a CARP address, then, the route will NOT get distributed.

    Bug!!!!


  • LAYER 8 Moderator

    Hmm in our case one side is a CARP cluster, but the other side (who also doesn't publish) is a single box and that should work, too. Would only make sense (about a possible bug) that it's enough if one side is a CARP cluster, that both sides won't publish correctly.

    Anything you could think of @jimp that could be related to that?


  • Rebel Alliance Developer Netgate

    Nothing special about CARP VIPs or them being on interfaces, not that I know of anyhow.

    We updated the FRR package for 2.4.4-p3 to FRR 6 the other day, you might try it again with that to see if there is a difference. Static route handling moved to a new daemon, staticd.

    And 2.5.0 snapshots now have FRR 7 if you want to try something even newer.


  • LAYER 8 Moderator

    @jimp said in FRR/OSPF won't distribute static routes from FRR/Zebra:

    And 2.5.0 snapshots now have FRR 7 if you want to try something even newer.

    Would like to, but that's a production customer system - no expermiental there ;)

    We updated the FRR package for 2.4.4-p3 to FRR 6 the other day, you might try it again with that to see if there is a difference. Static route handling moved to a new daemon, staticd.

    Will definetly do and test again, perhaps nearer to Friday evening to see if I can make it work.


Log in to reply