OpenBGP with CARP in 2.2.4 and two ISPs



  • We want to switch to a multihome setup with our DC ISP. To do this, we've been working on a pfsense setup with 2 pfsense VM's running OpenBGP and with CARP addresses on both the WAN and LAN side. I'm now trying to setup BGP sessions with both upstream routers and have them use our WAN CARP address for the nexthop like https://forum.pfsense.org/index.php?topic=57614.0. I've tried the setup suggested by Reiner030 but have had the same issues as IcePick had in that thread. OpenBGP

    Is it possible to have BGP sessions with both ISP's on each of our pfsense routers and have the nexthop set to the WAN CARP IP?

    Here is my OPENBGP config:

    
    AS 65500
    fib-update yes
    listen on 0.0.0.0
    router-id 10.11.11.2
    network 172.22.0.0/24 # set nexthop 10.11.11.1
    network 66.150.104.96/27 # set nexthop 10.11.11.1
    
    group "WAN1" {
    remote-as 66510
    neighbor 10.11.11.5 {
    descr "WAN1 Neighbour" 
    remote-as 65510 
    set nexthop no-modify  
    local-address 10.11.11.2
    }
    }
    group "WAN2" {
    remote-as 66520
    neighbor 10.22.22.5 {
    descr "WAN2 Neighbour" 
    remote-as 65520 
    set nexthop no-modify  
    local-address 10.22.22.2
    }
    }
    
    match to group WAN1 inet set nexthop 10.11.11.1
    match to group WAN2 inet set nexthop 10.22.22.1
    
    deny from any
    deny to any
    allow from 10.11.11.5
    allow to 10.11.11.5
    allow from 10.22.22.5
    allow to 10.22.22.5
    
    

    It appears that pfsense is annoucing the correct nexthop info:

    $ bgpctl show ip bgp detail out neighbor 10.11.11.5
    
    BGP routing table entry for 66.150.104.96/27
        Nexthop 10.11.11.1 (via 10.11.11.1) from LOCAL (10.11.11.2)
        Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
        Last update: 00:01:13 ago
    
    BGP routing table entry for 172.22.0.0/24
        Nexthop 10.11.11.1 (via 10.11.11.1) from LOCAL (10.11.11.2)
        Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
        Last update: 00:01:13 ago
    

    and

    $ bgpctl show ip bgp detail out neighbor 10.22.22.5
    
    BGP routing table entry for 66.150.104.96/27
        Nexthop 10.22.22.1 (via 10.22.22.1) from LOCAL (10.11.11.2)
        Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
        Last update: 00:03:22 ago
    
    BGP routing table entry for 172.22.0.0/24
        Nexthop 10.22.22.1 (via 10.22.22.1) from LOCAL (10.11.11.2)
        Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
        Last update: 00:03:22 ago
    

    However the upstream router continue to use the pfsense WAN addresses 10.x.x.2 not the CARP 10.x.x.1 address

    The nexthops to CARP ips show as invalid.

    $ bgpctl show nexthop
    Flags: * = nexthop valid
    
      Nexthop         Route              Prio Gateway         Iface               
    * 10.11.11.1      10.11.11.1/32        48 connected       lo0 (UP, invalid)
    * 10.11.11.5      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
    * 10.22.22.1      10.22.22.1/32        48 connected       lo0 (UP, invalid)
    * 10.22.22.5      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)
    

    If i disable CARP and restart OpenBGP then the CARP nexthops show like this:

    $ bgpctl show nexthop
    Flags: * = nexthop valid
    
      Nexthop         Route              Prio Gateway         Iface               
    * 10.11.11.1      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
    * 10.11.11.5      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
    * 10.22.22.1      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)
    * 10.22.22.5      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)
    

    So it looks like OpenBGP doesn't "see" the CARP addresses as valid.


Log in to reply