OpenBGP with CARP in 2.2.4 and two ISPs

gmathers

We want to switch to a multihome setup with our DC ISP. To do this, we've been working on a pfsense setup with 2 pfsense VM's running OpenBGP and with CARP addresses on both the WAN and LAN side. I'm now trying to setup BGP sessions with both upstream routers and have them use our WAN CARP address for the nexthop like https://forum.pfsense.org/index.php?topic=57614.0. I've tried the setup suggested by Reiner030 but have had the same issues as IcePick had in that thread. OpenBGP

Is it possible to have BGP sessions with both ISP's on each of our pfsense routers and have the nexthop set to the WAN CARP IP?

Here is my OPENBGP config:


AS 65500
fib-update yes
listen on 0.0.0.0
router-id 10.11.11.2
network 172.22.0.0/24 # set nexthop 10.11.11.1
network 66.150.104.96/27 # set nexthop 10.11.11.1

group "WAN1" {
remote-as 66510
neighbor 10.11.11.5 {
descr "WAN1 Neighbour" 
remote-as 65510 
set nexthop no-modify  
local-address 10.11.11.2
}
}
group "WAN2" {
remote-as 66520
neighbor 10.22.22.5 {
descr "WAN2 Neighbour" 
remote-as 65520 
set nexthop no-modify  
local-address 10.22.22.2
}
}

match to group WAN1 inet set nexthop 10.11.11.1
match to group WAN2 inet set nexthop 10.22.22.1

deny from any
deny to any
allow from 10.11.11.5
allow to 10.11.11.5
allow from 10.22.22.5
allow to 10.22.22.5

It appears that pfsense is annoucing the correct nexthop info:

$ bgpctl show ip bgp detail out neighbor 10.11.11.5

BGP routing table entry for 66.150.104.96/27
    Nexthop 10.11.11.1 (via 10.11.11.1) from LOCAL (10.11.11.2)
    Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
    Last update: 00:01:13 ago

BGP routing table entry for 172.22.0.0/24
    Nexthop 10.11.11.1 (via 10.11.11.1) from LOCAL (10.11.11.2)
    Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
    Last update: 00:01:13 ago

and

$ bgpctl show ip bgp detail out neighbor 10.22.22.5

BGP routing table entry for 66.150.104.96/27
    Nexthop 10.22.22.1 (via 10.22.22.1) from LOCAL (10.11.11.2)
    Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
    Last update: 00:03:22 ago

BGP routing table entry for 172.22.0.0/24
    Nexthop 10.22.22.1 (via 10.22.22.1) from LOCAL (10.11.11.2)
    Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
    Last update: 00:03:22 ago

However the upstream router continue to use the pfsense WAN addresses 10.x.x.2 not the CARP 10.x.x.1 address

The nexthops to CARP ips show as invalid.

$ bgpctl show nexthop
Flags: * = nexthop valid

  Nexthop         Route              Prio Gateway         Iface               
* 10.11.11.1      10.11.11.1/32        48 connected       lo0 (UP, invalid)
* 10.11.11.5      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
* 10.22.22.1      10.22.22.1/32        48 connected       lo0 (UP, invalid)
* 10.22.22.5      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)

If i disable CARP and restart OpenBGP then the CARP nexthops show like this:

$ bgpctl show nexthop
Flags: * = nexthop valid

  Nexthop         Route              Prio Gateway         Iface               
* 10.11.11.1      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
* 10.11.11.5      10.11.11.0/29        48 connected       xn0 (UP, 10 Mbps)
* 10.22.22.1      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)
* 10.22.22.5      10.22.22.0/29        48 connected       xn4 (UP, 10 Mbps)

So it looks like OpenBGP doesn't "see" the CARP addresses as valid.