OPENBGP with CARP, nexthop<carp ip=""></carp>

[acherman]

Hey everyone, I am trying to finalize my config for adding BGP to peer with our two upstream providers (Shaw and Telus).  Here is what our basic connections look like:

As is discussed in multiple places (like this tread: http://forum.pfsense.org/index.php/topic,46032.0.html, I have 2 routers that currently use CARP to connect to both providers with a shared IP address, and I want to create a BGP session from each router to each provider in order to allow the failover times to stay on par with CARP's capabilities.  I'm stuck in trying to figure out exactly how to do this.  There is reference to using the an attribute set line like network xx.xx.125.0/24 set nexthop xx.xx.127.250 to set the nexthop the provider uses to the CARP IP, but I'm confused on how this works for both providers.  Do I add a line for each CARP address like this:

network xx.xx.125.0/24 set nexthop xx.xx.127.250
network xx.xx.125.0/24 set nexthop xx.xx.241.4

I'm just not sure how that will be interpreted at each provider.

Any help is appreciated.  Thanks in advance.

Aaron

[jimp]

IIRC, the BGP state can't sync between the two nodes, so you're better off getting separate feeds for each wan on each box (4 feeds total) rather than trying to do it with two.

Otherwise when CARP fails you have to wait for BGP to completely rebuild its relationship(s) with the ISPs.

[acherman]

Hey Jim, that's exactly what I have - one feed from each WAN provider to each router, 4 in total.  So, I want to create a BGP session from each router to each provider, 4 sessions in total, so that if a failure occurs we don't have to wait for BGP to establish a session.  But, I need to figure out how to tell each provider to forward packets (nexthop) to my carp addresses instead of the real interface addresses.  I assume this is done via the network set nexthop as noted, but I don't understand how it works with both providers when there is a different carp IP for each.  It seems like the providers would get confused if I advertise the same network twice, with a different nexthop IP in each.

If it's not possible to do this with OpenBGP and CARP (which I have assumed it is, there is some info out there, but nothing specific to this detail), is it likely that I can tell my providers to statically enter my CARP addresses as their nexthop?

[jnex26]

Sounds like your having the same problems as me.

in Theory you should be able to have a bgp session set up on the primary and secondary router.

In the case of a failure of the primary wan then a carp failover to the secondary firewall would give you the a much faster response time as the secondary firewall should already have the session "Setup" but idle, however carpdev is not yet implemented into freebsd so we can't use the demote feature which quite frankly sucks.

in your situation I would probably use "depend on" to allow BGP to follow carp failover it does mean that the case of a router failure your looking at longer failover times.

alternatively only have one session per router then allow CARP to manage the failover, this allows for fast failover but if carp cannot detect a physical interface failure (connection plugged though a switch) then you will need something to manage the failover of carp.

[Reiner030]

Hi

the setup is eays… when you find out how it woks... it had costs me severald long days and nights :(

=> OpenBGP has a very unlogical way to create the routing tables  and announcing metric and nexthops (and guessable other things, too).

=> I have realized it this way (adopted to your needs): => it can be only configured "raw":
Important is always "the order" of filter rules (and the combination^^)
And don't forget to "mesh" the peers...

as-own              = 12345
as-peer1           = 123
as-peer2           = 124

local_ip_gw1    = xx.xx.127.xx
local_ip_gw1_v6 = 
local_gw1       = xx.xx.127.250
peer_ip_gw1     = xx.xx.127.xx
peer_ip_gw1_v6  =   

local_ip_gw2    = xx.xx.241.xx
local_ip_gw2_v6 = 
local_gw2       = xx.xx.241.4
peer_ip_gw2     = xx.xx.241.xx
peer_ip_gw2_v6  =  

AS         $as-own
fib-update yes
router-id  $local_ip_gw1   # or what you want

group "Peer1" {
        remote-as  $as-peer1
        holdtime   20 
        # no-modify needed in combination with filter
        # rule to set CARP IP address as nexthop :-(
        set        nexthop  no-modify
        neighbor $peer1_gw {
                descr "GW_Peer1"
                local-address $local_ip_gw1
        }
        neighbor $peer1_gw_v6 {
                descr "GW_Peer1_v6"
                local-address $local_ip_gw1_v6
        }
}

group "Peer2" {
        remote-as  $as-peer2
        holdtime   20 
        # no-modify needed in combination with filter
        # rule to set CARP IP address as nexthop :-(
        set        nexthop  no-modify
        neighbor $peer2_gw {
                descr "GW_Peer2"
                local-address $local_ip_gw2
        }
        neighbor $peer2_gw_v6 {
                descr "GW_Peer2_v6"
                local-address $local_ip_gw2_v6
        }
}

group "internal" {
        remote-as  $as-own
        holdtime   6
        neighbor $peer_ip_gw1 {
                descr "internal neighbor GW1"
                local-address $local_ip_gw1
        }
        neighbor $peer_ip_gw1_v6 {
                descr "internal neighbor GW1 v6"
                local-address $local_ip_gw1_v6
        }
        neighbor $peer_ip_gw2 {
                descr "internal neighbor GW2"
                local-address $local_ip_gw2
        }
        neighbor $peer_ip_gw2_v6 {
                descr "internal neighbor GW2 v6"
                local-address $local_ip_gw2_v6
        }
}

match to   group Peer1     inet    set { nexthop $local_gw1         }
match to   group Peer1     inet6  set { nexthop $local_gw1_v6    }

match to   group Peer2     inet    set { nexthop $local_gw2         }
match to   group Peer2     inet6  set { nexthop $local_gw2_v6    }

# do not forget deny/allow rules

Its important to set 2 filter rules… why ever... one who set the nexthop and the neighbor/peer based "no-modify". ::)

Not needed for this task but to document it for other cases:
Also there is a problem if you wand metric based routing.
The metric is only accepted if the localpref is lowered:

match to    group Peer2       set { metric 10 localpref -10 }

Because I split my AS onto 2 buildings I need iBGP rules…
When I try to use this filter to announce it to my iBGP peers there was the correct metric sent and set in my internal peer, but the external peer got all routes with metric 0 :(
Till I found out that I must use an internal filter on my internal peer side instead of an outgoing filter on the source side:

match from group internal    set { metric 10 localpref -10 }

And you must add an empty line after the last rule that OpenBGPd is happy with the config file  ???

Bests

Reiner

[IcePick]

I have been trying to get Reiner030's solution to work.

I have been working on it in the lab so I have access to the upstream routers.

Problem I am running into is that the next-hop IP is only set on routes learned from the CARP Backup cluster member.
The cluster node with the CARP IP Masters sends its route to the upstream using its bgp peering IP.

I can duplicate by disabling CARP on the master and then restarting the openbgpd on both nodes. The only routes with the CARP vIP set as next-hop set will be from the node with CARP disabled.

Any suggestions?

[Reiner030]

@IcePick:

Problem I am running into is that the next-hop IP is only set on routes learned from the CARP Backup cluster member.
The cluster node with the CARP IP Masters sends its route to the upstream using its bgp peering IP.

you used also this line?

match to   group Peer1     inet    set { nexthop $local_gw1         }

You can see what your OpenBGP would announces:

[2.1-BETA1][root@gw1.jws1.local]/root(31): bgpctl show ip bgp detail out neighbor PEERNAME-OR-IP

BGP routing table entry for xx.xx.xx.0/21
    Nexthop xx.xx.9.122 (via xx.xx.9.122) from LOCAL (xx.xx.9.125)
    Origin IGP, metric 0, localpref 100, weight 0, internal, valid, best, announced
    Last update: 5d21h15m ago

Tthis is my direct announcement of my master/slave… my IBPG peer announcements are not listed in this overview ("from LOCAL").

Bests

[IcePick]

@Reiner030:

you used also this line?

match to   group Peer1     inet    set { nexthop $local_gw1         }

yes using that line. It works perfect on the carp backup member, and if i disable carp on the master and restart bgpd it works there too.

Also found that if the peering is ibgp it works fine, only broken on the CARP master when doing ebgp peering.

[IcePick]

After making no headway with the ebgp/carp master issues we stopped trying to set the next hop to the carp IP in the announcement from the pfsense cluster.
We are now setting the next hop with a filter on the upstream router.

[Reiner030]

@IcePick:

After making no headway with the ebgp/carp master issues we stopped trying to set the next hop to the carp IP in the announcement from the pfsense cluster.
We are now setting the next hop with a filter on the upstream router.

yes, was our  first solution here, too till I found out why it happened on my side.

Problem was that even the read in output didn't helped much to understand why it won't work the "logical" way:

bgpd -v -n -f /var/etc/openbgpd/bgpd.conf

I guess there is a very special order of filtering rules but they are not officially explained (or I haven't them found)…

But Setting HOP on peer side should be good enough ;)

Bests