OpenBGP and Carp

newfirewallman

Running BGP and happy, but thinking i want to add a redundant pfsense box also. Can someone explain this farther "You can use the OpenBGPD package in combination with CARP. The key is to use "local-address <carp address="">" and "depend-on carpX". This way the backup firewall's bgpd only starts connecting when CARP has failed over (when the CARP interface becomes active)." From http://doc.pfsense.org/index.php/OpenBGPD_package

Is this the best way? I have a pfsense router with BGP between 2 ISP's Nat is turned off on it. Just want to add redundant pfsense box. Is CARP the right answer of IBGP(openospf)</carp>

jnorell

I've not tried that setup, but it sounds like that's having OpenBGP bind to the CARP ip address. Doing that would require the BGP session to reset upon failover when the carp backup took over the CARP ip address - again not having tried it, I would expect that to be very disruptive. If you try it, I'd be curious as to outage times on failover.

I setup OpenBGP + CARP not too long ago and took a different approach. I setup bgp sessions between the upstream router(s) and both pfsense boxes (carp master and backup) on their non-CARP addresses. I then advertised my routes from OpenBGP with set nexthop x.x.x.x where x.x.x.x is the CARP IP address. Failover is sub-second, exactly the same as CARP failover without BGP involved. (Bringing the carp master back online was consistently a 28 second outage, for what it's worth, which I decided to just live with .. I don't know why, or if it can be improved).

I'll note you do NOT want the set nexthop X parameter under the Neighbors tab - you want to specify it right on the Networks line under the Settings tab. Eg. instead of adding 1.2.3.0/24 as the network, specify 1.2.3.0/24 set nexthop 2.3.4.5 (where 2.3.4.5 is your CARP failover addr). Then your Raw config would show a line like:

network 1.2.3.0/24 set nexthop 2.3.4.5

cmb

I forgot that page had that note. There is a better option, check http://doc.pfsense.org/index.php/OpenBGPD_package again, I updated it.

jnorell

cmb: that page needs corrected. It has:

set the nexthop in your BGP neighbor configuration to a CARP IP on the interconnect subnet

but that's the wrong setting, it should to indicate the use of set nexthop x.x.x.x on the network statement, not in the neighbor config.

FWIW, OpenBGP's set nexthop in the neighbor statement actually changes the NEXTHOP attribute on routes learned from that neighbor, not routes sent to the neighbor. So if you set it to the CARP IP, all routes learned from that BGP peer will no longer have the usual gateway (eg. the peer router's address), they will have the CARP IP as their gateway - it can completely break your routing.

cmb

Thanks, I just quickly threw that in from memory and put it in wrong, fixed.

ISCGDave

Does anyone have an example of what this IP configuration would look like with the upstream provider? Just curious how to create multiple bgp sessions with the provider, would the bgp neighbor session be the same IP (CARP IP) for both firewalls or would you have to obtain a larger subnet on the upstream link (usually they do a /30).

For instance (fw1 IP) 1.1.1.1, (fw2 IP) 1.1.1.2, (CARP IP) 1.1.1.3, (provider IP) 1.1.1.4 and the BGP sessions would be with 1,2 and 4 and the set nexthop statement to 4?

jnorell

@ISCGDave:

Does anyone have an example of what this IP configuration would look like with the upstream provider? Just curious how to create multiple bgp sessions with the provider, would the bgp neighbor session be the same IP (CARP IP) for both firewalls or would you have to obtain a larger subnet on the upstream link (usually they do a /30).

Yes, you have to have a larger subnet, CARP doesn't work with a /30 (irrespective of whether or not you're using BGP).

@ISCGDave:

For instance (fw1 IP) 1.1.1.1, (fw2 IP) 1.1.1.2, (CARP IP) 1.1.1.3, (provider IP) 1.1.1.4 and the BGP sessions would be with 1,2 and 4 and the set nexthop statement to 4?

Pretty close; in your example, you'd have a bgp session between 1<->4 and 2<->4, but you set the nexthop to 3 in the routes sent to 4.

acherman

Hey guys, I just want to say thanks for this info. I am just starting the process to set this up with Telus and Shaw in Canada. Running pfSense 2.0.1 release and OpenBGP 0.5.6 so I am hoping it all goes smoothly. I assume there is no way in the GUI to set that nexthop address and it has to be done in the console?

Aaron

acherman

I know this is a dead thread, but my topic is the exact same as this. If the mods think I should start a new thread let me know.

I am hoping to get a call from my primary ISP today to set up and test our BGP (and CARP). I am just going through my OpenBGP config and want to make sure I have things set up right before then.

I want to use the method mentioned above, whereby I create 2 BGP sessions with my upstream providers, one from each CAPR interface. I only have one subnet to advertise. I understand using the Networks field of the Services tab to enter my advertised network and add the parameter set nexthop afterwards, but I don't understand how this works for both providers. Essentially, since I will have a nexthop address for each provider, do I make the entry for the advertised subnet twice, but use the CARP from both providers?

Here is my raw config:

AS aaa
fib-update yes
network z.z.125.0/24 set nexthop x.x.127.250
network z.z.125.0/24 set nexthop y.y.241.4
group "Telus" {
	remote-as Y
		neighbor y.y.241.3 {
    	 descr "Telus WAN"
		tcp md5sig password xxx
}
}
group "Shaw" {
	remote-as X
		neighbor x.x.127.249 {
    	 descr "Shaw WAN"
		tcp md5sig password xxx
}
}
deny from any
deny to any
allow from y.y.241.3
allow to y.y.241.3
allow from x.x.127.249
allow to x.x.127.249

Does that look right? I'm just not sure if I understand advertising the routes and the nexthop to each providers properly.

Thanks.

Metu69salemi

I admit, that i know nothing about openbgp, but does openbgp work with deny,allow order?

If not, then you might have problem with your rulesets

acherman

Thanks for the reply. That's another thing all together. From the OpenBGPD package info - the filter rules are evaluated in sequential order, from first to last. But, in the Raw config tab the top line of the config says # This file was created by the package manager. Do not edit! And I'm fairly certain those rules were created for me. I might delete all config entries and start from scratch to see if those come up. I'm mostly concerned about my advertisements right now and making sure everything goes to the right places.

Aaron