OpenBGP not switching to other ISP in multihomed setup
-
Hi Everyone,
I am fairly new to BGP and am running into an issue. Does anyone have any ideas? Here is a basic sanitized diagram of my setup, a raw config, results from basic bgpctl commands, and the IRR record.
I have a dual multi homed setup with 2 pfSense firewalls in failover mode using CARP. It has OpenBGP announcing our AS333 and block 3.3.3.0/24. It is working and we can hit systems on our block and outgoing shows the bgp ip as expected. I am trying to get it to be redundant in case one of the ISPs goes down.
To simulate I took turns disconnecting each ISP:
If I disconnect ISP1 I can still reach the our systems and they can still get out.
If I disconnect ISP2 I I can no longer hit our systems and they can no longer get out.I have the default gateway in pfsense to be automatic and it does switch over to the other provider once it detects one of the ISPs down. I also have static routes configured for each neighbor to ensure it goes out the correct ISP for bgp updates.
One of the things that seems off to me is under the 'bgpctl show rib' the route to the neighbor for ISP1 doesn't have > next to it so it makes it seem like its not active? The other thing I am unsure of is in that same route section it lists an origin AS 4444 that I have no idea where that comes from. I never entered that anywhere.
bgpctl show summary:
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd ISP2 BGP 222 3150 3149 0 1d02h13m 1 ISP1 BGP 111 3150 3150 0 1d02h13m 1bgpctl show neighbor:
BGP neighbor is 2.2.2.193, remote AS 222 Description: ISP2 BGP BGP version 4, remote router-id 1.2.3.121 BGP state = Established, up for 1d02h14m Last read 00:00:22, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast Route Refresh 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 1 2 Keepalives 3148 3148 Route Refresh 0 0 Total 3150 3151 Update statistics: Sent Received Updates 4 1 Withdraws 0 0 End-of-Rib 0 1 Local host: 2.2.2.204, Local port: 38342 Remote host: 2.2.2.193, Remote port: 179 BGP neighbor is 1.1.1.170, remote AS 111, Multihop (30) Description: ISP1 BGP BGP version 4, remote router-id 1.1.1.1.170 BGP state = Established, up for 1d02h14m Last read 00:00:28, holdtime 90s, keepalive interval 30s Neighbor capabilities: Multiprotocol extensions: IPv4 unicast Route Refresh Graceful Restart: Timeout: 120, IPv4 unicast 4-byte AS numbers Message statistics: Sent Received Opens 1 1 Notifications 0 0 Updates 2 2 Keepalives 3148 3148 Route Refresh 0 0 Total 3151 3151 Update statistics: Sent Received Updates 4 1 Withdraws 0 0 End-of-Rib 1 1 Local host: 1.1.1.147, Local port: 15993 Remote host: 1.1.1.170, Remote port: 179bgpctl show nexthop:
Flags: * = nexthop valid Nexthop Route Prio Gateway Iface * 2.2.2.193 2.2.2.193/32 48 2.2.2.193 igb2 (UP, 1000 Mbps) * 1.1.1.170 1.1.1.170/32 48 1.1.1.129 igb3 (UP, 1000 Mbps)bgpctl show rib:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale origin: i = IGP, e = EGP, ? = Incomplete flags destination gateway lpref med aspath origin *> 0.0.0.0/0 2.2.2.193 100 0 222 i * 0.0.0.0/0 1.1.1.170 100 0 111 4444 i AI*> 3.3.3.0/24 0.0.0.0 100 0 iraw config:
AS 333 fib-update yes holdtime 90 listen on 0.0.0.0 router-id 3.3.3.149 network 3.3.3.0/24 group "ISP1" { remote-as 111 neighbor 1.1.1.129 { descr "ISP1 BGP" announce self multihop 30 local-address 0.0.0.0 } } group "ISP2" { remote-as 222 neighbor 2.2.2.193 { descr "ISP2 BGP" announce self local-address 0.0.0.0 } } deny from any deny to any allow from 1.1.1.170 allow to 1.1.1.170 allow from 2.2.2.193 allow to 2.2.2.193IRR:
import: from AS111 accept ANY
import: from AS222 accept ANY
export: to AS111 announce AS333
export: to AS222 announce AS333 -
Just a follow up, this is resolved. It was actually an issue with the provider that was not working. They had to update a MPLS record on their side. Once they did that, it worked as expected and fails over automatically.