OpenBGP not switching to other ISP in multihomed setup

[CharlesA987]

Hi Everyone,

I am fairly new to BGP and am running into an issue. Does anyone have any ideas? Here is a basic sanitized diagram of my setup, a raw config, results from basic bgpctl commands, and the IRR record.

I have a dual multi homed setup with 2 pfSense firewalls in failover mode using CARP. It has OpenBGP announcing our AS333 and block 3.3.3.0/24. It is working and we can hit systems on our block and outgoing shows the bgp ip as expected. I am trying to get it to be redundant in case one of the ISPs goes down.

To simulate I took turns disconnecting each ISP:
If I disconnect ISP1 I can still reach the our systems and they can still get out.
If I disconnect ISP2 I I can no longer hit our systems and they can no longer get out.

I have the default gateway in pfsense to be automatic and it does switch over to the other provider once it detects one of the ISPs down. I also have static routes configured for each neighbor to ensure it goes out the correct ISP for bgp updates.

One of the things that seems off to me is under the 'bgpctl show rib' the route to the neighbor for ISP1 doesn't have > next to it so it makes it seem like its not active? The other thing I am unsure of is in that same route section it lists an origin AS 4444 that I have no idea where that comes from. I never entered that anywhere.

bgpctl show summary:

Neighbor                   AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
ISP2 BGP                222       3150       3149     0 1d02h13m      1
ISP1 BGP                111       3150       3150     0 1d02h13m      1

bgpctl show neighbor:

BGP neighbor is 2.2.2.193, remote AS 222
 Description: ISP2 BGP
  BGP version 4, remote router-id 1.2.3.121
  BGP state = Established, up for 1d02h14m
  Last read 00:00:22, holdtime 90s, keepalive interval 30s
  Neighbor capabilities:
    Multiprotocol extensions: IPv4 unicast
    Route Refresh
    4-byte AS numbers

  Message statistics:
                  Sent       Received  
  Opens                    1          1
  Notifications            0          0
  Updates                  1          2
  Keepalives            3148       3148
  Route Refresh            0          0
  Total                 3150       3151

  Update statistics:
                  Sent       Received  
  Updates                  4          1
  Withdraws                0          0
  End-of-Rib               0          1

  Local host:          2.2.2.204, Local port:  38342
  Remote host:         2.2.2.193, Remote port:   179

BGP neighbor is 1.1.1.170, remote AS 111, Multihop (30)
 Description: ISP1 BGP
  BGP version 4, remote router-id 1.1.1.1.170
  BGP state = Established, up for 1d02h14m
  Last read 00:00:28, holdtime 90s, keepalive interval 30s
  Neighbor capabilities:
    Multiprotocol extensions: IPv4 unicast
    Route Refresh
    Graceful Restart: Timeout: 120, IPv4 unicast
    4-byte AS numbers

  Message statistics:
                  Sent       Received  
  Opens                    1          1
  Notifications            0          0
  Updates                  2          2
  Keepalives            3148       3148
  Route Refresh            0          0
  Total                 3151       3151

  Update statistics:
                  Sent       Received  
  Updates                  4          1
  Withdraws                0          0
  End-of-Rib               1          1

  Local host:           1.1.1.147, Local port:  15993
  Remote host:        1.1.1.170, Remote port:   179

bgpctl show nexthop:

Flags: * = nexthop valid

  Nexthop         Route              Prio Gateway         Iface               
* 2.2.2.193    2.2.2.193/32            48 2.2.2.193       igb2 (UP, 1000 Mbps)
* 1.1.1.170    1.1.1.170/32            48 1.1.1.129       igb3 (UP, 1000 Mbps)

bgpctl show rib:

flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination          gateway          lpref   med aspath origin
*>    0.0.0.0/0            2.2.2.193          100     0 222 i
*     0.0.0.0/0            1.1.1.170          100     0 111 4444 i
AI*>  3.3.3.0/24           0.0.0.0            100     0 i

raw config:

AS 333
fib-update yes
holdtime 90
listen on 0.0.0.0
router-id 3.3.3.149
network 3.3.3.0/24
group "ISP1" {
    remote-as 111
    neighbor 1.1.1.129 {
        descr "ISP1 BGP"
        announce self  
        multihop 30
        local-address 0.0.0.0
}
}
group "ISP2" {
    remote-as 222
    neighbor 2.2.2.193 {
        descr "ISP2 BGP"
        announce self  
        local-address 0.0.0.0
}
}
deny from any
deny to any
allow from 1.1.1.170
allow to 1.1.1.170
allow from 2.2.2.193
allow to 2.2.2.193

IRR:

import: from AS111 accept ANY
import: from AS222 accept ANY
export: to AS111 announce AS333
export: to AS222 announce AS333

[CharlesA987]

Just a follow up, this is resolved. It was actually an issue with the provider that was not working. They had to update a MPLS record on their side. Once they did that, it worked as expected and fails over automatically.