OpenBGP not switching to other ISP in multihomed setup



  • Hi Everyone,

    I am fairly new to BGP and am running into an issue. Does anyone have any ideas? Here is a basic sanitized diagram of my setup, a raw config, results from basic bgpctl commands, and the IRR record.

    I have a dual multi homed setup with 2 pfSense firewalls in failover mode using CARP. It has OpenBGP announcing our AS333 and block 3.3.3.0/24. It is working and we can hit systems on our block and outgoing shows the bgp ip as expected. I am trying to get it to be redundant in case one of the ISPs goes down.

    To simulate I took turns disconnecting each ISP:
    If I disconnect ISP1 I can still reach the our systems and they can still get out.
    If I disconnect ISP2 I I can no longer hit our systems and they can no longer get out.

    I have the default gateway in pfsense to be automatic and it does switch over to the other provider once it detects one of the ISPs down. I also have static routes configured for each neighbor to ensure it goes out the correct ISP for bgp updates.

    One of the things that seems off to me is under the 'bgpctl show rib' the route to the neighbor for ISP1 doesn't have > next to it so it makes it seem like its not active? The other thing I am unsure of is in that same route section it lists an origin AS 4444 that I have no idea where that comes from. I never entered that anywhere.

    alt text

    bgpctl show summary:

    Neighbor                   AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
    ISP2 BGP                222       3150       3149     0 1d02h13m      1
    ISP1 BGP                111       3150       3150     0 1d02h13m      1
    

    bgpctl show neighbor:

    BGP neighbor is 2.2.2.193, remote AS 222
     Description: ISP2 BGP
      BGP version 4, remote router-id 1.2.3.121
      BGP state = Established, up for 1d02h14m
      Last read 00:00:22, holdtime 90s, keepalive interval 30s
      Neighbor capabilities:
        Multiprotocol extensions: IPv4 unicast
        Route Refresh
        4-byte AS numbers
    
      Message statistics:
                      Sent       Received  
      Opens                    1          1
      Notifications            0          0
      Updates                  1          2
      Keepalives            3148       3148
      Route Refresh            0          0
      Total                 3150       3151
    
      Update statistics:
                      Sent       Received  
      Updates                  4          1
      Withdraws                0          0
      End-of-Rib               0          1
    
      Local host:          2.2.2.204, Local port:  38342
      Remote host:         2.2.2.193, Remote port:   179
    
    BGP neighbor is 1.1.1.170, remote AS 111, Multihop (30)
     Description: ISP1 BGP
      BGP version 4, remote router-id 1.1.1.1.170
      BGP state = Established, up for 1d02h14m
      Last read 00:00:28, holdtime 90s, keepalive interval 30s
      Neighbor capabilities:
        Multiprotocol extensions: IPv4 unicast
        Route Refresh
        Graceful Restart: Timeout: 120, IPv4 unicast
        4-byte AS numbers
    
      Message statistics:
                      Sent       Received  
      Opens                    1          1
      Notifications            0          0
      Updates                  2          2
      Keepalives            3148       3148
      Route Refresh            0          0
      Total                 3151       3151
    
      Update statistics:
                      Sent       Received  
      Updates                  4          1
      Withdraws                0          0
      End-of-Rib               1          1
    
      Local host:           1.1.1.147, Local port:  15993
      Remote host:        1.1.1.170, Remote port:   179
    

    bgpctl show nexthop:

    Flags: * = nexthop valid
    
      Nexthop         Route              Prio Gateway         Iface               
    * 2.2.2.193    2.2.2.193/32            48 2.2.2.193       igb2 (UP, 1000 Mbps)
    * 1.1.1.170    1.1.1.170/32            48 1.1.1.129       igb3 (UP, 1000 Mbps)
    

    bgpctl show rib:

    flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
    origin: i = IGP, e = EGP, ? = Incomplete
    
    flags destination          gateway          lpref   med aspath origin
    *>    0.0.0.0/0            2.2.2.193          100     0 222 i
    *     0.0.0.0/0            1.1.1.170          100     0 111 4444 i
    AI*>  3.3.3.0/24           0.0.0.0            100     0 i
    

    raw config:

    AS 333
    fib-update yes
    holdtime 90
    listen on 0.0.0.0
    router-id 3.3.3.149
    network 3.3.3.0/24
    group "ISP1" {
        remote-as 111
        neighbor 1.1.1.129 {
            descr "ISP1 BGP"
            announce self  
            multihop 30
            local-address 0.0.0.0
    }
    }
    group "ISP2" {
        remote-as 222
        neighbor 2.2.2.193 {
            descr "ISP2 BGP"
            announce self  
            local-address 0.0.0.0
    }
    }
    deny from any
    deny to any
    allow from 1.1.1.170
    allow to 1.1.1.170
    allow from 2.2.2.193
    allow to 2.2.2.193
    

    IRR:

    import: from AS111 accept ANY
    import: from AS222 accept ANY
    export: to AS111 announce AS333
    export: to AS222 announce AS333



  • Just a follow up, this is resolved. It was actually an issue with the provider that was not working. They had to update a MPLS record on their side. Once they did that, it worked as expected and fails over automatically.


Log in to reply