OpenBGP Does not seem to be publishing routes from neighbor properly



  • Good afternoon,

    I am a BGP newbie, so please forgive me if this is the wrong forum.

    We are using pfSense 2.1.4 and OpenBGPD package 0.9.2.  We are trying to implement Amazon AWS direct connect.  I believe I have the bgpd.conf correct as I am seeing the following in routing.log, also bgpd status shows messages being exchanged.

    === snip routing.log ===
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: startup
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: rereading config
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5783]: route decision engine ready
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: session engine ready
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5783]: RDE reconfigured
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: listening on 192.168.55.1
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: SE reconfigured
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change None -> Idle, reason: None
    Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.3 now valid: directly connected
    Aug  7 17:16:01 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Idle -> Connect, reason: Start
    Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Connect -> OpenSent, reason: Connection opened
    Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenSent -> OpenConfirm, reason: OPEN message received
    Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenConfirm -> Established, reason: KEEPALIVE message received
    Aug  7 17:16:30 4slgbmernfw01 bgpd[5783]: Rib Loc-RIB: neighbor 192.168.55.5 (AWS-DC MER Peer) AS9059: update 172.16.24.0/21 via 192.168.55.5
    Aug  7 17:16:30 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.5 now valid: via 192.168.55.1

    However when a server on a local subnet in our AS tries to ping a server in the remote AS the traffic gets routed to the WAN interface and not over the BGP nexthop.

    Here is our BGPD config :

    === snip ===

    This file was created by the package manager.  Do not edit!

    ########

    Our AS

    ########

    AS 65458
    fib-update yes
    listen on 192.168.55.1
    log updates
    network 192.168.48.0/25 set nexthop 192.168.55.3
    network 192.168.48.128/25 set nexthop 192.168.55.3
    network 192.168.49.0/25 set nexthop 192.168.55.3

    ########

    Peer Groups

    ########
    group "AWSDC" {
    remote-as 9059
    neighbor 192.168.55.5 {
                                          descr "AWS-DC MER Peer"
                                          tcp md5sig password 8e484c715b2be0e50d576bc0bb0c29d4
                                          announce all 
                                          local-address 192.168.55.3
    }
    }
    deny from any
    deny to any
    allow from 192.168.55.5
    allow to 192.168.55.5

    ..here is the BGPD Status

    Summary:
    Neighbor                  AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
    AWS-DC MER Peer          9059        13        12    0 00:04:09      1

    Interfaces:
    Interface      Nexthop state  Flags          Link state
    opt6_vip249    ok            UP            CARP, master
    igb2_vlan300  ok            UP            active, 1000 MBit/s
    ovpns1        ok            UP            active
    wan_vip250    ok            UP            CARP, master
    opt4_vip251    ok            UP            CARP, master
    opt3_vip252    ok            UP            CARP, master
    opt2_vip253    ok            UP            CARP, master
    opt1_vip254    ok            UP            CARP, master
    wan_vip255    ok            UP            CARP, master
    lagg0_vlan50  ok            UP            active, 10 MBit/s
    lagg0_vlan30  ok            UP            active, 10 MBit/s
    lagg0_vlan20  ok            UP            active, 10 MBit/s
    lagg0_vlan10  ok            UP            active, 10 MBit/s
    lagg0          ok            UP            Ethernet, active, 1000 MBit/s
    pflog0        invalid                      invalid
    lo0            ok            UP            invalid
    pfsync0        ok            UP            invalid
    enc0          ok            UP            invalid
    igb7          ok            UP            active, 1000 MBit/s
    igb6          ok            UP            Ethernet, active, 1000 MBit/s
    igb5          ok            UP            active, 1000 MBit/s
    igb4          invalid                      Ethernet, invalid, 10 MBit/s
    igb3          ok            UP            active, 1000 MBit/s
    igb2          ok            UP            Ethernet, active, 1000 MBit/s
    igb1          ok            UP            active, 1000 MBit/s
    igb0          ok            UP            Ethernet, active, 1000 MBit/s

    Routing:
    flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
    origin: i = IGP, e = EGP, ? = Incomplete

    flags destination          gateway          lpref  med aspath origin
    >    172.16.24.0/21      192.168.55.5      100    0 9059 i
    AI
    >  192.168.48.0/25      192.168.55.3      100    0 i
    AI*>  192.168.48.128/25    192.168.55.3      100    0 i
    AI*>  192.168.49.0/25      192.168.55.3      100    0 i

    Forwarding:
    flags: * = valid, B = BGP, C = Connected, S = Static
          N = BGP Nexthop reachable via this route
          r = reject route, b = blackhole route

    flags prio destination          gateway
    *S      48 0.0.0.0/0            81.27.95.81
    *S      48 10.101.1.0/25        192.168.48.1
    *S      48 10.101.1.128/25      192.168.48.129
    *S      48 10.101.2.0/25        192.168.49.1
    *S      48 10.101.5.0/25        192.168.48.1
    *S      48 10.101.5.128/25      192.168.48.129
    *S      48 10.101.6.0/25        192.168.49.1
    *      48 81.27.95.80/28      81.27.95.84
    *C      48 81.27.95.84/32      link#11
    *C      48 81.27.95.93/32      link#23
    *C      48 81.27.95.94/32      link#18
    *C      48 84.20.199.91/32      link#1
    *C      0 127.0.0.0/8          link#0
    *C      48 127.0.0.1/32        link#11
    *B      48 172.16.24.0/21      192.168.55.1
    *S      48 192.168.44.0/23      192.168.48.1
    *S      48 192.168.46.0/24      192.168.48.1
    *C      48 192.168.48.0/25      link#14
    *C      48 192.168.48.118/32    link#11
    *C      48 192.168.48.126/32    link#19
    *C      48 192.168.48.128/25    link#15
    *C      48 192.168.48.246/32    link#11
    *C      48 192.168.48.254/32    link#20
    *C      48 192.168.49.0/25      link#16
    *C      48 192.168.49.118/32    link#11
    *C      48 192.168.49.126/32    link#21
    *C      48 192.168.49.128/25    link#17
    *C      48 192.168.49.246/32    link#11
    *C      48 192.168.49.254/32    link#22
    *S      48 192.168.50.0/24      192.168.48.1

    • N    48 192.168.55.0/29      192.168.55.1
      *C      48 192.168.55.1/32      link#11
      *CN    48 192.168.55.3/32      link#26
      *S      48 192.168.90.0/24      192.168.48.1
      *S      48 192.168.200.0/24    192.168.200.2
      *C      48 192.168.200.1/32    link#11
      *C      48 192.168.200.2/32    link#24
      *C      48 192.168.226.0/27    link#7
      *C      48 192.168.226.2/32    link#11
      *C      0 ::1/128              link#0
      *C      48 ::1/128              link#11
      *C      48 fe80:1::/64          link#1
      *C      48 fe80:1::225:90ff:feea:3074/128 link#11
      *C      48 fe80:2::/64          link#2
      *C      48 fe80:2::225:90ff:feea:3075/128 link#11
      *C      48 fe80:3::/64          link#3
      *C      48 fe80:3::225:90ff:feea:3076/128 link#11
      *C      48 fe80:4::/64          link#4
      *C      48 fe80:4::225:90ff:feea:3077/128 link#11
      *C      48 fe80:6::/64          link#6
      *C      48 fe80:6::225:90ff:fef3:8fc7/128 link#11
      *C      48 fe80:7::/64          link#7
      *C      48 fe80:7::225:90ff:fef3:8fc8/128 link#11
      *C      48 fe80:8::/64          link#8
      *C      48 fe80:8::225:90ff:fef3:8fc9/128 link#11
      *C      48 fe80🅱:/64          link#11
      *C      48 fe80🅱:1/128        link#11
      *C      48 fe80:d::/64          link#13
      *C      48 fe80:d::225:90ff:feea:3075/128 link#11
      *C      48 fe80:e::/64          link#14
      *C      48 fe80:e::225:90ff:feea:3074/128 link#11
      *C      48 fe80:f::/64          link#15
      *C      48 fe80:f::225:90ff:feea:3074/128 link#11
      *C      48 fe80:10::/64        link#16
      *C      48 fe80:10::225:90ff:feea:3074/128 link#11
      *C      48 fe80:11::/64        link#17
      *C      48 fe80:11::225:90ff:feea:3074/128 link#11
      *C      48 fe80:18::225:90ff:feea:3074/128 link#11
      *C      48 fe80:19::/64        link#25
      *C      48 fe80:19::225:90ff:feea:3074/128 link#11
      *      48 ff01:1::/32          fe80:1::225:90ff:feea:3074
      *      48 ff01:2::/32          fe80:2::225:90ff:feea:3075
      *      48 ff01:3::/32          fe80:3::225:90ff:feea:3076
      *      48 ff01:4::/32          fe80:4::225:90ff:feea:3077
      *      48 ff01:6::/32          fe80:6::225:90ff:fef3:8fc7
      *      48 ff01:7::/32          fe80:7::225:90ff:fef3:8fc8
      *      48 ff01:8::/32          fe80:8::225:90ff:fef3:8fc9
      *      48 ff01🅱:/32          ::1
      *      48 ff01:d::/32          fe80:d::225:90ff:feea:3075
      *      48 ff01:e::/32          fe80:e::225:90ff:feea:3074
      *      48 ff01:f::/32          fe80:f::225:90ff:feea:3074
      *      48 ff01:10::/32        fe80:10::225:90ff:feea:3074
      *      48 ff01:11::/32        fe80:11::225:90ff:feea:3074
      *      48 ff01:18::/32        fe80:18::225:90ff:feea:3074
      *      48 ff01:19::/32        fe80:19::225:90ff:feea:3074
      *      48 ff02:1::/32          fe80:1::225:90ff:feea:3074
      *      48 ff02:2::/32          fe80:2::225:90ff:feea:3075
      *      48 ff02:3::/32          fe80:3::225:90ff:feea:3076
      *      48 ff02:4::/32          fe80:4::225:90ff:feea:3077
      *      48 ff02:6::/32          fe80:6::225:90ff:fef3:8fc7
      *      48 ff02:7::/32          fe80:7::225:90ff:fef3:8fc8
      *      48 ff02:8::/32          fe80:8::225:90ff:fef3:8fc9
      *      48 ff02🅱:/32          ::1
      *      48 ff02:d::/32          fe80:d::225:90ff:feea:3075
      *      48 ff02:e::/32          fe80:e::225:90ff:feea:3074
      *      48 ff02:f::/32          fe80:f::225:90ff:feea:3074
      *      48 ff02:10::/32        fe80:10::225:90ff:feea:3074
      *      48 ff02:11::/32        fe80:11::225:90ff:feea:3074
      *      48 ff02:18::/32        fe80:18::225:90ff:feea:3074
      *      48 ff02:19::/32        fe80:19::225:90ff:feea:3074

    Network:
    flags: S = Static
    flags destination
    *S      0 192.168.48.0/25      192.168.55.3
    *S      0 192.168.48.128/25    192.168.55.3
    *S      0 192.168.49.0/25      192.168.55.3

    Nexthops:
    Flags: * = nexthop valid

    Nexthop        Route              Prio Gateway        Iface

    • 192.168.55.3    192.168.55.3/32      48 connected      opt6_vip249 (UP, master)
    • 192.168.55.5    192.168.55.0/29      48 192.168.55.1    igb2_vlan300 (UP, 1000 Mbps)

    IP:
    flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
    origin: i = IGP, e = EGP, ? = Incomplete

    flags destination          gateway          lpref  med aspath origin
    >    172.16.24.0/21      192.168.55.5      100    0 9059 i
    AI
    >  192.168.48.0/25      192.168.55.3      100    0 i
    AI*>  192.168.48.128/25    192.168.55.3      100    0 i
    AI*>  192.168.49.0/25      192.168.55.3      100    0 i

    Neighbors:
    BGP neighbor is 192.168.55.5, remote AS 9059
    Description: AWS-DC MER Peer
      BGP version 4, remote router-id 192.168.55.5
      BGP state = Established, up for 00:04:09
      Last read 00:00:23, holdtime 90s, keepalive interval 30s
      Neighbor capabilities:
        Multiprotocol extensions: IPv4 unicast
        Route Refresh
        Graceful Restart
        4-byte AS numbers

    Message statistics:
                      Sent      Received 
      Opens                    1          1
      Notifications            0          0
      Updates                  2          2
      Keepalives              9        10
      Route Refresh            0          0
      Total                  12        13

    Update statistics:
                      Sent      Received 
      Updates                12          1
      Withdraws                0          0
      End-of-Rib              1          1

    Local host:          192.168.55.1, Local port:    179
      Remote host:        192.168.55.5, Remote port: 59288

    … and lastly here is the traceroute from the client server:
    tracert 172.16.24.7

    Tracing route to 172.16.24.7 over a maximum of 30 hops

    1    <1 ms    <1 ms    <1 ms  192.168.48.118
      2    5 ms    2 ms    1 ms  81.27.95.83
      3    1 ms    1 ms    1 ms  109.104.114.134
      4    1 ms    1 ms    1 ms  betelgeuse-hardy.c4l.co.uk [109.104.114.105]
      5    1 ms    2 ms    70 ms  hardy-wolverine.c4l.co.uk [109.104.114.6]
      6    *        *        *    Request timed out.
      7    *        *        *    Request timed out.
      8    *        *        *    Request timed out.
      9    *        *        *    Request timed out.


Log in to reply