OpenBGP Does not seem to be publishing routes from neighbor properly

[relfie]

Good afternoon,

I am a BGP newbie, so please forgive me if this is the wrong forum.

We are using pfSense 2.1.4 and OpenBGPD package 0.9.2.  We are trying to implement Amazon AWS direct connect.  I believe I have the bgpd.conf correct as I am seeing the following in routing.log, also bgpd status shows messages being exchanged.

=== snip routing.log ===
Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: startup
Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: rereading config
Aug  7 17:16:00 4slgbmernfw01 bgpd[5783]: route decision engine ready
Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: session engine ready
Aug  7 17:16:00 4slgbmernfw01 bgpd[5783]: RDE reconfigured
Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: listening on 192.168.55.1
Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: SE reconfigured
Aug  7 17:16:00 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change None -> Idle, reason: None
Aug  7 17:16:00 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.3 now valid: directly connected
Aug  7 17:16:01 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Idle -> Connect, reason: Start
Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Connect -> OpenSent, reason: Connection opened
Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenSent -> OpenConfirm, reason: OPEN message received
Aug  7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenConfirm -> Established, reason: KEEPALIVE message received
Aug  7 17:16:30 4slgbmernfw01 bgpd[5783]: Rib Loc-RIB: neighbor 192.168.55.5 (AWS-DC MER Peer) AS9059: update 172.16.24.0/21 via 192.168.55.5
Aug  7 17:16:30 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.5 now valid: via 192.168.55.1

However when a server on a local subnet in our AS tries to ping a server in the remote AS the traffic gets routed to the WAN interface and not over the BGP nexthop.

Here is our BGPD config :

=== snip ===

This file was created by the package manager.  Do not edit!

########

Our AS

########

AS 65458
fib-update yes
listen on 192.168.55.1
log updates
network 192.168.48.0/25 set nexthop 192.168.55.3
network 192.168.48.128/25 set nexthop 192.168.55.3
network 192.168.49.0/25 set nexthop 192.168.55.3

########

Peer Groups

########
group "AWSDC" {
remote-as 9059
neighbor 192.168.55.5 {
                                      descr "AWS-DC MER Peer"
                                      tcp md5sig password 8e484c715b2be0e50d576bc0bb0c29d4
                                      announce all 
                                      local-address 192.168.55.3
}
}
deny from any
deny to any
allow from 192.168.55.5
allow to 192.168.55.5

..here is the BGPD Status

Summary:
Neighbor                  AS    MsgRcvd    MsgSent  OutQ Up/Down  State/PrfRcvd
AWS-DC MER Peer          9059        13        12    0 00:04:09      1

Interfaces:
Interface      Nexthop state  Flags          Link state
opt6_vip249    ok            UP            CARP, master
igb2_vlan300  ok            UP            active, 1000 MBit/s
ovpns1        ok            UP            active
wan_vip250    ok            UP            CARP, master
opt4_vip251    ok            UP            CARP, master
opt3_vip252    ok            UP            CARP, master
opt2_vip253    ok            UP            CARP, master
opt1_vip254    ok            UP            CARP, master
wan_vip255    ok            UP            CARP, master
lagg0_vlan50  ok            UP            active, 10 MBit/s
lagg0_vlan30  ok            UP            active, 10 MBit/s
lagg0_vlan20  ok            UP            active, 10 MBit/s
lagg0_vlan10  ok            UP            active, 10 MBit/s
lagg0          ok            UP            Ethernet, active, 1000 MBit/s
pflog0        invalid                      invalid
lo0            ok            UP            invalid
pfsync0        ok            UP            invalid
enc0          ok            UP            invalid
igb7          ok            UP            active, 1000 MBit/s
igb6          ok            UP            Ethernet, active, 1000 MBit/s
igb5          ok            UP            active, 1000 MBit/s
igb4          invalid                      Ethernet, invalid, 10 MBit/s
igb3          ok            UP            active, 1000 MBit/s
igb2          ok            UP            Ethernet, active, 1000 MBit/s
igb1          ok            UP            active, 1000 MBit/s
igb0          ok            UP            Ethernet, active, 1000 MBit/s

Routing:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination          gateway          lpref  med aspath origin
>    172.16.24.0/21      192.168.55.5      100    0 9059 i
AI>  192.168.48.0/25      192.168.55.3      100    0 i
AI*>  192.168.48.128/25    192.168.55.3      100    0 i
AI*>  192.168.49.0/25      192.168.55.3      100    0 i

Forwarding:
flags: * = valid, B = BGP, C = Connected, S = Static
      N = BGP Nexthop reachable via this route
      r = reject route, b = blackhole route

flags prio destination          gateway
*S      48 0.0.0.0/0            81.27.95.81
*S      48 10.101.1.0/25        192.168.48.1
*S      48 10.101.1.128/25      192.168.48.129
*S      48 10.101.2.0/25        192.168.49.1
*S      48 10.101.5.0/25        192.168.48.1
*S      48 10.101.5.128/25      192.168.48.129
*S      48 10.101.6.0/25        192.168.49.1
*      48 81.27.95.80/28      81.27.95.84
*C      48 81.27.95.84/32      link#11
*C      48 81.27.95.93/32      link#23
*C      48 81.27.95.94/32      link#18
*C      48 84.20.199.91/32      link#1
*C      0 127.0.0.0/8          link#0
*C      48 127.0.0.1/32        link#11
*B      48 172.16.24.0/21      192.168.55.1
*S      48 192.168.44.0/23      192.168.48.1
*S      48 192.168.46.0/24      192.168.48.1
*C      48 192.168.48.0/25      link#14
*C      48 192.168.48.118/32    link#11
*C      48 192.168.48.126/32    link#19
*C      48 192.168.48.128/25    link#15
*C      48 192.168.48.246/32    link#11
*C      48 192.168.48.254/32    link#20
*C      48 192.168.49.0/25      link#16
*C      48 192.168.49.118/32    link#11
*C      48 192.168.49.126/32    link#21
*C      48 192.168.49.128/25    link#17
*C      48 192.168.49.246/32    link#11
*C      48 192.168.49.254/32    link#22
*S      48 192.168.50.0/24      192.168.48.1

• N    48 192.168.55.0/29      192.168.55.1
  *C      48 192.168.55.1/32      link#11
  *CN    48 192.168.55.3/32      link#26
  *S      48 192.168.90.0/24      192.168.48.1
  *S      48 192.168.200.0/24    192.168.200.2
  *C      48 192.168.200.1/32    link#11
  *C      48 192.168.200.2/32    link#24
  *C      48 192.168.226.0/27    link#7
  *C      48 192.168.226.2/32    link#11
  *C      0 ::1/128              link#0
  *C      48 ::1/128              link#11
  *C      48 fe80:1::/64          link#1
  *C      48 fe80:1::225:90ff:feea:3074/128 link#11
  *C      48 fe80:2::/64          link#2
  *C      48 fe80:2::225:90ff:feea:3075/128 link#11
  *C      48 fe80:3::/64          link#3
  *C      48 fe80:3::225:90ff:feea:3076/128 link#11
  *C      48 fe80:4::/64          link#4
  *C      48 fe80:4::225:90ff:feea:3077/128 link#11
  *C      48 fe80:6::/64          link#6
  *C      48 fe80:6::225:90ff:fef3:8fc7/128 link#11
  *C      48 fe80:7::/64          link#7
  *C      48 fe80:7::225:90ff:fef3:8fc8/128 link#11
  *C      48 fe80:8::/64          link#8
  *C      48 fe80:8::225:90ff:fef3:8fc9/128 link#11
  *C      48 fe80:/64          link#11
  *C      48 fe80:1/128        link#11
  *C      48 fe80:d::/64          link#13
  *C      48 fe80:d::225:90ff:feea:3075/128 link#11
  *C      48 fe80:e::/64          link#14
  *C      48 fe80:e::225:90ff:feea:3074/128 link#11
  *C      48 fe80:f::/64          link#15
  *C      48 fe80:f::225:90ff:feea:3074/128 link#11
  *C      48 fe80:10::/64        link#16
  *C      48 fe80:10::225:90ff:feea:3074/128 link#11
  *C      48 fe80:11::/64        link#17
  *C      48 fe80:11::225:90ff:feea:3074/128 link#11
  *C      48 fe80:18::225:90ff:feea:3074/128 link#11
  *C      48 fe80:19::/64        link#25
  *C      48 fe80:19::225:90ff:feea:3074/128 link#11
  *      48 ff01:1::/32          fe80:1::225:90ff:feea:3074
  *      48 ff01:2::/32          fe80:2::225:90ff:feea:3075
  *      48 ff01:3::/32          fe80:3::225:90ff:feea:3076
  *      48 ff01:4::/32          fe80:4::225:90ff:feea:3077
  *      48 ff01:6::/32          fe80:6::225:90ff:fef3:8fc7
  *      48 ff01:7::/32          fe80:7::225:90ff:fef3:8fc8
  *      48 ff01:8::/32          fe80:8::225:90ff:fef3:8fc9
  *      48 ff01:/32          ::1
  *      48 ff01:d::/32          fe80:d::225:90ff:feea:3075
  *      48 ff01:e::/32          fe80:e::225:90ff:feea:3074
  *      48 ff01:f::/32          fe80:f::225:90ff:feea:3074
  *      48 ff01:10::/32        fe80:10::225:90ff:feea:3074
  *      48 ff01:11::/32        fe80:11::225:90ff:feea:3074
  *      48 ff01:18::/32        fe80:18::225:90ff:feea:3074
  *      48 ff01:19::/32        fe80:19::225:90ff:feea:3074
  *      48 ff02:1::/32          fe80:1::225:90ff:feea:3074
  *      48 ff02:2::/32          fe80:2::225:90ff:feea:3075
  *      48 ff02:3::/32          fe80:3::225:90ff:feea:3076
  *      48 ff02:4::/32          fe80:4::225:90ff:feea:3077
  *      48 ff02:6::/32          fe80:6::225:90ff:fef3:8fc7
  *      48 ff02:7::/32          fe80:7::225:90ff:fef3:8fc8
  *      48 ff02:8::/32          fe80:8::225:90ff:fef3:8fc9
  *      48 ff02:/32          ::1
  *      48 ff02:d::/32          fe80:d::225:90ff:feea:3075
  *      48 ff02:e::/32          fe80:e::225:90ff:feea:3074
  *      48 ff02:f::/32          fe80:f::225:90ff:feea:3074
  *      48 ff02:10::/32        fe80:10::225:90ff:feea:3074
  *      48 ff02:11::/32        fe80:11::225:90ff:feea:3074
  *      48 ff02:18::/32        fe80:18::225:90ff:feea:3074
  *      48 ff02:19::/32        fe80:19::225:90ff:feea:3074

Network:
flags: S = Static
flags destination
*S      0 192.168.48.0/25      192.168.55.3
*S      0 192.168.48.128/25    192.168.55.3
*S      0 192.168.49.0/25      192.168.55.3

Nexthops:
Flags: * = nexthop valid

Nexthop        Route              Prio Gateway        Iface

• 192.168.55.3    192.168.55.3/32      48 connected      opt6_vip249 (UP, master)
• 192.168.55.5    192.168.55.0/29      48 192.168.55.1    igb2_vlan300 (UP, 1000 Mbps)

IP:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination          gateway          lpref  med aspath origin
>    172.16.24.0/21      192.168.55.5      100    0 9059 i
AI>  192.168.48.0/25      192.168.55.3      100    0 i
AI*>  192.168.48.128/25    192.168.55.3      100    0 i
AI*>  192.168.49.0/25      192.168.55.3      100    0 i

Neighbors:
BGP neighbor is 192.168.55.5, remote AS 9059
Description: AWS-DC MER Peer
  BGP version 4, remote router-id 192.168.55.5
  BGP state = Established, up for 00:04:09
  Last read 00:00:23, holdtime 90s, keepalive interval 30s
  Neighbor capabilities:
    Multiprotocol extensions: IPv4 unicast
    Route Refresh
    Graceful Restart
    4-byte AS numbers

Message statistics:
                  Sent      Received 
  Opens                    1          1
  Notifications            0          0
  Updates                  2          2
  Keepalives              9        10
  Route Refresh            0          0
  Total                  12        13

Update statistics:
                  Sent      Received 
  Updates                12          1
  Withdraws                0          0
  End-of-Rib              1          1

Local host:          192.168.55.1, Local port:    179
  Remote host:        192.168.55.5, Remote port: 59288

… and lastly here is the traceroute from the client server:
tracert 172.16.24.7

Tracing route to 172.16.24.7 over a maximum of 30 hops

1    <1 ms    <1 ms    <1 ms  192.168.48.118
  2    5 ms    2 ms    1 ms  81.27.95.83
  3    1 ms    1 ms    1 ms  109.104.114.134
  4    1 ms    1 ms    1 ms  betelgeuse-hardy.c4l.co.uk [109.104.114.105]
  5    1 ms    2 ms    70 ms  hardy-wolverine.c4l.co.uk [109.104.114.6]
  6    *        *        *    Request timed out.
  7    *        *        *    Request timed out.
  8    *        *        *    Request timed out.
  9    *        *        *    Request timed out.