OpenBGP Does not seem to be publishing routes from neighbor properly
-
Good afternoon,
I am a BGP newbie, so please forgive me if this is the wrong forum.
We are using pfSense 2.1.4 and OpenBGPD package 0.9.2. We are trying to implement Amazon AWS direct connect. I believe I have the bgpd.conf correct as I am seeing the following in routing.log, also bgpd status shows messages being exchanged.
=== snip routing.log ===
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: startup
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: rereading config
Aug 7 17:16:00 4slgbmernfw01 bgpd[5783]: route decision engine ready
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: session engine ready
Aug 7 17:16:00 4slgbmernfw01 bgpd[5783]: RDE reconfigured
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: listening on 192.168.55.1
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: SE reconfigured
Aug 7 17:16:00 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change None -> Idle, reason: None
Aug 7 17:16:00 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.3 now valid: directly connected
Aug 7 17:16:01 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Idle -> Connect, reason: Start
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change Connect -> OpenSent, reason: Connection opened
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenSent -> OpenConfirm, reason: OPEN message received
Aug 7 17:16:30 4slgbmernfw01 bgpd[5790]: neighbor 192.168.55.5 (AWS-DC MER Peer): state change OpenConfirm -> Established, reason: KEEPALIVE message received
Aug 7 17:16:30 4slgbmernfw01 bgpd[5783]: Rib Loc-RIB: neighbor 192.168.55.5 (AWS-DC MER Peer) AS9059: update 172.16.24.0/21 via 192.168.55.5
Aug 7 17:16:30 4slgbmernfw01 bgpd[5653]: nexthop 192.168.55.5 now valid: via 192.168.55.1However when a server on a local subnet in our AS tries to ping a server in the remote AS the traffic gets routed to the WAN interface and not over the BGP nexthop.
Here is our BGPD config :
=== snip ===
This file was created by the package manager. Do not edit!
########
Our AS
########
AS 65458
fib-update yes
listen on 192.168.55.1
log updates
network 192.168.48.0/25 set nexthop 192.168.55.3
network 192.168.48.128/25 set nexthop 192.168.55.3
network 192.168.49.0/25 set nexthop 192.168.55.3########
Peer Groups
########
group "AWSDC" {
remote-as 9059
neighbor 192.168.55.5 {
descr "AWS-DC MER Peer"
tcp md5sig password 8e484c715b2be0e50d576bc0bb0c29d4
announce all
local-address 192.168.55.3
}
}
deny from any
deny to any
allow from 192.168.55.5
allow to 192.168.55.5..here is the BGPD Status
Summary:
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd
AWS-DC MER Peer 9059 13 12 0 00:04:09 1Interfaces:
Interface Nexthop state Flags Link state
opt6_vip249 ok UP CARP, master
igb2_vlan300 ok UP active, 1000 MBit/s
ovpns1 ok UP active
wan_vip250 ok UP CARP, master
opt4_vip251 ok UP CARP, master
opt3_vip252 ok UP CARP, master
opt2_vip253 ok UP CARP, master
opt1_vip254 ok UP CARP, master
wan_vip255 ok UP CARP, master
lagg0_vlan50 ok UP active, 10 MBit/s
lagg0_vlan30 ok UP active, 10 MBit/s
lagg0_vlan20 ok UP active, 10 MBit/s
lagg0_vlan10 ok UP active, 10 MBit/s
lagg0 ok UP Ethernet, active, 1000 MBit/s
pflog0 invalid invalid
lo0 ok UP invalid
pfsync0 ok UP invalid
enc0 ok UP invalid
igb7 ok UP active, 1000 MBit/s
igb6 ok UP Ethernet, active, 1000 MBit/s
igb5 ok UP active, 1000 MBit/s
igb4 invalid Ethernet, invalid, 10 MBit/s
igb3 ok UP active, 1000 MBit/s
igb2 ok UP Ethernet, active, 1000 MBit/s
igb1 ok UP active, 1000 MBit/s
igb0 ok UP Ethernet, active, 1000 MBit/sRouting:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incompleteflags destination gateway lpref med aspath origin
> 172.16.24.0/21 192.168.55.5 100 0 9059 i
AI> 192.168.48.0/25 192.168.55.3 100 0 i
AI*> 192.168.48.128/25 192.168.55.3 100 0 i
AI*> 192.168.49.0/25 192.168.55.3 100 0 iForwarding:
flags: * = valid, B = BGP, C = Connected, S = Static
N = BGP Nexthop reachable via this route
r = reject route, b = blackhole routeflags prio destination gateway
*S 48 0.0.0.0/0 81.27.95.81
*S 48 10.101.1.0/25 192.168.48.1
*S 48 10.101.1.128/25 192.168.48.129
*S 48 10.101.2.0/25 192.168.49.1
*S 48 10.101.5.0/25 192.168.48.1
*S 48 10.101.5.128/25 192.168.48.129
*S 48 10.101.6.0/25 192.168.49.1
* 48 81.27.95.80/28 81.27.95.84
*C 48 81.27.95.84/32 link#11
*C 48 81.27.95.93/32 link#23
*C 48 81.27.95.94/32 link#18
*C 48 84.20.199.91/32 link#1
*C 0 127.0.0.0/8 link#0
*C 48 127.0.0.1/32 link#11
*B 48 172.16.24.0/21 192.168.55.1
*S 48 192.168.44.0/23 192.168.48.1
*S 48 192.168.46.0/24 192.168.48.1
*C 48 192.168.48.0/25 link#14
*C 48 192.168.48.118/32 link#11
*C 48 192.168.48.126/32 link#19
*C 48 192.168.48.128/25 link#15
*C 48 192.168.48.246/32 link#11
*C 48 192.168.48.254/32 link#20
*C 48 192.168.49.0/25 link#16
*C 48 192.168.49.118/32 link#11
*C 48 192.168.49.126/32 link#21
*C 48 192.168.49.128/25 link#17
*C 48 192.168.49.246/32 link#11
*C 48 192.168.49.254/32 link#22
*S 48 192.168.50.0/24 192.168.48.1- N 48 192.168.55.0/29 192.168.55.1
*C 48 192.168.55.1/32 link#11
*CN 48 192.168.55.3/32 link#26
*S 48 192.168.90.0/24 192.168.48.1
*S 48 192.168.200.0/24 192.168.200.2
*C 48 192.168.200.1/32 link#11
*C 48 192.168.200.2/32 link#24
*C 48 192.168.226.0/27 link#7
*C 48 192.168.226.2/32 link#11
*C 0 ::1/128 link#0
*C 48 ::1/128 link#11
*C 48 fe80:1::/64 link#1
*C 48 fe80:1::225:90ff:feea:3074/128 link#11
*C 48 fe80:2::/64 link#2
*C 48 fe80:2::225:90ff:feea:3075/128 link#11
*C 48 fe80:3::/64 link#3
*C 48 fe80:3::225:90ff:feea:3076/128 link#11
*C 48 fe80:4::/64 link#4
*C 48 fe80:4::225:90ff:feea:3077/128 link#11
*C 48 fe80:6::/64 link#6
*C 48 fe80:6::225:90ff:fef3:8fc7/128 link#11
*C 48 fe80:7::/64 link#7
*C 48 fe80:7::225:90ff:fef3:8fc8/128 link#11
*C 48 fe80:8::/64 link#8
*C 48 fe80:8::225:90ff:fef3:8fc9/128 link#11
*C 48 fe80:/64 link#11
*C 48 fe80:1/128 link#11
*C 48 fe80:d::/64 link#13
*C 48 fe80:d::225:90ff:feea:3075/128 link#11
*C 48 fe80:e::/64 link#14
*C 48 fe80:e::225:90ff:feea:3074/128 link#11
*C 48 fe80:f::/64 link#15
*C 48 fe80:f::225:90ff:feea:3074/128 link#11
*C 48 fe80:10::/64 link#16
*C 48 fe80:10::225:90ff:feea:3074/128 link#11
*C 48 fe80:11::/64 link#17
*C 48 fe80:11::225:90ff:feea:3074/128 link#11
*C 48 fe80:18::225:90ff:feea:3074/128 link#11
*C 48 fe80:19::/64 link#25
*C 48 fe80:19::225:90ff:feea:3074/128 link#11
* 48 ff01:1::/32 fe80:1::225:90ff:feea:3074
* 48 ff01:2::/32 fe80:2::225:90ff:feea:3075
* 48 ff01:3::/32 fe80:3::225:90ff:feea:3076
* 48 ff01:4::/32 fe80:4::225:90ff:feea:3077
* 48 ff01:6::/32 fe80:6::225:90ff:fef3:8fc7
* 48 ff01:7::/32 fe80:7::225:90ff:fef3:8fc8
* 48 ff01:8::/32 fe80:8::225:90ff:fef3:8fc9
* 48 ff01:/32 ::1
* 48 ff01:d::/32 fe80:d::225:90ff:feea:3075
* 48 ff01:e::/32 fe80:e::225:90ff:feea:3074
* 48 ff01:f::/32 fe80:f::225:90ff:feea:3074
* 48 ff01:10::/32 fe80:10::225:90ff:feea:3074
* 48 ff01:11::/32 fe80:11::225:90ff:feea:3074
* 48 ff01:18::/32 fe80:18::225:90ff:feea:3074
* 48 ff01:19::/32 fe80:19::225:90ff:feea:3074
* 48 ff02:1::/32 fe80:1::225:90ff:feea:3074
* 48 ff02:2::/32 fe80:2::225:90ff:feea:3075
* 48 ff02:3::/32 fe80:3::225:90ff:feea:3076
* 48 ff02:4::/32 fe80:4::225:90ff:feea:3077
* 48 ff02:6::/32 fe80:6::225:90ff:fef3:8fc7
* 48 ff02:7::/32 fe80:7::225:90ff:fef3:8fc8
* 48 ff02:8::/32 fe80:8::225:90ff:fef3:8fc9
* 48 ff02:/32 ::1
* 48 ff02:d::/32 fe80:d::225:90ff:feea:3075
* 48 ff02:e::/32 fe80:e::225:90ff:feea:3074
* 48 ff02:f::/32 fe80:f::225:90ff:feea:3074
* 48 ff02:10::/32 fe80:10::225:90ff:feea:3074
* 48 ff02:11::/32 fe80:11::225:90ff:feea:3074
* 48 ff02:18::/32 fe80:18::225:90ff:feea:3074
* 48 ff02:19::/32 fe80:19::225:90ff:feea:3074
Network:
flags: S = Static
flags destination
*S 0 192.168.48.0/25 192.168.55.3
*S 0 192.168.48.128/25 192.168.55.3
*S 0 192.168.49.0/25 192.168.55.3Nexthops:
Flags: * = nexthop validNexthop Route Prio Gateway Iface
- 192.168.55.3 192.168.55.3/32 48 connected opt6_vip249 (UP, master)
- 192.168.55.5 192.168.55.0/29 48 192.168.55.1 igb2_vlan300 (UP, 1000 Mbps)
IP:
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incompleteflags destination gateway lpref med aspath origin
> 172.16.24.0/21 192.168.55.5 100 0 9059 i
AI> 192.168.48.0/25 192.168.55.3 100 0 i
AI*> 192.168.48.128/25 192.168.55.3 100 0 i
AI*> 192.168.49.0/25 192.168.55.3 100 0 iNeighbors:
BGP neighbor is 192.168.55.5, remote AS 9059
Description: AWS-DC MER Peer
BGP version 4, remote router-id 192.168.55.5
BGP state = Established, up for 00:04:09
Last read 00:00:23, holdtime 90s, keepalive interval 30s
Neighbor capabilities:
Multiprotocol extensions: IPv4 unicast
Route Refresh
Graceful Restart
4-byte AS numbersMessage statistics:
Sent Received
Opens 1 1
Notifications 0 0
Updates 2 2
Keepalives 9 10
Route Refresh 0 0
Total 12 13Update statistics:
Sent Received
Updates 12 1
Withdraws 0 0
End-of-Rib 1 1Local host: 192.168.55.1, Local port: 179
Remote host: 192.168.55.5, Remote port: 59288… and lastly here is the traceroute from the client server:
tracert 172.16.24.7Tracing route to 172.16.24.7 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.48.118
2 5 ms 2 ms 1 ms 81.27.95.83
3 1 ms 1 ms 1 ms 109.104.114.134
4 1 ms 1 ms 1 ms betelgeuse-hardy.c4l.co.uk [109.104.114.105]
5 1 ms 2 ms 70 ms hardy-wolverine.c4l.co.uk [109.104.114.6]
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.
9 * * * Request timed out. - N 48 192.168.55.0/29 192.168.55.1