Openbgpd not coupling RIB with FIB
Recently we've been looking into an issue where our RRD graphs have been out of step with our traffic graphs for our multiple WANs. As it turns out, it looks like the route table is telling an outgoing packet being routed from an internal host to go one way, but then a reply-to rule and a state for the packet kick in and the packet is routed out the correct interface – but not before statistics have been counted for both WANs. We're still trying to determine if this is truly the cause, but in the process of doing so, we've come to realize that OpenBGPD is not adding the routes it has downloaded from its peers to the fib (aka the kernel's routing table). We've confirmed this by setting up a developmental pfSense VM as another peer, receiving routes from the live pfSense firewall we have running, and it's experiencing the exact same problem despite only having one interface at all. Even manually running bgpctl fib couple (on the dev) doesn't do anything. Worse, the bgpd daemon doesn't even seem to be emitting any log messages about failing to add those routes into the fib.
Has anyone run into this before? This wasn't a problem in the bgpd package included with pfSense 1.2.3 (package version 0.4.2 – current (according to 2.0 package manager) is 0.5.2)... what am I missing?
This was actually due to the nexthop received from the peer not being our proper nexthop; we had to update the config to have openbgpd set the nexthop to be our actual next hop. Moral of the story: nexthop has to be locally reachable, somehow.
That is to say, you must have a route that covers the IP given in the "Gateway" column of bgpctl show rib; if you do not, you must either add such a route or add this line:
set nexthop <gateway ip="" for="" isp="">to the neighbor config for the peer that's sending you BGP routes. (This automatically translates to a "match" filter rule that simply does the same thing, but I find this looks simpler and does not appear to be deprecated.)</gateway>