Redistributing BGP route over OSPF

Elegant

Good evening,

We're seeking to distribute one our 10.0.30.15/32 BGP route from our on-site network to our off-site network using OSPF. We've implemented the appropriate prefix-list and route-map in FRR per the below:

interface ipsec1000
  ip ospf area 0.0.0.0
interface vmx1.30
  ip ospf area 0.0.0.0

router ospf
  ospf router-id 10.1.100.1
  redistribute bgp route-map RMAP
  ospf abr-type cisco
  passive-interface vmx1.30

# Prefix Lists
ip prefix-list PLIST seq 1 permit 10.0.30.15/32 
ip prefix-list Traefik description PLIST

# Route Maps
route-map RMAP permit 10
  match ip address prefix-list PLIST

Our off-site network successfully picks up the new route through OSPF but we are unable communicate with it (can't arping over IPsec interface either). We also have an a passive interface which carries the 10.0.30.0/24 subnet from the on-site to off-site router which is working (firewall rules are allowing traffic over IPsec interfaces, no issues).

# On-Site pfSense Router
B>* 10.0.30.15/32 [20/0] via 10.0.30.10, vmx1.30, 00:14:01

# Off-Site pfSense Router
O>* 10.0.30.0/24 [110/20] via 10.1.100.1, ipsec1000 onlink, 00:24:26
O>* 10.0.30.15/32 [110/20] via 10.1.100.1, ipsec1000 onlink, 00:24:24

Below is a diagram of the network flow, what changes do we need to implement so that the off-site router can properly communicate with the BGP route distributed from the on-site router over OSPF? It seems like the current config should work.

BGP - OSPF.png

Thanks!

tman904

Hi,

Do either routers have any ACLs applied that may be blocking the packets? Also what does a traceroute show?

Just to verify the router with RID 10.0.30.10 is sending packets to the off-site router with RID 10.1.100.2 correct?

Try running the traceroute from the on-site to the router off-site and vice-versa.

A good test might also be:
ping from RID 10.0.30.10->ping from 10.0.30.1->RID 10.1.100.2.

Then reverse the pings in the opposite direction. Just because they reach one router doesn't mean something isn't interfering with them returning. ACL or route wise.

Does the off-site router have the appropriate routes to send packets over the ipsec interface?

Elegant

No ACLs created at this time. Traceroutes for 10.0.30.15 return nothing on either router but the route is working for clients on-site.

The neighbor with address 10.0.30.10 is sending off packets to the on-site router with RID 10.0.30.1. That same on-site router is then sending off the routes using OSPF to the off-site router using RID 10.1.100.1. The 10.1.100.0/30 subnet is used for the IPsec VTI interfaces which is why the RID is different from the same router. The on-site router makes use of two different RIDs as a result.

No issues reaching/pinging 10.1.100.2 (off-site) from 10.0.30.1 (on-site) and vice-versa.

The off-site router should have the appropriate routes as we are able to communicate with the neighbor (10.0.30.10) over SSH. At least, this is our understanding since the route does appear on the off-site router:

O>* 10.0.30.0/24 [110/20] via 10.1.100.1, ipsec1000 onlink, 00:24:26

Updated the diagram to reflect the neighbor address of 10.0.30.10 instead of RID so that it is cleaner. Sorry about that.

BGP - OSPF.png

We know that IPsec VTI supports BGP and OSPF but there seems to be something amiss...

Thanks!

tman904

Could it be that the metric of the ipsec routes on the off-site router is causing packets to use the wrong route? Or is the ipsec interface the only route out of that network?

Are you using the passive interface to suppress the OSPF route updates being sent to the off-site's LAN? If not that maybe part of the issue. As passive interfaces don't send routing updates.

Check to see if the pfsense router that's in ASN 65000 has a route back to the off-site's LAN.

Elegant

IPsec interface is the only route of that network, everything happens over the VTI interface and routes are shared using OSPF for dynamic routing.

Every interface with the exception if ipsec1000 is set to passive to suppress route updates.

The on-site pfSense router with ASN 65000 is able to reach the off-site's LAN without issue.

Must be related to how the on-site router is reaching the BGP route. Traceroute is not providing any information for 10.0.30.15 and the address cannot be ping'd, only arping'd. This is despite the entry existing in the ARP table for on-site router...

tman904

Unless the off-site router is in the cloud It may be possible that the underlay network helping to connect the IPsec VPN connection. Is interfering with the sending of packets.

Does the traceroute just timeout or does it stop at a certain hop? Some devices won't send time-exceeded messages, but it's still helpful to run. On the off-site pfSense go to Diagnostics->Packet Capture on the LAN interface. Then ping from a client PC off-site. That should be able to tell us if the packets are even reaching your pfSense.

Even if they show up in the capture many things could still prevent traffic from flowing. You also might want to try running the capture for all packet types on either the WAN or LAN to see what traffic is getting where.

On the off-site check under Diagnostics->Routes to see which route, the BGP one or otherwise. Is being used to route packets destined to the on-site network. Also check that on the on-site as well.

Elegant

Worked on repairing ping and traceroute functionality. Neither router is in the cloud. The underlying network used for the VTI may be interfering but this should be possible to overcome similar to this example.

For the on-site router, traceroute immediately determines the correct route for 10.0.30.15 and ends. For the off-site router, nothing happens, just times out. Both routers are able to ping 10.0.30.15 without issue now. Per your suggestion, I ran a Packet Capture on the off-site router and ping'd from a client off-site, no issues:

01:01:32.361651 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63366, length 40
01:01:32.528067 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63366, length 40
01:01:32.637977 IP 10.2.10.106.53156 > 10.0.30.15.443: tcp 1
01:01:32.804140 IP 10.0.30.15.443 > 10.2.10.106.53156: tcp 0
01:01:33.375298 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63367, length 40
01:01:33.542460 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63367, length 40
01:01:34.388038 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63368, length 40
01:01:34.554422 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63368, length 40
01:01:35.401860 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63369, length 40
01:01:35.568636 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63369, length 40

Per Diagnostics -> Routes, the off-site router is using the ipsec1000 interface which is expected since it is the VTI to the on-site router.

Glad to see some normalcy now for the ping and traceroutes. Perhaps it would be easier to establish BGP between both routers for that route instead of redistributing through OSPF? The only issue there is the RIDs won't be too nice given the 10.1.100.0/30 subnet for the VTI interface and the on-site router's RID is 10.0.30.1.

Thanks!

Elegant

I've learned that the traceroute failing appears to be normal over IPsec VTI (no IP address works). New things I've noticed, curl from the off-site network returns the appropriate response but accessing the router through a web browser fails with a timeout.

Looks like the route is complete but the issue is somewhere else? OpenVPN clients are not encountering this issue, only those connected to the off-site router where IPsec is used...

tman904

@Elegant said in Redistributing BGP route over OSPF:

accessing the router through a web browser fails with a timeout.

Do you have the web gui port accessible in your firewall rules? If you're accessing the web gui from an interface other than the default LAN interface, the anti lockout rules for administration won't have an effect. Leading to the packets being dropped.

Good to hear the routing and packet captures came back ok. But it's still bitter sweet since we still haven't discovered the root of the problem.

I apologize for suggesting running a traceroute over an ipsec vti interface. I actually wasn't aware that it wouldn't work. Thus leading us deeper down a dead end.

After looking closer at the packet capture I'm wondering about something that maybe should be checked. You said the BGP neighbership has formed correctly? What about the OSPF adjacency that you're redistributing into?

Try running a packet capture for tcp port 179 and ospf ip protocol number 89.

tman904

What do the clients connected to the off-site show? As far as pinging/traceroutes/routing tables.

Or have you isolated the issue to the off-site ipsec at this point? It's odd that openvpn isn't having issues even though ipsec is. That makes me think it could be more related to the ipsec tunnel configuration.

Is it possible just as a test to convert the ipsec configuration into an openvpn one to see if that illuminates any issues with the ipsec setup? Make sure to take a configuration backup on pfsense first.

tman904

@Elegant said in Redistributing BGP route over OSPF:

I ran a Packet Capture on the off-site router and ping'd from a client off-site, no issues:

If the client connected to the router can use the route properly but the router itself can't that may point to the router's configuration. I don't think pfsense treats packets sour ced from itself any different with regards to the routing table than it would non-locally sourced ones.

The only way that could be an issue is with vrfs but I don't believe pfsense supports those.

Elegant

We managed to resolve the issue, turned out to be related to the network plugin that was offering 10.0.30.15/32. Even though we were not using one of its features, having the feature enabled by default was disrupting everything else. We have opened an issue with that project to make them aware. In the interim, we just disabled the unneeded feature. Thanks!

tman904

This post is deleted!

tman904

@Elegant Good to hear you got it sorted out. Which pfsense plugin was it frr or openbgpd?