Redistributing BGP route over OSPF

tman904

Could it be that the metric of the ipsec routes on the off-site router is causing packets to use the wrong route? Or is the ipsec interface the only route out of that network?

Are you using the passive interface to suppress the OSPF route updates being sent to the off-site's LAN? If not that maybe part of the issue. As passive interfaces don't send routing updates.

Check to see if the pfsense router that's in ASN 65000 has a route back to the off-site's LAN.

Elegant

IPsec interface is the only route of that network, everything happens over the VTI interface and routes are shared using OSPF for dynamic routing.

Every interface with the exception if ipsec1000 is set to passive to suppress route updates.

The on-site pfSense router with ASN 65000 is able to reach the off-site's LAN without issue.

Must be related to how the on-site router is reaching the BGP route. Traceroute is not providing any information for 10.0.30.15 and the address cannot be ping'd, only arping'd. This is despite the entry existing in the ARP table for on-site router...

tman904

Unless the off-site router is in the cloud It may be possible that the underlay network helping to connect the IPsec VPN connection. Is interfering with the sending of packets.

Does the traceroute just timeout or does it stop at a certain hop? Some devices won't send time-exceeded messages, but it's still helpful to run. On the off-site pfSense go to Diagnostics->Packet Capture on the LAN interface. Then ping from a client PC off-site. That should be able to tell us if the packets are even reaching your pfSense.

Even if they show up in the capture many things could still prevent traffic from flowing. You also might want to try running the capture for all packet types on either the WAN or LAN to see what traffic is getting where.

On the off-site check under Diagnostics->Routes to see which route, the BGP one or otherwise. Is being used to route packets destined to the on-site network. Also check that on the on-site as well.

Elegant

Worked on repairing ping and traceroute functionality. Neither router is in the cloud. The underlying network used for the VTI may be interfering but this should be possible to overcome similar to this example.

For the on-site router, traceroute immediately determines the correct route for 10.0.30.15 and ends. For the off-site router, nothing happens, just times out. Both routers are able to ping 10.0.30.15 without issue now. Per your suggestion, I ran a Packet Capture on the off-site router and ping'd from a client off-site, no issues:

01:01:32.361651 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63366, length 40
01:01:32.528067 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63366, length 40
01:01:32.637977 IP 10.2.10.106.53156 > 10.0.30.15.443: tcp 1
01:01:32.804140 IP 10.0.30.15.443 > 10.2.10.106.53156: tcp 0
01:01:33.375298 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63367, length 40
01:01:33.542460 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63367, length 40
01:01:34.388038 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63368, length 40
01:01:34.554422 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63368, length 40
01:01:35.401860 IP 10.2.10.10 > 10.0.30.15: ICMP echo request, id 2, seq 63369, length 40
01:01:35.568636 IP 10.0.30.15 > 10.2.10.10: ICMP echo reply, id 2, seq 63369, length 40

Per Diagnostics -> Routes, the off-site router is using the ipsec1000 interface which is expected since it is the VTI to the on-site router.

Glad to see some normalcy now for the ping and traceroutes. Perhaps it would be easier to establish BGP between both routers for that route instead of redistributing through OSPF? The only issue there is the RIDs won't be too nice given the 10.1.100.0/30 subnet for the VTI interface and the on-site router's RID is 10.0.30.1.

Thanks!

Elegant

I've learned that the traceroute failing appears to be normal over IPsec VTI (no IP address works). New things I've noticed, curl from the off-site network returns the appropriate response but accessing the router through a web browser fails with a timeout.

Looks like the route is complete but the issue is somewhere else? OpenVPN clients are not encountering this issue, only those connected to the off-site router where IPsec is used...

tman904

@Elegant said in Redistributing BGP route over OSPF:

accessing the router through a web browser fails with a timeout.

Do you have the web gui port accessible in your firewall rules? If you're accessing the web gui from an interface other than the default LAN interface, the anti lockout rules for administration won't have an effect. Leading to the packets being dropped.

Good to hear the routing and packet captures came back ok. But it's still bitter sweet since we still haven't discovered the root of the problem.

I apologize for suggesting running a traceroute over an ipsec vti interface. I actually wasn't aware that it wouldn't work. Thus leading us deeper down a dead end.

After looking closer at the packet capture I'm wondering about something that maybe should be checked. You said the BGP neighbership has formed correctly? What about the OSPF adjacency that you're redistributing into?

Try running a packet capture for tcp port 179 and ospf ip protocol number 89.

tman904

What do the clients connected to the off-site show? As far as pinging/traceroutes/routing tables.

Or have you isolated the issue to the off-site ipsec at this point? It's odd that openvpn isn't having issues even though ipsec is. That makes me think it could be more related to the ipsec tunnel configuration.

Is it possible just as a test to convert the ipsec configuration into an openvpn one to see if that illuminates any issues with the ipsec setup? Make sure to take a configuration backup on pfsense first.

tman904

@Elegant said in Redistributing BGP route over OSPF:

I ran a Packet Capture on the off-site router and ping'd from a client off-site, no issues:

If the client connected to the router can use the route properly but the router itself can't that may point to the router's configuration. I don't think pfsense treats packets sour ced from itself any different with regards to the routing table than it would non-locally sourced ones.

The only way that could be an issue is with vrfs but I don't believe pfsense supports those.

Elegant

We managed to resolve the issue, turned out to be related to the network plugin that was offering 10.0.30.15/32. Even though we were not using one of its features, having the feature enabled by default was disrupting everything else. We have opened an issue with that project to make them aware. In the interim, we just disabled the unneeded feature. Thanks!

tman904

This post is deleted!

tman904

@Elegant Good to hear you got it sorted out. Which pfsense plugin was it frr or openbgpd?