FRR BGP With Multiple Neighbors - Routing Issue
-
I have a NetGate 1541 configured to connect to 2 different BGP neighbors. Routes are showing from both and all indicators are that routing and connecting to either connection should work. Traffic across the first connection works as expected. When I added the the second connection it all looks good but traffic that is expected to go over that connection is not.
Each neighbor is setup to be assigned to the correct interface and with the correct IP. As I said, routes and BGP information is shown in status.
Is there anything that needs configured to allow traffic to cross the second BGP connection in a scenario where there are multiple neighbors?
Just for reference, I followed the NetGate docs on FRR and created a generic route map as described here https://docs.netgate.com/pfsense/en/latest/packages/frr/bgp/example.html. It’s the only area that I can think of that might need some config but not sure.
Any ideas?
-
One thing I forgot to mention is that on the first neighbor, I can see ARP entries and the routes are showing up on the neighbor side, so the far end network is getting my routes, as well as my device is seeing the BGP routes from the far side.
The second neighbor shows routes on my device BUT when I look for routes on the neighbor side on the far end, there is nothing. So my routes are not being sent to the second BGP neighbor.
I have checked the config many times and both neighbors look correct but only the first neighbor is sending/receiving traffic.
What could cause this?
-
@djtech2k said in FRR BGP With Multiple Neighbors - Routing Issue:
The second neighbor shows routes on my device BUT when I look for routes on the neighbor side on the far end, there is nothing. So my routes are not being sent to the second BGP neighbor.
Are you advertising routes out to that neighbor?
You can validate this by looking at the BGP status in the GUI or by going into the CLI of pfsense, typing in 'vtysh' and issuing the command 'show ip bgp summary' -
@michmoor I am a novice so please bear with me.
We have (2) /29 networks that are specified in the "networks to Distribute" area in the UI. Is that what you are referring to? If so, when I look at the far end of the 1st neighbor, I see the CIDR blocks showing in the route table. On the 2nd neighbor, there are no routes showing on the far end route table.
The gear is in a lab that I do not have hands-on right now so unfortunately, I cannot get that data right now. Yesterday when I was hands-on, I did check the BGP status in the UI. I saw lots of BGP routes in the status. Some had the BGP neighbor IP of the 1st neighbor and some routes had the 2nd BGP neighbor listed. So it tells me my Netgate is getting BGP routes from both neighbors. Both neighbors were listed in the summary and both had the far end router IP's and such listed.
The big difference seems to be that I cannot seem to pass any traffic over the 2nd connection/neighbor. When I run tests like tracert from the Netgate, the results look different when I choose the 1st neighbor vs the 2nd neighbor. It makes me think its a routing issue but I don't know for sure. I have thought it could maybe be something with the route map or something to do with the fact that the config is allowing the 1st connection to route but for some reason the 2nd will not.
-
Anyone have any ideas how to get traffic going over the 2nd connection and routes pushed to the 2nd neighbor far end?
-
Here is the bgp summary info. It looks like everything is ok to me. It shows both neighbors as expected.
I am looking in the logs for FRR but not seeing anything so far.
Any ideas?
-
@djtech2k said in FRR BGP With Multiple Neighbors - Routing Issue:
like everything is ok to me. It
Ok, if you are not routing out the other gateway its because BGP doesn't see it as a valid gateway.
You can either use local-pref or as-path to manipulate routing. as-path would be my technique of choice unless you have another bgp peering within your AS that's sharing those routes.If you are trying to use a dual WAN gateway set up then you can use Gateway groups to achieve that as well.
Considering the received-routes count isn't the same from each upstream gateway you would have reachability problems (i would imagine). Is that by design?
-
@michmoor Thanks. Gonna do my best to answer.
Essentially I have (2) /29's that are used for connecting to 2 public clouds via BGP. So when I say connection 1 is working, consider it cloud 1. My ASN covers both of my /29's.
Cloud 2 is the one that is not working. When I run a tracert from a workstation to a resource on each cloud, the tracert comes back with many ID'd hops for cloud 1, but cloud 2 only gets to the IP of the FW interface and then all other hops fail to ID.
I do not have a Gateway setup for either of them. This environment is not internet connected so it is just LAN to cloud and that's it. The routes retrieved from each cloud is not expected to be the exact same so it is not surprising that the count is different. Each cloud has different prefixes for different parts of the cloud that we will need to access.
The local-pref and as-path is something I am not familiar with. I see it in the BGP settings and route maps. How could I set this up so that it knows how to route? I mean I thought the routing was ok because the test for cloud 2 goes to my FW interface used for cloud 2, but it seems to get lost after that hop.
-
@djtech2k said in FRR BGP With Multiple Neighbors - Routing Issue:
I do not have a Gateway setup for either of them. This environment is not internet connected so it is just LAN to cloud and that's it
I don't understand your set up then. You are connected to a cloud provider but not using the internet?
Create a drawing and share please.
-
@michmoor I will throw together a drawing, but basically I have a dedicated P2P 10Gbps Fiber connection to the cloud provider. I have a peering connection to 2 different regions of the cloud, which is why I have the 2 BGP neighbors.
The idea is that I will push data into 1 cloud in some situations and other situations I will push data into the other cloud. I do not intend to "tie" the clouds together at all.
My network is a dedicated L2 fiber line so no internet access.
-
@djtech2k
ok, i understand now.
At this point its basic routing. BGP uses attributes to identify which routes are best for forwarding (uses NLRI).
Whatever destination you are selecting, its learned from both peers and your peer-to-provider 1 has the better metric. Simple.
There are ways to steer traffic to the second peer and as i already mentioned using AS-PATH may be your best option.
For route x.x.x.x/x you will use route-maps to set as-path pre-pending on the route received from cloud 1. -
@michmoor Ok here is some more info, hopefully will help.
I am connecting to 2 completely different clouds and I can see BGP routes for both neighbors on my FW. So I am receiving routes from both neighbors/clouds. I think the problem is that only the 1st neighbor/cloud is getting my route advertisement but the 2nd neighbor/cloud is not getting my routes. I have confirmed that I can see my routes on the cloud/neighbor 1, but the 2nd one shows no routes coming from my FW.
Does the route map AS-Path stuff you mentioned fix that or is it something else?
-
@michmoor In my situation, the routes to the 2 different destinations should not be the same so does the metric come into play in that scenario?
As I mentioned in my previous post, I can confirm that routes are working fine in both directions for connection 1. On connection 2, the routes are being advertised from the cloud to my FW, but my FW is not advertising its routes to the cloud and I cannot figure out why.
Does your suggestion of using AS-Path in a route map fix this?
-
Are you referring to setting an AS-Path filter and then setting in a route map or something different? As I look at it and try to understand it, it looks like a filter that can be set on the ASN of my IP subnets. Is that correct? If so, I am not sure how I can apply it because I have (2) /29 subnets specified, one for each cloud, but they both belong to the same ASN (part of my /28). My apologies if I am missing something basic here but just trying to understand if this is what I need to fix my issue and how I can apply it.
On a side note, I found this thread that mentions using Prefix Lists as a filter and was wondering if this could help my situation as well.
https://blog.matrixpost.net/set-up-dynamic-routing-with-frr-free-range-routing-in-pfsense-openbgpd-now-depricated/
-
I will get hands-on tomorrow so I am hoping to nail down what exactly needs done to get this to work.
I am not exactly clear on how to implement an AS-Path filter in my situation. I may try to test the Prefix filter mentioned in that last link with the any/any option to see if it makes my 2nd connection advertise the routes to the cloud.
Any thoughts or suggestions?