FRR 7.3 -> 7.5 - BGP not announcing routes

mfld

Does anyone have experience with upgrading from 7.3 to 7.5 ?

Mine stopped announcing the route even though session is established once I upgraded pfSense 2.4.5-p1 to 2.5.0-RC.

Only clues in documentation I could find it are about RFC8212.

mfld

Update to anyone finding this by search:

Between FRR versions 7.3 and 7.5 some things changed:

frr defaults datacenter -> frr defaults traditional

and bgp ebgp-requires-policy is now enabled by default as is bgp network import-check

As a result and depending on your previous setup and peers you may need to manually set frr defaults datacenter and/or no bgp network import-check

mfld

https://redmine.pfsense.org/issues/11392

jimp

The default changed to not accept or distribute routes to a neighboer without a policy in place. This would only affect those who don't already use a route map on their neighbors, so for most the main change will be adding an "Allow-All" route map and then setting that as the route map filter on each neighbor.

Under FRR, go to Global Settings, Route Maps, add a new one that just has:

Name: Allow-All
Action: Permit
Sequence: 100 (or whatever)

Then under BGP, edit each neighbor and set it as both entries (inbound and outbound) for Route Map Filter under Peer Filtering.

It's a more secure default, but it may catch some by surprise.

That can be added before upgrade, too, which would prevent this from ever being a problem.

mfld

@jimp Awesome.

On the prod machine I have prefix lists for the neighbor.

On this test box there was no policy so it didn't announce the prefix after upgrade.

I did try

no bgp ebgp-requires-policy

which is exposed in the GUI and the flag works as expected. But setting this alone and bouncing the session did not announce the prefix.

Only when I set

no bgp network import-check

Does it announce the prefix. And the flag for bgp network import-check in the GUI is weird because there is no setting (checked or unchecked) that will result in

no bgp network import-check

being set in the config. Clearing the checkbox removes the statement from the config which since FRR 7.4 or 7.5 means the inherent default is yes. Setting the checkbox in the GUI creates

bgp network import-check

In the config which is superfluous because of the inherent default.

jimp

Have you tried not setting that at all, and instead setting the route map as I described?

mfld

@jimp said in FRR 7.3 -> 7.5 - BGP not announcing routes:

Have you tried not setting that at all, and instead setting the route map as I described?

I will try restore the 2.4.5 snapshot of the test instance, set the route map as you described and upgrade to 2.5.0-RC anew.

Will get to it around 0200 UTC and report back.

mfld

@jimp said in FRR 7.3 -> 7.5 - BGP not announcing routes:

Allow-All

So I restored 2.4.5-p1 snapshot on this demo install.

It had only filtered by way of prefix list. Announcing IPv6 prefix to one neighbor.

Prefix list filter for the neighbor was set to

inbound
order 100
accept ::/0
order 200
deny any

Outbound was to the
order 100 accept the prefix I want to announce
order 200 deny any

This worked and I assumed this counts as a policy for the purposes of RFC8212.

Now I added a route map as per your description and upgraded to 2.5.0-RC.

The prefix is no longer announced!

Something else I notice is that my prefix list is no longer selectable in the peer filter.

I will revert to the 2.4.5 snapshot, remove my prefix list filter, then add a route map and see how that goes.

mfld

Notice another thing.

In 2.4.5 a prefix list did not have an IP type.

Upgrading to 2.5.0 it imported my prefix list which was exclusively IPv6 but did not recognize so. The prefix list is there but cannot be selected. Editing it and pressing "save" I see for exery row:

Network in row 0 IP Type mismatch.

I notice it now has an IP type selection in the GUI and it has imported / migrated my IPv6 prefix list but defaults to IPv4 in the GUI. After upgrading to 2.5.0-RC I need to go into the IPv6 prefix list, select IP Type IPv6 and save it in order to use it again.

mfld

I had a few things going on that caused my FRR issues after upgrading.

My understanding of what broke me:

First off, the RFC8212 enforcement in FRR 7.5 (which is a good thing!) will accept either a route map as @jimp suggested or a prefix list. I see that with either or both of them defined, there is no "(Policy)" filter indicated in BGP summary.

In my case I had prefix list for inbound and outbound filtering configured on my neighbor that worked well in 2.4.5-p1. Upon upgrading to 2.5.0-RC they were migrated across in a broken state.

I noticed by accident when my other test box that receives full routing table from the neighbor and only has 1GB of RAM started going OOM and failing as soon as I ugpraded.

This here:

when assigned to neighbor's Inbound Prefix List Filter will only accept default route and you can run a 1GB RAM instance where the neighbor is sending the whole table.

When upgrading to 2.5.0-RC there is a new "IP Type" field for prefix-lists and my IPv6-only prefix lists migrated across with the value empty, the lists were being ignored.

This ties into the outbound prefix not being announced because my outbound prefix list was equally broken. Absent an override flag this means we announce nothing and receive the full table if the peer sends one.

show bgp summary

will show "(Policy)" under PfxRcd / PfxSnt to indicate that stuff is being withheld for lack of an explicit policy. An explicit policy can be a route map or a prefix list. My IPv6 prefix lists broke during upgrade thus I ran into trouble.

Second issue is that with FRR 7.5 the default behavior of FRR is to only announce prefixes already existing in the RIB. If this is not the case in your environment you need to set "no bgp network import-check" which 2.5.0-RC also exposes in the GUI but I feel it is not currently working and I have expressed that in this bug report.

I kept snapshots of these lab instances so can replicate or test other theories if needed.

Summary:

RFC8212 is now enforced. You can disable it via the GUI which will set this flag but long term one probably wants to be a good neighbor and have a route map and/or prefix filter in place.
If you announce prefixes not in RIB, you need another flag and the GUI implementation is currently not working as one would expect.
If your peer filtering was relying on prefix lists in 2.4.5-p1 and is now broken, you may find that the IP Type was not detected and it migrated across with it blank so check BGP neighbor status to see if it claims the prefix lists are being applied.

viktor_g

@mfld said in FRR 7.3 -> 7.5 - BGP not announcing routes:

If your peer filtering was relying on prefix lists in 2.4.5-p1 and is now broken, you may find that the IP Type was not detected and it migrated across with it blank so check BGP neighbor status to see if it claims the prefix lists are being applied.

The same issue can happen with access lists, since 2.4.5 doesn't have an 'IP Type' either.

Issue created: https://redmine.pfsense.org/issues/11404

mfld

@viktor_g

Cool, I did another rollback to 2.4.5-p1 and upgrade to
2.5.0.r.20210214.0300.

All I need to do now is go into the prefix lists and set IP Type to IPv4/6 and save them to work around https://redmine.pfsense.org/issues/11404

But https://redmine.pfsense.org/issues/11392 remains a more troubling issue because checking or unchecking the box in the GUI does not change the FRR behaviour due to the new inherent default. The only way to work around it is to edit RAW config by adding "no bgp network import-check" and override the GUI.

Will rollback and re-test once the two bugs are in the update.

manu

Hi,
I have you correct your problem?

I migrate to 2.5.1 : I had ipv4 route but no ipv6 route with bgp
I had route maps and prefix lists on neighbors with no success

have you an configuration example ?