FRR config for pfsense-vpn (Cloud A)

##################### DO NOT EDIT THIS FILE! ###################### ################################################################### # This file was created by an automatic configuration generator. # # The contents of this file will be overwritten without warning! # ################################################################### ! frr defaults traditional hostname pfsense-vpn.**hidden** password **hidden** log syslog service integrated-vtysh-config service password-encryption ! ip router-id 192.168.240.6 ! interface gre0 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 1 interface vtnet0 ip ospf network broadcast ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 0 interface ovpns1 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 0 interface ovpns2 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 0 ! router ospf ospf router-id 192.168.240.6 log-adjacency-changes detail redistribute connected metric 1 redistribute kernel metric 1 redistribute static metric 1 passive-interface ovpns1 passive-interface ovpns2 area 0 shortcut default area 0 authentication message-digest area 1 shortcut default area 1 authentication message-digest ! line vty ! end

FRR config for pfsense-main (Cloud B)

##################### DO NOT EDIT THIS FILE! ###################### ################################################################### # This file was created by an automatic configuration generator. # # The contents of this file will be overwritten without warning! # ################################################################### ! frr defaults traditional hostname pfSense-cloud.**hidden** password **hidden** log syslog service integrated-vtysh-config service password-encryption ! ip router-id 10.0.148.2 ! interface gre0 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 1 interface ovpns1 ip ospf authentication message-digest ip ospf message-digest-key 1 md5 **hidden** ip ospf area 2 ! router ospf ospf router-id 10.0.148.2 log-adjacency-changes detail redistribute connected metric 1 passive-interface ovpns1 area 1 shortcut default area 1 authentication message-digest area 2 shortcut default area 2 authentication message-digest neighbor 10.0.148.1 priority 100 poll-interval 40 ! line vty ! end

@jcook-atlas so I think in a hub and spoke topology with OSPF I think the issue here might be incorrect nexthops? I would check that.
But if this were me I would do BGP. Specifically iBGP with your hub acting as a route reflector. If that won’t work then eBGP.
That’s more of a typical design than using ospf

@schannes The documentation is not entirely clear but you are correct. If the config is saved than any GUI elements you set are not saved. You have to clear it first. Its wonky.

@yon-0 Did you submit a redmine? https://redmine.pfsense.org/

@mystique_ OSPF is pretty simple to set up.

Enable it and add the interfaces to area 0 and you're done.

One generally sets interfaces that are to be in the OSPF database that are not intended to communicate with other OSPF routers to passive.

That's generally all that HAS to be done to get the IGP working and exchanging routes.

@wblanton FRR does not currently by default do any asic offload of BFD. 50 BFD peers should be ok on modern cpu's I would think though

@jamiegb said in Publish 2 Identical Routes with Different Metric:

te twice but with a different metric so my peer will pre

Its common to adjust either local-pref or as-path prepend attributes. In your case, if you want your peer to use one path for a destination vs another then as-path prepend that route out a few times[using your local-as of course].

@rebelboy1988 I would remove the route-map from the neighbor command so you have no filter applied and then see if you are getting routes. If not then the problem is with the AWS peer.

@ersany If you wish to look at the FRR config file when using the GUI to configure it (not raw config), Go to Status > FRR, Configuration.

As explained above, using the raw configuration disables the GUI config. You have assumed responsibility for the FRR configuration in that case and changes must be made to the raw configuration instead.

@pete35 I grabbed a Plus lab license and changed a CE 2.6.0 install to 22.5 and saw that FRR is unchanged in that version, and the underlying FRR version is still only 7.5.1 at this point in time. Not sure where Netgate's priorities are here, as solid/dependable routing is absolutely key for corporate customers - the ones most likely to pay for the upper tier support contacts. IMHO FRR should be a top priority. FTIW pfSense CE 2.7.0 is still only on FRRR 7.5.1 but granted that is still early beta, but would be a shame if it too was still only 7.5.1 when final and not on a more recent FRR 8.x release.

Summarising the initial 6 things I raised:

1. #1. SPF algorithm firing causes OSPF "redistribute connected" routes to flush.
This was raised in #11835.
I can see that no one has worked on this critical bug. I have tested and this is still an issue (!!)

#2. OSPF protocol filtering (FRR GUI - Global Settings / Route Handling) causes FRR to do strange things (and make OSPF routes invalid / crash FRR etc)
I avoid the "FRR GUI - Global Settings / 'Route Handling'" way of filtering as I found that too unstable so haven't tested it since finding it a problem. I have done filtering elsewhere on my Mikrotik routers instead.

I did raise 11836 for a related issue, and some things improved there, but not sure if this actual issue is fixed or not. Since I don't use the "route handling" features I stopped looking at this issue.

#3. ACL's no-longer have an implicit deny at the end.
I did raise 11841 but I am not looking at this issue as I found that prefix-lists weren't affected so I swapped over from access-lists (ACLs) to prefix lists for my needs (for the redistributing of specific connected routes into OSPF).

#4. OpenVPN links re-establishing can cause "onlink" routes to become inactive
@mdomnis How did you end up going with this? I didn't actually raise a ticket for this but you've been working with pfSense on it I see. I'm not seeing it in pfSense 2.6, but my test lab might be different to when I had it last. Solved?

Issues #5 and #6 - ACCEPTFILTER prefix list entries to be duplicated, and Interface descriptions cumulative
These got fixed - am not seeing these issues in pfSense 2.6. They would have been pretty trivial to sort out.

======
so in short, #5 and #6 are fixed. #4 seems to be fixed (to be confirmed).. #2 and #3 - I have worked around (have avoided those features, thus I'm not affected).

The only thing that that I am affected by right now (and cannot avoid) is issue #1. And it's still really bad. Here's one of my connected routes dropping the moment a backup link comes back up:
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 01:07:33
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 01:07:35
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 01:07:37
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 01:07:38
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 01:07:40
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 00:00:01
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 00:00:03
pfsense01.it.somecompany.com.au# show ip route | include 10.24.1
O>* 10.24.194.0/24 [110/20] via 10.255.195.2, ovpns2 onlink, weight 1, 00:00:04
pfsense01.it.somecompany.com.au#

10.255.195.2 is the far end of the primary link (p2p). The backup p2p link re-establishing should not cause this route learned over the primary link to flush and relearn. I'm testing pfSense 2.6.0-RELEASE which is built on FreeBSD 12.3-STABLE and has FRR version 7.5.1

update: I cloned my lab and updated pfSense to 2.7.0
2.7.0-DEVELOPMENT (amd64)
built on Mon Oct 17 06:04:34 UTC 2022
FreeBSD 14.0-CURRENT

It is still happening on there. The FRR on 2.7 is still only 7.5.1. Why so old? https://frrouting.org/release/ That's from March 7 2021. FRR is up to 8.3.1 now - 5 releases on from that. Really would like to see what happens in a later version of FRR and hoping the devs can update the FRR package to the latest release soon.

@wstocker I have a strange issue where I can propagate routes into AWS and see them in Transit Gateway's route table, however I am getting no routes back from Transit Gateway for Propagated VPC attachments.

Did you use a Virtual IP address or and enable the P2 VTIs as an interface?

@itsdave @srain @nzkiwi68 @awebster
I managed to redistribute exact openvpn networks without usage of supernets in other way:
interfaces > assignments > and assigned openvpn interfaces and enabled them in interfaces menu:

c4974312-a0e8-4d2e-a506-f268071cbe59-image.png

and added "static route targets" pointing to those previously created (and enabled) openvpn interfaces: services > FRR global/zebra > global settings > route handling

7efd11ba-f046-4727-b19c-d4840d9339d7-image.png

hope this will help someone and sorry for reviving a quite old post:)

@netravnen afaik pfsense doesn't have a GUI to configure ISIS - but i see that the isisd executable is present on the filesystem. you could try to craft a config manually to get it running.

if you wish to get it supported/included (quickly) in the GUI then you could create a pull request yourself or find a volunteer to create said pull request.
PR's should probably go against the devel branch:
https://github.com/pfsense/FreeBSD-ports/tree/devel/net/pfSense-pkg-frr

This got resolved with the prefix list filter incorrectly configured. Added allow all on prefix filter on inbound list filter and now traffic flows through the BGP routes.

An additional concern is, we receive default route i.e. 0.0.0.0/0 from the ISP and some prefixes from another peer group. I have given weight of 1 to ISP neighbor and weight of 10 to peer group. Still majority of the traffic to the routes available from the peer group flows through the default routes. Please suggest, if any further changes required on this.

Thanks

@marceloalm_

Hi we are building a similar network and in need to decide > between netgate or mikrotik router. There is any chance to > enable ecmp on current pfsense?

I would assume that you will be better go with MikroTik
RouterOS or VyOS.

https://wiki.mikrotik.com/wiki/ECMP_load_balancing_with_masquerade

@mdomnis I have since upgraded to 22.01 with FRR version 1.1.1_6. In my preliminary testing, the routes seems to be working closer to what is expected. I still have a weird issue where sometimes the neighbors don't like to peer fully and I have to force restart FRR, but from some quick tests, it looks like at least the route is being added to the table correctly. For now at least.

My problem solved, the problem was with filters in ip6tables

@michmoor just tried with BGP and this too is failing to establish. There might just be a missing configuration on the pfsense I'm not seeing but this should all work. This isn't a multicast or unicast problem.

this issue has be resolved by upgrading to pfSense Plus 22.01
the other side was already on 2.6.0

@viktor_g
Issue created https://redmine.pfsense.org/issues/12820

@mdomnis Sorry for the late reply! I haven't experience this issue anymore since upgrading to 21.05.2. However, I have also removed the L2TP connection since I no longer need it. I think that is the same behaviour I saw, but I can no longer confirm it.