I applied the first patch and then the second, no issues there. Repeating the steps above, initial test seem positive. I will let it bake for a few days and report the status of it. It seems the AWS tunnels drop and reconnect one at a time in sequence once or twice a day. As long as no one texts me about connectivity, it will be a success. :)

@Zawi deceptively simple to say the least, and it took me a few times to see it in the documentation. I think I did try that before, but the key is that on my Site1 the Outbound NAT did not automatically include the subnet's from Site2, so once I put the Outbound NAT into Hybrid Mode and added the subnets, well things are now working as expected.

I am still using BGP though simply to avoid the static routes, I have a few subnets and am lazy. Couple of things I've learned also is under the Gateway entries, in Advanced you can define the thresholds for latency and packet loss for the gateway to be considered up/down, which is key here. Also, I had the VTI gateway set to disable monitoring, which in my testing also broke the failover, which was another key problem.

you need to apply it by neighbor.

your should use pf2.4.5

Bingo. I removed the brackets from the password and i ca now see information under OSPF status.

Thank you so much.

That's part of why it's still just a patch and not in the code. It helps some situations, but not others. That said, you can't really have it both ways. It can either restart the packages or not restart the packages.

The reason it has to restart is typically what you have seen -- FRR needs to restart to latch back onto an interface which has changed (or probably deleted and recreated).

IPsec may fare better because of the changes I made in https://redmine.pfsense.org/issues/9668 which cycles the interface in FRR live without restarting the package. I'm not sure if a similar change for OpenVPN would be viable.

Thank you ... and yes i was laughing at that point ..... feel ashamed 😁

I tried it again, but no success. OVPN routes and static routes are classified as external routes (AS External Link States) into the OSPF database,
so according to this: https://github.com/FRRouting/frr/issues/5290 they are not summarized. Only local LAN addresses have a chance to get summarized.

@0daymaster said in FRR OSPF Ext 1 Default Route With Unexpected Cost.:

default-information originate metric 50 metric-type 1

You can try to change metric-type 1 to metric-type 2 which is the default. It shouldnt change the costs then... as far as i understand that metric-type ...

Got it figured with some basic reading. What i was missing was the policy based routing via firewall rules as described here:

https://docs.netgate.com/pfsense/en/latest/routing/directing-traffic-with-policy-routing.html

Hi Jim,
Thanks for your advice. I checked and our upstream gateway routers are indeed configured to block bogons at that point, so there is no issue with leaving this removed from the WAN interface.

Regards,
Erik

Great! Thank you on both accounts. I disabled the gateway, and we're still operational. :-)
And I didn't realize you could just start typing names in the network box.... From the GUI, it sure looks like it wants a network address.
But now the alias is in use.

Thanks again, and Happy New Year!

@barryboden

You can get all that info from the GUI too

70e24528-fe0c-45b8-af6e-9ddbb82c980d-image.png

cdd7c3a4-5ab5-494f-98dc-8d053a3cebf9-image.png

I'm sure it is. We'll pick up the update eventually. 2.5.0 snapshots are already on 7.1.

It's unlikely that we'll bring FRR 7.x back to 2.4.4-p3 at this point, though.

@gdi2k
We deployed VTI+FRR (2.4.4p3) and that was a disaster. We're thinking about trying OpenVPN+FRR. Do you know if this issue is resolved in the newer (v0.6.3_1) versions of FRR?

That is its own complete router distribution (like pfSense), not something that could be added to pfSense.

Yes, the ring setup is to simplify and to test any possible solutions. The real system has 8 nodes, so 28 tunnels according to your equation - so some connections will go over double hops, especially those which do not see a lot of traffic. Obviously, finding a cost distribution which fixes the issue is not that easy either with 8 nodes.

If it were possible to have ACK traffic be forced to go the same way SYN traffic came, problem solved. Jimp hinted at something in the other thread, with a solution in 2.5.0, but it doesn't seem to work yet. (See here).

Yes, it sounds like the problem others had where the patches on 9668 helped.