FRR BGP over IPsec , when HA happens (slave-> master, master ->slave)
-
Pretty sure this matches up to your issue.
https://redmine.pfsense.org/issues/14633Boils down to A) FRR Upstream doesn't have an option enabled and B) Once upstream gets the option enabled , netgate would need to provide functionality for FRR working in a high availability set up.
The Request here has been pretty much stalled.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276534Unless the maintainer for the FreeBSD package modifies this, FRR with High Availability is non-existent.
-
@michmoor thank you, i was going crazy the last months. i've tried everything that i know...but eventually these that you point out are pretty much what happens. i'll keep and eye on those links if there are updates in the future. for the moment we'll deal with the manual clean-up
-
@michmoor Thanks for the info michmoor.
I was about to implement HA with FRR, I guess that I'll have to use policy route instead (two VTIs), at least for the time being.
Perhaps a gateway group with lower packet loss threshold to make the convergence faster.
-
@mcury
When it comes to convergence the best you can do , generally with any platform, is by enabling BFD.
On some vendors such as Juniper and PaloAlto of which i am familiar with, you can enable BFD with static routing.
If that's supported on pfSense that would be my recommendation and to not rely on gateway groups. Gateway groups are just a means of providing failover or load distribution but not quick failover which you get from BFD.Whats your tolerance? If you are ok with a few seconds outage then your method may be desirable but if you need milliseconds then bfd all the way.
-
@vinns If you really need this feature i urge you to join the free bsd mailing list. According to the website, that seems to be the best method of getting an issue addressed or at the very least getting the community engaged enough to maybe offer some workarounds.
https://docs.freebsd.org/en/articles/mailing-list-faq/
https://lists.freebsd.org/ -
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
by enabling BFD.
I have FRR OSPF with BFD enabled, it works perfectly, but not in a HA setup.
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
you can enable BFD with static routing.
I'm almost 100% sure that this is not possible with pfSense.
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
Whats your tolerance? If you are ok with a few seconds outage then your method may be desirable but if you need milliseconds then bfd all the way.
Two VMs, one hosting AD (samba-ad and consequently DNS), and another VM as a fileserver..
Any minute that the connections isn't working will be a very big problem..
I was expecting to be able to use FRR with BFD but it won't be possible anymore..So, as I see it, the only option I have right now, with HA setup, is policy routing.
-
@mcury
Without CARP supporting active/active there is no other option for fast high availability with routing in any capacity which granted is a very serious limitation.
For my deployments, that's a big issue so i tend to not use pfSense in much more "complicated" routing setups -
@mcury
this may help. There is a solution if you want to call it that here in this redmine.https://redmine.pfsense.org/issues/9141
The first statement here is nonsensical.
"" AFAIR it was done deliberately since in nearly all cases it would be an error to run an identical configuration on two routers running a routing protocol. You'd want separate feeds/connections to neighbors and to work out the failover using priorities/cost/etc in the routing protocols. ""
This is obviously ridiculous and counter-intutive to how high availability is supposed to work but moving the saved configuration to the other standby node looks to be a workable solution
-
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
This is obviously ridiculous and counter-intutive to how high availability is supposed to work but moving the saved configuration to the other standby node looks to be a workable solution
No progress here obviously, just wanted to add that in the mean time I'm using a workaround: every time i change something on the primary GUI I transfer the raw FRR running config onto the standby cluster (as saved config).
Ow, so it is possible..
I'll perform some tests to see how that goes, thanks a lot michmoor, I wasn't aware of any of this and I was about to jump into it.
-
@mcury let us know if that works
But if it does i cant imagine why that cant be sync'd. Lots of maintenance on the admins end to keep the configs in order
-
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
let us know if that works
I'll post here my findings.
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
But if it does i cant imagine why that cant be sync'd. Lots of maintenance on the admins end to keep the configs in order
If it works, I'll try to build a script..
updated the request: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276534
-
@mcury
curious...
Is FRR running on the standby firewall?If it is there needs to be a way to have the process down and only running when it becomes active otherwise the standby is going to attempt peering with upstream.
Im not to familiar with FRR in HA mode.
-
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
curious...
Is FRR running on the standby firewall?Not at the moment, I'm about to build the slave to form the HA, only a single firewall running at the moment, just waiting for two NICs to arrive.
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
If it is there needs to be a way to have the process down and only running when it becomes active otherwise the standby is going to attempt peering with upstream.
If state is slave, pfSsh.php playback disable frr.. perhaps a good logic for the script to run every second.
-
@mcury Curious. Got it working reliably?
-
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
@mcury Curious. Got it working reliably?
Unfortunately no, I depend on someone else to access the cluster, so I'm just waiting for him to call for the tests..
-
I tested this yesterday, if both nodes in the HA have FRR enabled, no routes are exchanged between peers.
I have both nodes with the exact same configuration, but backup node is with FRR disabled.In case primary node goes down, all I have to do is to enable FRR in the backup peer.
-
@mcury nice !
Still requires an admins interaction BUT the concept works.
I see no reason why it cant be automated. -
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
Still requires an admins interaction BUT the concept works.
I see no reason why it cant be automated.Exactly, a little intervention but nothing that takes a lot of time, tick two things, save and that is it. :)
I'll start to plan a script, something to check, am I the primary, if so, enable frr, something like that.
-
@mcury maybe the script can check the CARP status? So check if i am Master?
Also a secondary check as well. Maybe ping the SYNC interface of the neighbor. If its down and if you are master than bring up FRR.So high level
Every GUI change in FRR needs to be sync'd to the standby
The standby needs to monitor CARP status
The standby needs a reliable detector to know it should take over routing - pings the SYNC interface of the master. -
@michmoor said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):
@mcury maybe the script can check the CARP status? So check if i am Master?
Also a secondary check as well. Maybe ping the SYNC interface of the neighbor. If its down and if you are master than bring up FRR.Yes, I'll have to learn carp CLI commands to check the status, any help is much appreciated because I'll probably need to parse the output to get what we need..
Then, set up some ifs and elses in the master and in the backup.
A ping test would also help this checking..
And lastly, a cron job in both nodes