You were right. There was a configuration error on one of the ospf sites that was causing the asymmetric routing.

I admit, that was sort of a RTFM moment...

Thanks jimp... what I didnt mention these pfsense's were VMs that were between to different KVM hypervisors. these links provide a solution for any VNF / NFV peeps were kvm qemu is being used.

I added OSPF to FW rules on both sides and then had to do this as well. (virsh edit domain) and vm (pfsense) reboot

https://superuser.com/questions/944678/how-to-configure-macvtap-to-let-it-pass-multicast-packet-correctly
https://libvirt.org/formatdomain.html#elementsNICS

the new version has add to pfsense?

frr 5.0.1_2

@napsterbater

jim said that it is cant use vtysh for pfsense. i think this bug for show ipv6 bgp up.
always it cant show ipv6 bgp summary up .

link text

Status of 2.4.4...

I still loose routes when combining 2.4.3 with 2.4.4 and Cisco.

But the good news is that it's not the case anymore with the following configurations:
1-pfSense 2.4.4 (priority 0), pfSense 2.4.4 (priority 0), DR Cisco (priority 1)
2-pfSense 2.4.4 (priority 0), BDR pfSense 2.4.4 (priority 1), DR Cisco (priority 1)

In addition, both pfSense devices have the following setting: redistribute connected route-map DNR6

New issues:
1-After the upgrade to 2.4.4, connected interfaces are not redistributed anymore. As a workaround, disabling/enabling the interface sometimes works! And when not, it has to be re-created!

2-In order to have pfSense act as the BDR (the way we typically need it), it has to redistribute a default route to other routers in the area. At a minimum, the option "default-information originate" should be available on the UI with ideally the possibility to also select "always". When configured this way for both the DR and BDR, 2 default routes will end up on all the routers.

You could go for a completely manual config but the easiest workaround is what you did before, just add those networks to the manual list to distribute.

@jimp said in 2.4.4.a.20180716.1125 & frr 0.2_2 issues:

That's a side effect of how the pkg edit interfac

Many thanks Jim, think I'll pop in a redmine re the length of the password string, to either check the length before saving or mention in the text there is a length limit.

Digging further into the FRR OSPF IPV6 GUI functions i see more problems within the GUI and function of the FRR package:

OSPF IPv6 doesn´t work with OPENVPN IPv6 P2P tunnels. Changing the OPSFv6 Interface to use the WAN Interface works perfectly. The IPv6 tunnel is working perfectly, FW Rules are set to "pass ipv4+6 * any any" but there is no OSPF "Hello" activity on the IPv6 tunnel, when OSPF6 ist set to use this tunnel as IPv6 activated interface with another FRR Pfsense on the other site. Usually OSPF IPv6 routes are based on the Link Local IP address of the interface, maybe this is a problem here, just guessing.

OSPF IPv6 current version cannot use areas (not implemented yet) - so the OPSFv6 GUI is really misleading, we can change the area to some other, but there is no warning that there is no function behind that. There maybe a future version, where areas are supported in OPSF IPV6.

OSPF Global Settings: The subnet field ist too short for a full IPv6 address, so a long IPv6 address is only partially displayed.

OSPF6 Settings : the last part is really a problem, we can suppose, that there should be "Distribute Networks" and "Disable Redistribution" but non is there - only a subnet/area id field. There are some parts missing and it doesn´t work … even in OSPF v4 it doesn´t work.

We really need an updated version of the FRR routing package, the current version is 5.x, where in Pfsense we are at 3.x.

I really like that FRR package, but it is in a "BETA" State and with all this GUI problems not easy to implement.

Regards Pete

Update. I think I fixed it. But I don't really understand it properly. I'd appreciate it if anyone can explain what's happening!
I had to add some rules to Outbound NAT. On each pfSense I added a rule for all OpenVPN tunnel IP addresses (10.127.0.0/16 in my case) sources on the WAN to translate to the WAN interface address. This got the ping working via the third pfSenses during VPN outages. I then also added a rule for all IP addresses sourced from the LAN on the OpenVPN interface to translate to the OpenVPN interface address. In my example above, this meant NATing 192.168.128.0/24 on test1, 192.168.129.0/24 on test 2 etc. Now it works. If I set a ping going from one pfSense's lan to another, and I stop the VPN between the two, the pings get re-routed via the third pfSense. A few pings get lost while it's swapping, but this is what I wanted! Back of the net!
Anyway, as I said, if anyone can explain what's happening here, that would be great. I won't mark this [SOLVED] just yet until I'm sure I've done this correctly.
p.s. Don't you love it when things start to work just before pub o'clock!

In order for FRR to work with BFD you currently need PTMD.  This is planned to be fixed in a future release of FRR.

As far as I know, making any changes in the pfSense FRR UI will bounce the FRR service.

You can make changes from vtysh without having the FRR service bounce and write them. The problem with doing this though, is vtysh and the UI don't sync for some reason.

:'(

RIP can't get no love. Yeah I should use something more secure anyway.. It's just handy because in a home lab it just works and everything supports it.

Packages do not synchronize with XMLRPC unless they implement their own XMLRPC synchronization settings, and FRR does not do that currently.

You will have to set it up on both nodes separately for the time being.

In FRR, the accept filter list is under Global Settings in the Route Handling section since it's a function of Zebra, not OSPF. It was hacked into Quagga since it only handled OSPF.

Is it assigned under Interfaces > Assignments?

OpenVPN interfaces are a special case that don't need an assignment, all others do.