OSPF routes are not loaded as kernel routing rules
It seems I found a bug in pfSense with my setup…
I have two sites: one of them is a colocation rack in a datacenter (site DC) and the other one is our company (site HQ).
Site DC has only one WAN connection while site B has two connections: one to ISP A and the other to ISP B.
The DC firewall has two OpenVPN servers responding at different IP addresses and site HQ connects to one IP via ISP A and to the other IP via ISP B (static routes are configured in the pfSense box to make this happen).
Over the OpenVPN tunnels I run Quagga OSPF with different costs on site DC and HQ. ISP A has cost 1 and ISP B has cost 5.
I ran into trouble when I had to physically disconnect one of the ISP connections (ISP A) because they stopped doing peering with our datacenter and started using their transit provider (latency jumped from 12ms to 70!!).
It seems somehow Quagga (OSPFD and Zebra) was restarted and Zebra inserted kernel routes pointing to the ISP A OpenVPN tunnel were not deleted from the routing table. When the OSPF algorithm ran again it already had a better route already installed and therefore did not install the route through ISP B.
I can login to the pfSense box and manually remove all routes to ISP A tunnel but I really which this kind of failover worked.
Sounds like you are hitting this:
If the Fix in that post solves it, be sure to put a reply in that thread
Thanks but no VLANs here. :-(
that doesn't seem to matter … its a appears to be a general issue: