Dynamic routing with multiple gateways?

[wayne47]

Given a setup like:

What I really want to do is have no gateway-based routing on the pfsense box. Instead, run OSPF on the outbound interface of the pfsense box as well as on the DMZ of all of the Edgerouters and have the routing "just work" where the OSPF routes make all the routing decisions and the firewall rules don't use the concept of gateways at all.

This would mean that all the failover conditions would be handled by OSPF, not the firewall rules on the pfsense box.

This seems to be in conflict with the design of pfsense where the firewall rules ant to have control of the traffic after it leaves the box.

I can not figure out how to do this. Any suggestions?

[Ulrik]

@wayne47

First of all not entirely sure why you would not want to use gateway groups based routing in your setup. Natually it looks like a fun lab project :-) That being said I am not sure that the setup you are purposing is able to do what you expect.

You might be able to have each of your edgerouters announce their default routes with different priorities to the pfsense box. Having the pfsense box use the best priority route.

What i guess you expect is that if one of your internet connections disconnects pfsense will fail over the route to one of the other edgerouters. The "problem" as far as I can see is that ospf will not detect that the Internet link is down if the link between the pfsense and the edgerouter is still up. So failover will only detect if the Edgerouter goes down not the internet link "behind" it.

Both pfsense gateway groups and EdgeOS failover solves this by running a "probe" that will periodically ping some address at the other end of the isp link and fails the link over in case a specific nuber of pings are lost.

[wayne47]

It's a real-world setup, not lab.

@ulrik My issue is not specifically failover, it's that I do not seem to be able to get pfsense to ignore gateways in firewall rules. The Edgerouters run various VPNs to different real IP addresses with OSPF routing.

A specific example:

• A.B.C.D is reachable via ER 1 as well as via ER 3.
• OSPF will prefer ER 1 if it's Internet link is up.
• OSPF will re-route to ER 3 if ER 1 loses it's Internet link.
• pfsense uses the IP address of ER 1 as the default gateway for the WAN interface (ER 1 has the fastest internet link and OSPF does not contain a default route).

If I use default for the gateway in a firewall rule, I expected that pfsense would just pick the best OSPF route. But it does not. So I defined multiple gateways for each ER. At that point, I can define firewall rules using each gateway and force traffic to a specific ER but then OSPF routing is ignored.

[Ulrik]

@wayne47 It would be interessting to see your pfsense interface config and frr status. It sounds to me that you have an upstream gateway configured on at least one interface (the one you call wan)

My expirence is that If you want routing to be handled by OSPF you should not have an upstream gateway configured on any of the interfaces receiving ospf routes and naturally also not override routing from firewall rules. If you want you default route to be one of the upstream edgerouters think you will need them to announce a default route.