Forcing pfSense internal traffic over gateway that's currently "up"
-
I'm configuring a multi-WAN setup, the first WAN being the ISP's and the second an OpenVPN client.
By default, the OpenVPN provider creates routes to route all traffic via the VPN, which creates a few problems:
-
Gateway failover doesn't work (unless the OpenVPN service actually stops, in which case the routes created by the VPN disappear, else if the service is running but no traffic is going through, the routes persist and it doesn't move to the other gateway)
-
Doesn't respect disabling a gateway - if I disable the VPN gateway the routes are still up, so traffic is still routed via the VPN.
However, the good things about it is that any internal pfSense traffic is properly routed via the gateway (NTP, DNS, pfSense update, package downloads, etc etc..)
The ideal scenario for proper failover would be to prevent the VPN from creating the routes and use firewall rules to route traffic via the a gateway group with both gateways in there on different tiers.
But this introduces the problem of internal pfSense traffic not being routed via the gateway. With DNS resolver and forwarding mode disabled (which I'd like to keep) I can't see a way of forcing DNS traffic via the gateway that the system currently considers up. Same goes for any other internal traffic, such as NTP, downloading packages, etc.
So, how can this be done?
NOTE: I don't really care about NTP, downloading packages etc not going over the VPN, but I'm asking how to do it on everything for completeness and because I have seen this question on the forums several times with no definitive answer.
-
-
-in openvpn there is a "route-nopull' option in the webgui … this will stop openvpn from pulling routes.
-assign an interface to openvpn (interfaces-->assign).
-configure the interface with ipv4/ipv6 type 'none'
-create a gateway for the above interface if this hasnt been done automatically
-create a failover group with the openvpn interface on the lowest 'tier'
-adjust firewall rules to make use of the failover groupif you set the optX_gateway to be the default gateway, then pfsense will use it for "interal traffic"
-
if you set the optX_gateway to be the default gateway, then pfsense will use it for "interal traffic"
and if the OpenVPN server end is found by a name (rather than a fixe IP address) then you probably need to enable default gateway switching, so that during boot the system can resolve DNS on WAN to get the IP of the OpenVPN target server and thus establish the OpenVPN link.
Once the link comes up, then the default gateway should be switched over/back to the OpenVPN link gateway. -
Right, completely forgot about the "default gateway" feature…
Still, if I'm configuring OpenVPN as the default gateway with default gateway switching, why would I still need both a gateway group and a firewall rule?
Bearing in mind that all my clients use pfSense as the gateway, whatever traffic goes through pfSense will go out the currently active default gateway regardless of where it originated (internal to pfSense or from clients behind pfSense).
And if the default gateway switches automatically when the other is down, that should provide the automatic failover I'm after without the need for gateway groups and/or firewall rules, or am I missing something?
Unless I'm mistaken, the only reason I'd want to have gateway groups and/or firewall rules is if I wanted to shape the traffic (ie, don't VPN some clients for example, or some type of traffic).
The only question is... with default gateway switching enabled, if WAN1 is default gateway and it goes down, it obviously makes WAN2 default gateway automatically, but if WAN1 comes back online does it switch back to WAN1 or does it keep WAN2 as default until WAN2 goes down?
-
It will switch back to the real selected default gateway when that comes back up.
I just suspect you will need the default gateway switching to get through a catch-22 at boot time, when the firewall itself needs a way out to to a DNS server and to the internet to bring up the OpenVPN tunnel.
Internally-generated traffic from the firewall itself does not have a chance to have the policy-routing rules applied to it.Anyway, you really need to check/test this to make sure of what is needed or not needed here - I am only going on theory and do not have any similar OpenVPN configuration to verify.
-
Completely agree with you on needing default gateway switching enabled to prevent the issue you described, but I'm still not convinced I need either gateway groups or firewalls rules to create policy routing (unless ofc I want to setup exceptions)
Would you disagree?
-
Umm, I've just tested the default gateway switching feature…
I set the OpenVPN as the default gateway. I do not have gateway groups or policy based routing at all.
With both WANs up, the traffic goes out the OpenVPN gateway.
If I disable or mark gateway as down on the OpenVPN gateway... the default gateway does NOT switch. All traffic is still going through the OpenVPN gateway, even though it's considered to be down/disabled on pfSense.
If I disable the OpenVPN service, the default gateway switches over to the other gateway (as expected).
However, if I start the OpenVPN service again, the default gateway is still not considered to be OpenVPN gateway and no traffic is being routed via the OpenVPN gateway, even 10m after starting the service back up. pfSense acknowledges both gateways are up, OpenVPN has a latency of 30ms... but it's not using it as the default gateway.
What am I missing?
-
In order to force it to change the default gateway back to OpenVPN, I had to mark the WAN gateway as down and back again as up (ofc marking it as up was not needed to make the default gateway change but I just wanted it to be up).
But the same thing does NOT work for forcing it to change from the OpenVPN gateway to the WAN gateway.