Routing of Gateway Group for VPN providers: Trigger level not working

MichaelAnders · Aug 22, 2023, 9:20 AM

Hi everyone,

What I want to achieve: if any of the VPNs (details below) have a high latency, switch to a different VPN.

The problem: Even though high latency is detected by pfSense no other VPN is used.

My setup:

I have one WAN connection
I have 3 VPN connections set up to different OpenVPN providers (ProtonVPN, Surfshark, NordVPN)
Each Gateway has a different monitor IP
I have set up a Gateway Group "VPN" which contains all three VPN gateways as well as WAN. Each VPN gateway has a different Tier (1,2,3). Trigger Level is set to "High Latency"
Firewall rules are set up for my different VLANs to use the same "VPN" Gateway Group

Behaviour:

The VPN Tier 1 is always used by default, and the VLAN firewall rules work properly
If VPN Tier 1 has a high latency (which I am informed about in the pfsense WebUI) the connection is very slow
No fallback to Tier 2 or Tier 3 happens

Any ideas?

Thanks!

viragomann · Aug 22, 2023, 1:48 PM

@MichaelAnders
So you direct the traffic to the VPN gateway group by policy routing rules. Consider that this does not affect upstream traffic from pfSense itself like DNS lookups.
Did you verify that the next tier wasn't use for the upstream traffic?

Also check if the latency threshold gets really overshot. Maybe you want to adjust the threshold for your needs.

MichaelAnders · Aug 22, 2023, 1:48 PM

@viragomann
I mainly use these for e.g. Youtube, surfing and my "guest" VLAN, so I don't think that is mainly "upstream traffic"?

DNS lookups are done on a separate VLAN VM (pihole) which directly connects to WAN, so that is fine

I changed the latency threshold - that helped, thanks a lot!

Follow up question - can also be a new topic if that is better?

How would I handle the case "Member Down" for the same gateways? I assume this would be the case when an OpenVPN client can't connect/crashed/failed to restart?

I see it in the "Trigger Level" drop down, but it can't be combined with "Packet Loss or High Latency".

For testing, I just set up a 2nd Gateway Group to handle that, but that does not help as in the "firewall rules" I have to choose which of the two gateways I want to use... even if I add to the list, I can only send all the "sources in my "guest" VLAN out through one gateway?

viragomann · Aug 22, 2023, 2:05 PM

@MichaelAnders said in Routing of Gateway Group for VPN providers: Trigger level not working:

How would I handle the case "Member Down" for the same gateways? I assume this would be the case when an OpenVPN client can't connect/crashed/failed to restart?

I see it in the "Trigger Level" drop down, but it can't be combined with "Packet Loss or High Latency".

Don't understand. "Member down" is determined by packet losses, I assume. So if you have set the trigger level to "packet loss + latency", why do you need a "member down" trigger additionally?

MichaelAnders · Aug 22, 2023, 2:22 PM

@viragomann

"Member down" is determined by packet losses, I assume.

That was my assumption as well.

But then this should not be happening I think:

I stopped the OpenVPN Instance
pfsense shows the Gateway as "packet loss, offline" - correct!

However, for 25-30 seconds till the OpenVPN instance is auto-restarted and shows up as "online", I try opening any other website like I did with "high latency" but nothing happens, the traffic graph for none of the configured Gateways shows any traffic (apart from e.g. 50 byte or so on the running ones). That lead me to assume that this is "member down". Could also be another reason.

viragomann · Aug 22, 2023, 2:36 PM

@MichaelAnders
Consider that existing states are bound to the gateway. So if the client is aware of an existing connection, but the gateway is offline, he will run into a timeout.

To clear the states immediately in such case you can check System > Advanced > Miscellaneous > State Killing on Gateway Failure.

MichaelAnders · Aug 22, 2023, 3:49 PM

@viragomann said in Routing of Gateway Group for VPN providers: Trigger level not working:

@MichaelAnders
To clear the states immediately in such case you can check System > Advanced > Miscellaneous > State Killing on Gateway Failure.

Mine was set to "Flush all states on gateway failure" - I changed it to "Kill states...". Same behaviour...

Maybe the description of that parameter which contains "Not triggered by gateways... which have been forced down" applies in this case. I guess there would be some way to "kill" the OpenVPN client from terminal to simulate this better (and it's not a manual force down) - but that is out of my league...

Maybe I'll just wait and see what these changes do when the connection to the VPN is actually down (happens now and then for some minutes).

Q: Can I see something in the system logs to know that Tier 1 is no longer used but pfsense switched to e.g. Tier 2? That would help me when playing around

Bob.Dig · Aug 22, 2023, 3:52 PM

@MichaelAnders Activate E-Mail notifications if you don't already have. You will get lots.

MichaelAnders · Aug 22, 2023, 3:57 PM

@Bob-Dig said in Routing of Gateway Group for VPN providers: Trigger level not working:

@MichaelAnders Activate E-Mail notifications if you don't already have. You will get lots.

Thanks, I enabled that now, test mail works. Let's wait and see :)