pfSense Multiple WAN Failover fails to switch when ISP Bandwidth is Throttled

SherifMalek · Dec 20, 2021, 5:39 AM

Good evening,

I have 3 internet connections coming into my place.

WAN1 - ISP1 (VDSL - 75Mbps)
WAN2 - ISP2 (VDSL - 75Mbps)
WAN3 - 4G Modem

Currently, I was able to setup each WAN correctly, and associate it to the relative Gateway settings. I was able to setup Gateway Groups, and Tiers and monitor their status (online/offline) and the fail over correctly works only in the scenario where the actual ISP is down (packet loss / abnormal high latency).

So the current setup is
ISP1 -> ISP2 -> 4GModem
Tier 1 -> Tier 2 -> Tier 3

ISP1 has the best speed and ok latency, ISP 2 is not as good but functional as a backup. Unfortunately, ISP1 and ISP2 have a Data Cap of about 250GB/month each, after which the speed slows down from 75Mbps to 256kbps.

Here is where the issue comes. When the ISP caps the speed, the gateway monitor still detects that there is internet (successfully pings Gateway Monitor IP) however, the speed is soo slow, and pfSense does not switch to Tier 2, 3 etc..

I tried playing with different Packet Loss & Latency Settings to consider the gateway as down, but was unsuccessful since the VDSL in my area already has high latency rates so these factors aren't accurate enough to trigger fail over switching.

How can I setup pfSense to detect the slower speed (bandwidth throttling) and switch to the other gateway tiers when the ISP download Cap is reached or download speed is reduced? I can't seem to find a solution for that.

Any Ideas?

Thanks