@cmb:
Sounds like what would happen if you're maxing out your upload bandwidth on that connection, or if that connection has issues in general. Is it that high with the connection idle?
It's idle. No further traffic goes over WAN2 when WAN1 is down. It's not a failover. Only UDP source port 27005 and some game client UDP destination ports are permanently sent through WAN2. The increased latency is always 500 or 999 ms + the actual route latency to be exact.
Even when WAN1 is up and I'm playing a game, the game is going through WAN2. During this time the WAN2 latency is normal. If WAN1 fails, the latency goes up shortly.
A related problem as mentioned in the linked thread is when WAN2 is down, the above rules don't fall back to WAN1.