Add pass rules above your default any to loadbalancer rule:
pass, proto any, source any, destination WAN subnet, gateway WAN-gateway
pass, proto any, source any, destination OPT-WAN subnet, gateway OPT-WAN-gateway
You have to exclude these subnets from loadbalancing.