I've been playing with this as well since i have connections with 6/1, 10/1, 30/2 and 100/8 speeds, all together in the same pfSense box. I'm starting to think the only way to achieve this is to create 1 VM for each wan link, with it's own pfsense and within them make the up/down limiting with the last box just doing all the current traffic. I really don't like that solution.

So I tried luckman212's suggestion and it worked! :) I think something like this should be commented on within Sticky Connection topic. Cheers all  ;D

You probably have an incorrect configuration for the monitor IP addresses and DNS servers for each gateway. Set each gateway monitor IP address to a DNS server on System > Routing and set the DNS server to the same gateway under System > General Setup If you have some things mixed up, such as all the DNS servers pointing to your WAN and also using them as monitor IP addresses on the other WAN gateways, that can cause exactly the behavior you describe.

See there's your problem - using logic and sense while evaluating how i got here:-) I had no idea what pfsense could and couldn't do before i started this.  Never used it until a week ago, only marginally aware of it. I had the perception that it is a platform of components that one can use.  You see pfsense as a firewall - i see it is a comprehensive 'security platform' of things i can use as i see fit. Both are true. I don't need the pfsense firewall / NAT(because i can't turn off the one i have.. and don't want to double NAT or double firewall) As such all the outbound connections i want to block with my off the shelf pfsense box are done at the squidguard / suricata level. This can be done in one of two locations - between the cable modem and my router or between my router and the rest of the network.  Either way i need to bridge as I want it transparent, i don't want to mess around with wpad. pfsense gives me easy to use turnkey system to do this, i don't want to install linux - i have no no interest in maintaining a linux machine -  i bought a box with pfsense installed that does the job i need it to in an easy way is great - not sure why you are so horrified about what pfsense modules / features and packages I do or don't choose to use I wasn't worried about the performance - I was just checking to see if the bridging might be causing the drop in throughput - turns out comcast mucked up my connection - pfsense in transparent mode has no impact in my scenario (home use) I have a turnkey tool that does what I need it to and easy to get working and maintain - bloody brilliant in my book.  If you want to install a linux distro and install packages on that  etc etc more power to you i won't judge.  But that's not for me.  Having just learnt about security onion - maybe that's more suited to my need, thats where i will play and experiment next. So consider this just a journey of discovery for me - i now understand what pfsense is.  I have made no call on what I will finally do.  And if and when UBNT let me turn off the firewall / NAT on my USG device i will do that and likely revert the pfsense to non-bridged mode and use it as my NAT and firewall at that point.

Hey, looking to do the exact same thing. Just one question, will the server still be on the local network even though it uses a different wan network? Basically will I be able to use Plex server that is hosted on the server that is using a different wan connection? Thanks.

I have read it can be done with a standard firewall rule as per this thread: https://forum.pfsense.org/index.php?topic=112350.0 I will give it a shot when we get the second WAN!

@kapara: Did you look at the states?  Do you have s conflicting rule?  Why not force connections to OVH via a single gateway or via failover instead of both using same tier? Didn't look at the states which I will do now. I have tried in both NAT outbound and also a firewall rule, but obviously I don't know what I am doing!  If I want to force all outgoing connections to a certain domain (or a certain IP) to go via one WAN what is the best way?

Just make sure your rules specifying a gateway are only matching traffic you want to force to that gateway (group). Add rule(s) above that to pass traffic between internal networks.

You need to enable forwarding mode in Resolver, or default gateway switching.

IPsec does not route, so you can't use static routes. You need a separate Phase 2 entry for each distinct pairing of local and remote networks. The easiest way to reduce that is to summarize the remote networks. Are they all close by each other? Can you use a larger subnet mask to include all of them? Or at least reduce the number to something manageable?

I have done a lot of reading in regards to this issue, and pre-2.3 apinger was terrible and was often the issue. However, I had some pre-2.3 boxes working with failover, but there was no voip traffic on these boxes. This issue seems to be just for voip traffic. I have not been on site to do some more on depth testing. pfsense does not recognise that a gateway is down and does not switch. I can watch the state for voip traffic just sitting there and not changing. If I delete them manually, than it will failover to the second WAN connection but does not work automatically.

Also, on your identifiers, I usually manually type those, as different versions have captured and relayed this differently.

Try traffic shaping to within a megabit of your download. Most ISP's just drop packets once the max is hit, where as traffic shaping can be a much more smooth process. This is not a cure-all, softly queueing traffic might not drop it, but will add latency. Also, are you seeing any packet loss?

What do your phase2 entries look like per site? Do you have rule son the ipsec interfaces to allow such traffic?

@kapara: Are you doing 1to1 nat? With your IP for plex? Ah, I haven't tried that yet. Omg how did I not think of that…. I'll try it and post results here.

Seems that I solved the problem changing "State type" in "none" on the "Rule 1".