That would most likely be a good choice, no need for static routes, the system will do that for you. Check with traceroute if you can find another closer upstream address that responds to icmp

No no, you understood it correctly. Currently, in Jordan, there is a monopoly in place where all ISPs have to use the same carrier, JTC-Orange, and to protect their monopoly, they have all the modems locked, now to add to that, it is quite difficult to obtain a modem locally.

Add 2 rules over the default one dest port: http gateway: wan2gateway dest port: https gateway: wan2gateway

Hmmm, well that is what I thought, but it in practice it is employing the top one which is the first one I enter, not the second one so I have to do it the opposite of what is shown.

Yes

Select the interface on which traffic should go out for the route We assume in our case it would be our wan2? Yes Enter in "Destination network" the IP you want to always send to WAN2 Would this be our dns server for wan2? Yes Select as subnet /32 if you want only a single IP. We assume /32 here since it is only one dns server that it is going to? Not exaclty. In CIDR notation you express always subnets. A subnet of a single IP is /32. If you had 4 DNS server next to each other (x.5, x.6, x.7, x.8 ) you could express them as a single CIDR subnet x.5/30 –> x.5 to x.8 However if these 4 DNS servers were spread (x.5, x.10, x.15, x.20) you would need 4 separate expressions: x.5/32, x.10/32, x.15/32, x.20/32. Enter in Gateway the IP of your WAN2 gateway. Would this be 192.168.1.1 or 192.168.1.250 or 69.4.xxx.yyy? -From ISP 2, we have the address of 69.4.xxx.yyy which is connected to a standard Dlink router. The router is setup to have the address of 192.168.1.1 and pfsense a static ip of 192.168.1.250 (which is our DMZ ip). ISP DNS server is 64.33.128.10 and is pingable. Since the pfSense has as IP 192.168.1.250/24 (i assume it's /24), the immediate gateway for it is 192.168.1.1. –> The next router directly connected to the pfSense.

This is pretty much what i have running (multiple clients, multiple servers, dual wan, behind NAT). @bob76535: (both connections are from the same ISP but the IPs are not sequential if that matters) Only thing that could be confusing: if both links are on the same subnet/router. The rest should be no problem for pfSense.

Yes.

Thx for sharing your solution, but maybe somebody know it works with different public subnets? Public WAN IPs   xx.xx.1.1/29 gw xx.xx.1.1   xx.xx.2.1/29 gw xx.xx.2.1   xx.xx.3.1/29 gw xx.xx.3.1 Local LAN IPs   192.168.0.0/16 Problem: to handle with NAT 1:1 from public to lan IPs

Replying to myself… After having thought a bit more on how I wish my design to function, I realized that I need vIP (CARP) on each interface. That's fine. Quite normal : internet client wants to talk to IP #1, it's not expecting a response from IP #2. So, design v2 : vIP#1 --> pfsense #1, ISP#1 master vIP#2 --> pfsense #1, ISP#2 master vIP#3 --> pfsense #2, ISP#1 master vIP#4 --> pfsense #2, ISP#2 master (the other box being the passive of each master vice-versa) inbound nat is (sample): vIP#1 TCP 80 (dst) --> server #1 vIP#2 TCP 80 (dst) --> server #1 vIP#3 TCP 80 (dst) --> server #2 vIP#4 TCP 80 (dst) --> server #2 outbound nat is (following same sample): server #1 TCP 80 (src) --> WAN NIC #1 --> vIP#1 server #1 TCP 80 (src) --> WAN NIC #2 --> vIP#2 server #2 TCP 80 (src) --> WAN NIC #1 --> vIP#3 server #2 TCP 80 (src) --> WAN NIC #2 --> vIP#4 this being the same rules on both boxes So, to give a practical example : client wants to browse to vIP#3 reaches pfsense box #2 on WAN#1 translated to server #2 server #2 replies through pfsense #1 (master of LAN vIP) server #2's reply is through WAN NIC#1 as it's a TCP state already set in the state table (am I right here ?) outbound NAT as vIP#3 since it's server #2 and it is on WAN NIC#1 However I need to add a reverse rule on the LAN interface, allowing traffic originating from the server on the TCP 80 as src. I need to test this further later on when I have enough vIP available on my secondary ISP (and there is another problem there, as they use static ARP entries in their systems... But that's another story), as currently I test on my primary ISP which is the default WAN for my pfsense firewalls. I will reply back here once it is confirmed as working on both WANs. In the meantime, a question : is this normal that the reverse rule is needed to be set up in the firewalls ? pfsync does not sync that info on the other nodes ? Guillaume

@Shredder: Sorry, should have said it was fo normal http: addresses. ie Google. Can someone tell me how to set up the DNS rules for load balancing. At least I can try that and see if it might work. Thanks, Shredder The same happens to me! But I'm not sure it's a DNS issue: when this hangup happens to me if I do a ping to an "unchecked" site (a site I know but I haven't checked recently) reverse lookup of the address is done and I get numerical IP but I'm unable to contact the site. At the moment the only "solution" I've found is to use just one connection and shut down the WAN2 modem.

It's possible that your provider has a traffic shaper in place that allows short bursts of upload of more than 2Mbit. The other possibility is, that this speedtest uses more than one connection to test the upload. (Probably not). It won't make a difference if you add two more routers. This would only be an issue if you have two connections from the same provider which gives you two IPs out of the same subnet with the same gateway. What you can try: Start multiple speedtests at once. The sum of all speedtests should be 60/4.

I've never used a router with no nat  :-\ so I guess that makes me useless. So just looking at one subnet, 10.10.x.x: lets say the routers lan is 10.10.0.1 and its wan is 10.3.0.101. You have a static route for 10.10.x.x pointing to 10.3.0.101 and can ping a pc (say 10.10.1.50) from pfsense so you know that basic routing is working. So then you would still need some firewall rules: question though, Do you need both of these for 10.10 pings to go out and back from the dmz? or is the second one only needed to ping from the DMZ? Lan Rule:  proto icmp from 10.10.0.0/16 to 192.168.1.0/24 and DMZ Rule:  proto icmp from 192.168.1.0/24 to 10.10.0.0/16 sadly I spent a fair amount of time using ping for testing while my rules were set for tcp  :-[

Well, I am pretty sure that I was testing it all wrong. I will certainly post back if I have more troubles, but things are starting to look pretty good. I wanted to pass along something that Hoba posted: Advanced outbound nat does not determine where the traffic gets routed. That is done with the firewall rules. AON only adds the natting when it gets routed out through the one or the other interface. You have to check your firewallrules or your loadbalancer status to see [what] is happening. And [sticky connections] will keep a client on one wan until all it's states have expired I found that quite informative, because I was having troubles getting this all straight in my head. The only real solid advice I can offer at this point is: Try to get a good nights sleep before trying to figure this all out  ;D

let me try and rephrase this. on my LAN side of my firewall i have a subnet 172.16.0.0/24 i'm using OSPF and there are multiple paths to the 172.16.0.0/24 subnet. when the firewall sends traffic destined for the 172.16.0.0/24 subnet i want to load balance between 172.16.1.1/29 and 172.16.2.1/29 as the gateway for the 172.16.0.0/24 subnet. make more sense?