Thx for sharing your solution, but maybe somebody know it works with different public subnets?

Public WAN IPs
  xx.xx.1.1/29 gw xx.xx.1.1
  xx.xx.2.1/29 gw xx.xx.2.1
  xx.xx.3.1/29 gw xx.xx.3.1
Local LAN IPs
  192.168.0.0/16

Problem: to handle with NAT 1:1 from public to lan IPs

Replying to myself…
After having thought a bit more on how I wish my design to function, I realized that I need vIP (CARP) on each interface. That's fine.
Quite normal : internet client wants to talk to IP #1, it's not expecting a response from IP #2.

So, design v2 :
vIP#1 --> pfsense #1, ISP#1 master
vIP#2 --> pfsense #1, ISP#2 master
vIP#3 --> pfsense #2, ISP#1 master
vIP#4 --> pfsense #2, ISP#2 master
(the other box being the passive of each master vice-versa)

inbound nat is (sample):
vIP#1 TCP 80 (dst) --> server #1
vIP#2 TCP 80 (dst) --> server #1
vIP#3 TCP 80 (dst) --> server #2
vIP#4 TCP 80 (dst) --> server #2

outbound nat is (following same sample):
server #1 TCP 80 (src) --> WAN NIC #1 --> vIP#1
server #1 TCP 80 (src) --> WAN NIC #2 --> vIP#2
server #2 TCP 80 (src) --> WAN NIC #1 --> vIP#3
server #2 TCP 80 (src) --> WAN NIC #2 --> vIP#4
this being the same rules on both boxes

So, to give a practical example :

client wants to browse to vIP#3 reaches pfsense box #2 on WAN#1 translated to server #2 server #2 replies through pfsense #1 (master of LAN vIP) server #2's reply is through WAN NIC#1 as it's a TCP state already set in the state table (am I right here ?) outbound NAT as vIP#3 since it's server #2 and it is on WAN NIC#1

However I need to add a reverse rule on the LAN interface, allowing traffic originating from the server on the TCP 80 as src.

I need to test this further later on when I have enough vIP available on my secondary ISP (and there is another problem there, as they use static ARP entries in their systems... But that's another story), as currently I test on my primary ISP which is the default WAN for my pfsense firewalls. I will reply back here once it is confirmed as working on both WANs.

In the meantime, a question :
is this normal that the reverse rule is needed to be set up in the firewalls ? pfsync does not sync that info on the other nodes ?

Guillaume

@Shredder:

Sorry, should have said it was fo normal http: addresses. ie Google.

Can someone tell me how to set up the DNS rules for load balancing. At least I can try that and see if it might work.

Thanks,
Shredder

The same happens to me!
But I'm not sure it's a DNS issue: when this hangup happens to me if I do a ping to an "unchecked" site (a site I know but I haven't checked recently) reverse lookup of the address is done and I get numerical IP but I'm unable to contact the site.

At the moment the only "solution" I've found is to use just one connection and shut down the WAN2 modem.

It's possible that your provider has a traffic shaper in place that allows short bursts of upload of more than 2Mbit.
The other possibility is, that this speedtest uses more than one connection to test the upload. (Probably not).

It won't make a difference if you add two more routers.
This would only be an issue if you have two connections from the same provider which gives you two IPs out of the same subnet with the same gateway.

What you can try: Start multiple speedtests at once.
The sum of all speedtests should be 60/4.

I've never used a router with no nat  :-\ so I guess that makes me useless.

So just looking at one subnet, 10.10.x.x: lets say the routers lan is 10.10.0.1 and its wan is 10.3.0.101.

You have a static route for 10.10.x.x pointing to 10.3.0.101 and can ping a pc (say 10.10.1.50) from pfsense so you know that basic routing is working.

So then you would still need some firewall rules:

question though, Do you need both of these for 10.10 pings to go out and back from the dmz? or is the second one only needed to ping from the DMZ?

Lan Rule:  proto icmp from 10.10.0.0/16 to 192.168.1.0/24
and
DMZ Rule:  proto icmp from 192.168.1.0/24 to 10.10.0.0/16

sadly I spent a fair amount of time using ping for testing while my rules were set for tcp  :-[

Well, I am pretty sure that I was testing it all wrong.

I will certainly post back if I have more troubles, but things are starting to look pretty good.

I wanted to pass along something that Hoba posted:

Advanced outbound nat does not determine where the traffic gets routed. That is done with the firewall rules. AON only adds the natting when it gets routed out through the one or the other interface. You have to check your firewallrules or your loadbalancer status to see [what] is happening.

And [sticky connections] will keep a client on one wan until all it's states have expired

I found that quite informative, because I was having troubles getting this all straight in my head.

The only real solid advice I can offer at this point is: Try to get a good nights sleep before trying to figure this all out  ;D

let me try and rephrase this.

on my LAN side of my firewall i have a subnet 172.16.0.0/24

i'm using OSPF and there are multiple paths to the 172.16.0.0/24 subnet.

when the firewall sends traffic destined for the 172.16.0.0/24 subnet i want to load balance between 172.16.1.1/29 and 172.16.2.1/29 as the gateway for the 172.16.0.0/24 subnet.

make more sense?

How would I specifically know that I'm having connectivity problems with the NICs?

Would turning off hardware checksum offloading under advanced turn off this feature for ALL NICs?

Thanks for your guidance,
I have followed it,and I can get ping of the gateway of WAN2  now.
I have set the PPPoe in the router,and set it as WAN2.
but after I can not connect to the internet after stop WAN.

Any advice?

@stevemitchell:

Yes, I inherited this mess from another person who is no longer with the company. We have gotten it down to a list of 10 or so routes of larger subnet masks, which should be better than 100 :)

Oh those are always fun. Well, at least you got it down to 10.

@stevemitchell:

Also, I bought the book last week and read the entire thing this weekend.  Well done - and I did see the CIDR details in it as well.

Great, thanks!

Lan and wan are bridged. I just turned off the advanced option "Bypass firewall rules for traffic on the same interface" Did not help.

What I have found out is that my Mcafee firewalls on our clients have started treating its own subnet as an un-trusted network. I do not know if this is a coincidence it happened the same day I installed the pfsense firewall. To fix it I had to manually put in the IP address for the trusted network on each machine.

That works! I don't know why I didn't think of that. I did have a lot of ports to forward, but aliases make it a little simpler. I control which gateway is used for the servers with a LAN firewall rule like all the user systems now. I tried using outbound NAT rules, but it seemed to break everything as soon as I enabled it. I can't create firewall rules to use a virtual IP as the outgoing gateway. I guess I will keep a 1:1 NAT mapping for my mail server. Obviously the mail server needs to send mail from the IP which its domain resolves to, which I think was why I started using 1:1 NAT to begin with.

@eihcet:

Lastly, has anyone purchased the PFSense book, is it mainly a hardcopy of the WIKI guides online or does it expand upon items like this?

The wiki guides aren't all that great, they were written by outside contributors. The multi-WAN coverage in the book is extensive, and written by the person who has done more of those setups than anyone (i.e. me  ;D), it's far and away better than anything freely available.

@eihcet:

However, when it gets to the gateway section I can only select gateways from the drop down list, of which none are the right option and I can't manually type in a gateway address… My choices are:

Default
10.1.10.1
LoadBalancer
Wan1FailstoWan2
Wan2FailstoWan1

I've got it working now using a failover rule "WanXFailstoWanY".  The guide makes it seem as though you can specify just the one gateway and if there is a failover it'll just block the traffic

The individual gateways as you have them configured are there, 10.1.10.1 would be your OPT WAN, and default is your WAN. If you pick one of those it will behave as you describe. If that gateway isn't right, your interface isn't configured right.

There isn't any way to properly accommodate shaping in that scenario at this time.

You could, with caveats. Reconnection wouldn't be guaranteed to get the same server if the previous connection was closed and expired from the state table. It's best to use either something like Citrix for that, or MS NLB, but it should work.