Solved - it works as I needed. Now I have 2-port switch on my WAN side.

It should be possible. On PFS-A, add a route that states that all traffic bound for 10.2.0.0/16 goes to 10.10.0.2. Create a rule in LAN that states that anything going to 10.2.0.0/16 goes through that new gateway. You will need to put that rule above all rules. The in the OPT1 rules tab, set a rule that allows everything to anywhere.
You will want to do the same thing on PFS-B but using the PFS-A LAN subnet. You might need to restart to make sure there is not anything left in memory that would not allow this to work. On the OPT interface rules, you will want to change the default behavior to keep states. You will not want to keep states on these since it should just be routing. On the LAN rules that use the new gateways, you will also want not to keep states.
Keeping states will use unnecessary memory space.

Use this: http://blog.stefcho.eu/?p=754

We configured OPT2 which is connected to a gigabit switch.  Into the switch we have connected a wireless access point for guest wifi.  In this scenario, the "guest" network is on a differnet subnet, therefore isolated from our private network by the router.  Any wifi connections on the WAP, or computers patched into the separate switch are therefore isolated.

Otherwise SteFcho's setup worked great.  Just make sure you choose a subnet that you're not using for VPN…a mistake I made :-)

@luckman212:

If I do not wish to receive alarms for high latency but DO want alarms for high watermark – is it OK to set both values to the same number?

Probably. Or just set it to 100 and 101 so that there is just a very small difference

@cmb:

AFAIK that should work without any additional considerations. Though I haven't tried it, nor recall of anyone trying it. Give it a shot and let us know what you find.

That's the plan, once I get to it.

A quick update, moving the icmp filter above the tcp/upd filter resolved the problem and now traffic goes both ways.

How for another twist, I added a second IPSec tunnel to 10.66.102.0 and now I'm back to the same type of symptoms on this link. But a little more strange ….

From a workstation at 192.168.1.2 I can ping 192.168.6.2 all the time, if I stop that and ping 10.66.102.2 the ping never gets through the VPN.... Now if I start a ping from 10.66.102.2 and let it run for a while, then all of a sudden the reverse ping starts to work....if I stop the reverse and ping 192.168.6.2, stop it and then ping 10.66.102.2 it doesn't complete, and then after about 30 - 40 secs.. It starts again

Could this be an arp, routing problem, or a IPSec tunnel problem ?
Ken

http://doc.pfsense.org/index.php/Multi-WAN_2.0#Policy_Route_Negation

tl;dr version: yes, make a firewall rule from LANx to LANy with no gateway and place it above the failover rule. Do the same from LANy to LANx.

it depends. Do you have a NAT entry applying to traffic on the ACL line posted?

The monitor IP is the next-hop address (public IP of my ISP).  So it is on the other end.  Monitoring is not disabled (default).  And it appears down when I remove the link (I checked Status > Gateways).

I already chose "packet loss or high latency".

The state killing option is not disabled (default).

Thanks for the help but I already checked all this.  I couldn't think of anything else and that's why I asked for help here.

As I said before, the weird thing is that it did kill states when I changed the configuration of my interface from dhcp mode to static mode (the whole configuration is exactly the same, except for the mode).

Not sure about your squid and such, that is probably because packages on 2.1 still need work.

The rules and failover should be approximately the same.

As for the depreciation messages, we're aware: http://redmine.pfsense.org/issues/2114

It's completely relevant. The use of load balancing or failover for Multi-WAN is governed by firewall rules. Traffic doesn't pass "through" the load balancer in the way you imply. It also doesn't adjust itself based on traffic load.

You just need multiple Gateway Groups, one for LB, and one that prefers each WAN.

Then use firewall rules like so:

pass from (lan subnet) to stuff_for_wan1 using the PreferWAN1 gateway group
pass from (lan subnet) to stuff_for_wan2 using the PreferWAN2 gateway group
pass from (lan subnet) to any using the LoadBalance gateway group

The bandwidth of a client connection has nothing to do with how it's balanced between WANs. Balancing happens in a pure round-robin manner, adjusted by the weighting configured for a gatway, on a per-connection basis. It can't tell the difference between a download and a web page load, it would just send them out whatever WAN was due to be used at the time.

If you have the default weights configured, and everything load balanced, you'd see something like this:

Connection 1 -> WAN1
Connection 2 -> WAN2
Connection 3 -> WAN1
Connection 4 -> WAN2
Connection 5 -> WAN1
Connection 6 -> WAN2

Yes,if you don't set a gateway it will try the default. It should switch with failures as you mentioned. However; unless there is some dyer need for using ISP assigned DNS server, I would use a national one or public one. I would even go so far as to setup 2 of my own before using so many ISP based DNS servers.

Best to use something on the Internet, or at least on your ISP's network like their DNS servers as you're doing, to actually measure connectivity through your connection not just to a router that's probably sitting right by the firewall depending on what type of connection you have. Yes when your connection is under load near or at its limit, latency and jitter are going to increase, that's just a fact of networking. Where it's most useful is seeing significant loss or high latency or jitter when it's not near its limit, which is helpful to show problems with your connection, especially in the past if it's something that happens overnight or otherwise at a time when you can't catch it while it's happening.

The rules here say wait at least 24 hours to bump your post. creating noise isn't helping your cause.

Coming up as 1000 full seems sane. You don't have an option to change the media because you're looking at the VLAN not the parent.

no one ? pls help me

For you to get MLPPP, they have to give you MLPPP. If they don't set it up correctly on their end, then it won't work on your end.