Just in case someone else wants to know, here's what I encountered and what I did to fix it.

What I wanted to do was add a private LAN address to each server so that it would appear on both networks at the same time. The problem with this (as listed above) is that only one default route exists.  If the default route is Net.A.1, traffic coming from the new ISP would have already been modified by NAT, so replies from the server would be useless.  If the default route is 172.17.2.1, traffic coming from the existing ISP was blocked by the firewall with a default rule.

I tried adding adding a firewall rule so that all traffic inbound on 172.17.2.1 with a source of Net.A would be rerouted through Net.A.1.  Unfortunately, this didn't work (I think because of the same default firewall rule).

My solution was to change the server to have a default route of Net.A.254.  With this, I can use the asymmetric routing techniques listed in Chapter 8 of "pfSense: The Definitive Guide".  I had to add specific firewall rules with State Type: none, but the setup works.  Now, the server appears on both networks and will respond to traffic from both networks until I can change DNS and confirm that all traffic has been removed from the existing ISP.

My WAN2 Firewall Rules look like this:
  Allow TCP/UDP from WAN2:any to LAN:any using default gateway
      … this allows traffic to flow freely for machines that have already moved to the new net.
  Allow TCP/UDP from WAN2:any to any:any using gateway Net.A.1 with State Type: none
      ... this allows traffic responses from Net.A to be returned to the Internet basically untouched.

Note that I'm not concerned about blocking traffic between networks because (1) I already have a traffic filter on the Internet router for Net.A, and (2) I will be removing WAN2 as soon as that network is no longer needed.

When I finish with Net.A, I can remove the IP address for that net from the server.  This will cause the default route to change to 172.17.2.1, and the server will continue to work properly with pfSense.

I hope this helps someone.  If you have any questions or just want to comment, feel free to respond or PM me.

Carl

You'll need to 'learn' the 192.168.53.1 pfsense box that it needs to use the 192.168.53.254 box as a gateway to reach 192.168.54.0.
I think a static route will do the trick.

I'm mostly dealing with embedded hardware, for the bandwidths I work with that is more than sufficient.
Depending on the WAN bandwidth and structure of traffic you'll need some more muscle. If you look in the hardware section of forum you'll find some threads discussing Hardware specs vs performance.

On lan1 have a reject rule with the destination to lan2 subnet and vice versa on lan2

der57,

First, let me say that I'm new to pfSense, so I may not provide the best information; but I've read the book and used the product enough to think that I understand this part (at least of how I have mine configured).

From my understanding, if you have a range of IP addresses .64/27 (usable addresses of .65 to .94).  The router to your ISP is using one of those addresses, the pfSense WAN interface is using another, and any 1-to-1 NAT you have configured are used as well.  The 2nd router's WAN can be configured for any unused address for testing.

If I'm way off, let me know.

Carl

Jimp is correct and you'd be wise to heed his advice.

It is far simpler to purchase some old AP's and run them as bridges as in this example.

http://www.cheesyboofs.co.uk/home.htm

Just purchase some old AP's, disable everything (wireless) and stick the pfSense wan interfaces in the DMZ's of each router/AP. You can pick up old 802.11b AP's for about £10 on eBay.

Yes. There is no limit to how many uplinks you can have, so long as they have unique gateway/subnet values.

Can you show the NAT, WAN and WAN2 rules? That way it is a lot easier to determine if its just a configuration problem or something else.

Like Jimp I use this on several pfsense boxes without problems.

Squid+multi-wan will not work in 1.2.x.

There have been reports of success on 2.0 after some recent commits by Ermal. I don't think we have a set of instructions ready to test though.

Must have been a typo on my part because now it seems to be working just fine!  ::)

Steve

No. If you route a new subnet, you do not use NAT or Virtual IPs at all. You really should start a new thread since your issue is not at all like the issue that started this thread.

Just make sure that your multiple WANs don't have the same gateway.
–> Each WAN needs it's unique gateway.

It should work fine, just make sure that all of the interfaces are bridged in the same way, i.e.

OPT1 bridged to WAN
OPT2 bridged to WAN
… and so on.

The info should be the same, yeah.

Not sure quite how you mean the second one. If the IPs are in use on WAN (like with CARP) they can't also be used behind the router. Though you could setup 1:1 NAT between one of the other CARP VIPs and one device on that VLAN, or you could setup outbound NAT and/or port forwards on a CARP VIP to let you access things on the inside.

Yeah, waste of resources. Since I have other server to deal with other services. And this is my dedicated pfSensebox running on a small Atom mITX setup.

So there is no other way than vmware?

Not easily in 1.2.x. In 2.0 it can be set on a per-gateway basis in the GUI.

Success!

I copied all the configurations (manually) from my 1.2.3 VMWare VM to a brand new 2.0B4 (Oct 8) VirtualBox VM, and the three ADSL connections are working perfectly fine.

Could it be that the problem it is not pfSense, but VMWare? I was using Workstation 7.1

In any case, it seems to be solved! (At last)

Thanks again guys, for this awesome piece of app!

glemDot

Yes, if I understand what you're asking, that should be quite feasible. It sounds like what you need to do is proxy ARP for your public scopes upstream, then push them downstream to each campus with a series of 1:1 NAT rules.

Regarding removing the pfSense nodes downstream, I'd be cautious about that. It's a good idea to have something segmenting the schools off from eachother downstream. Keep in mind that students are often brighter and more capable than school faculty, especially in technical matters, and should not be underestimated.

One final piece of advice would be to stage as much as you can before it goes live. Also might want to run some serious torture tests on the hardware/software stack you plan to deploy, ensuring that:

The hardware is reliable and won't be a bottleneck for the amount of traffic you're expecting, + predictable growth.

pfSense / FreeBSD is reliable enough on your hardware stack, and has all the features you need.

You know exactly what to expect in terms of configuration, backing up and recovering configurations (if the interface names don't match you're in for a fun time), etc.

Regarding the stability of pfSense / FreeBSD, I ran into some rather serious issues myself which essentially blocked me from deploying pfSense 1.2.3 in an overly-hostile environment. YMMV of course, but here's the record of my endeavors for reference: http://forum.pfsense.org/index.php/topic,24337.0.html