Perfect, thank you.

hm, this is more an architectural question than a technical. if you go for one firewall with all 5 networks and alias interfaces and nat-ing, you get it all in one place, but are also creating a single-point-of-failure (that can also fail upon mis-configuration) on the other hand, visualization with vmware or similar has become quite stable, why not running 5 pfsense installations in parallel? just hook all WAN-if's into the fiber, and then lan-patch-up your customer with or without vlan's, that depends on how the cabling can be done on-site. bandwidth-mgmt becomes then a bit tricky, but getting-started is much more easy, especially if you're new to these technologies. when you're settled you still can change the architecture later on and implement what you learned. going gold is platinum!

Sometimes people actually want the traffic blocked if the gateway is down (e.g. they have a VPN uplink and do not want traffic to go anywhere (on the unencrypted WAN…) when the VPN is down) so it all depends on your requirements. If you want traffic to fail over to another WAN, then you are better off making a gateway group containing the required gateways with required tier1/2... selected. Then use the gateway group in your rules. Default gateway switching does work work for ordinary pfSense configurations that have 2 ordinary WANs - in that case there is ambiguity about what to do when the default gateway is down. I don't think that the "Mark Gateway as Down" setting is implemented in all places in the code! I suspect that if you have Default Gateway Switching enabled and then set "Mark Gateway as Down" on your default gateway, it probably won't switch the default gateway - but that would be good to try...

If you can identify the uploads in some way (e.g. have static-mapped IP addresses for some LAN clients where you do the uploads, or if it is big attachments on sending emails then match the SMTP… port number/s) then you can make policy-routing rules that will match that traffic and put it on the WiMax link.

In my experience you absolutely must enable the "Enable default gateway switching" option if you wish to receive email alerts from pfSense about Gateway failures. Unless I have missed something, policy rules are not applied to traffic coming from pfSense itself.  So, even if you have routing groups set up, they will be ignored for SMTP alerts from the router and so if the primary GW goes down, you won't get an alert unless pfS can switch its default GW.

I'm not sure I understand why the packets couldn't find their way back to the 172 network since the communications are always initiated from the 172 network.  I assume it is something to do with the NAT that the Internet router does. I guess your pfSense firewall was not applying NAT to the traffic from 172 as it exited to 10.0.0.1 - and so the router at 10.0.0.1saw the source IP as 172.0.0.2 but when trying to reply it had no route to there. Most routers (and even stateful firewall/routers) do not remember specifically where incoming state/flows came from in order to reply - they just use their own routing table to reply.

Answered! As always seems to happen I post in desperation but continue poking around.  The "fix" is to go into your LAN rules under firewall and scroll all the way to the bottom. Under:  Gateway > Advanced > drop the menu down to your gateway group > save and apply. That was the fix to push all traffic through the gateway group rather than the "default" or default gateway. I also noticed since I have ipv6 disabled that if I had any FW rule with ipv6 in it I didn't have the gateway group available until I made only an ipv4 rule. Dang!  Hope that helps someone.

@pdrass: I was just browsing this for a solution to my problem unrelated to your issue BUT I don't think Allworx can do WAN and LAN port at the same time regardless of what IP's you put on them.  I had this going on and it was a mess until we unplugged the WAN and all was well.  Supposedly the Allworx can be your router but I think it's a phone system not a router and thus, should do one thing and one thing well…phones! Moreover, even if you wanted to do dual WAN - the Allworx phones on the other end can't connect to multiple public IP's as their voice gateway...or whatever they call that option on their desk phones so doing dual WAN is a moot point. It has been a while since I touched one so all the above is from memory if it serves me right. The provider says it's possible to dual WAN an allworx, but they said the only device that allows it is some device they sell. I've never seen it before and cannot remember the name.

Set-up a firewall rule so any traffic coming from the LAN uses the WAN1 Gateway. I've done this with my DMZ VLAN, all other traffic flows out of a separate interface using the default WAN gateway. I've tested this and as soon as I connect to the DMZ with my computer my external IP address changes immediately to the DMZ_WAN address I've set-up from my normal default public IP.

@ash45: i have a multiwan setup with load balancing and fail over both of these work perfect But not really needed, because if one of the WAN interfaces is dying or fails the entire traffic is routed through the other WAN interface, so you got fail over on top by using load balancing. And there are three main possibilities to load balance. session based load balancing service based load balancing policy based routing So that would be enough to solve the problem, or?

Am I just weird… or??? Everyone will notice that internet is down. Why do they need a special message telling them the absolutely obvious fact?

Yes, a default Allow All rule is automatically added for LAN, but not for any other local networks like OPT1, OPT2 etc.  You have to add it manually.

@MnM: I wills soon have 2 ISP at my place. I have 2 separate subnets that do not talk to each toher at all. Ok What is the best option to allow subnet 1 to have internet via ISP1 and subnet 2 via ISP2? Set up dual WAN laod balancing together with policy based routing! Set up two VLANs and sort each with his own subnet.

I was hoping someone could help me with finding an answer to a problem I am having with my pfSense failover. The failover is working proper as you mentioned, or am I wrong with it? The situation is that I have my primary WAN operating, I also have my secondary WAN connection set as a failover connection. Ok, but why not using load balancing together with policy based routing? If I drop my primary WAN and the secondary kicks in, I my phones that connect to an external SIP server wont register any longer. Is this the WAN connection from the ISP that offers also the VOIP line? Is this a STUN Server or the SIP Gateway from the ISP? When I reconnect my primary connection my phones still dont recover untill I disconnect the Secondary WAN and reset the States table. For sure it would be better to know for us who is offering the VOIP service and Who the Internet services! Please if anyone could offer some guidance here I would really appreciate it. I would try the following out, load balancing together with policy based routing because if one WAN is failing the entire traffic will take the other WAN port only, so you get also a failover but without the hassle of a unused WAN line. Pointing something to the SIP problem must be stated because on the poor informations you where pulling out about.

@artl: Hi, ive a netgear 4G usb Hi what do you mean with netgear 4G usb? A router or a USB 4G Stick or what? which assign a private ip 192.168.1./24 to a billion router. Would be so friendly and draw a small network schematic for us and a better understanding, please? Or is this in first a Netgear Router and then behind a Billion router? (router cascade oder double NAT) And is this a typo, 192.168.1.0/24 ive set 4G netgear to DMZ 192.168.1.4(Billion on usb connection) then Billion to DMZ * billion DMZ to a laptop on its wifi works. billion DMZ to cat5 pfsense dont? any ideas anyone? And where is there a question about pfSense?

some ISP's dont like it when people use other dialers except their own… spoof your cables modem MAC address to pfSense and try it ... maybe you will get lucky otherwise its propably a routing issue post lan / wan details....

The problem you are seeing may be related to apinger.  Search these forums and you will see a LOT of apinger problems with 2.1.x.  This functionality has been improved in pfSense 2.2.  You might want to look at upgrading.