I had a similar scenario except with outbound SMTP instead of VoIP. I was missing part 1 of 2. I would configure the OB NAT, but the connection wasn't going through at all. After adding the OB Firewall Rule, everything started working properly. I also noticed that when making changes to the NAT rules (and possibly others), it takes a couple seconds before they actually get put into effect.

pfSense: Remote Network Point to Point Link to VPN Over DSL Failover The scenario: You have two locations with Internet connections and a dedicated point-to-point connection between the two and two pfSense systems performing all routing at both sites.  You desire the two sites remain connected should the dedicated connection fail. The solution: Create a pfSense configuration with failover from the point-to-point connection to a site-to-site VPN utilizing the existing Internet connections at each site. Steps: 1.  Create an OpenVPN Server on the main pfSense and Client setup on the remote pfSense (I used pre-shared keys).  DO NOT set a route option in the Advanced box as most instructions for configuring OpenVPN will suggest nor should you have a static route to your remote network defined under System -> Routes.  Also note that  IPSec can not be used in this scenario as it doesn't create a new adapter that we can work with in the firewall rules and gateways. 2.  Check and see that the VPN turns on and connects via Status -> OpenVPN before proceeding.  If it does not then troubleshoot your Internet connectivity and OpenVPN settings. 2.  Go to Interfaces -> Assign and add Interface OPT3 with Network port ovpns1 on both the server and the client pfSense systems. 3.  On both your local and remote pfSense add a new Firewall Rule allowing all protocols from any source to to any destination under both OPT3 and OpenVPN. 4.  On both your local and remote pfSense add OPT3 as a Gateway under System -> Routing -> Gateways leaving the Gateway and other options blank. 5.  On both your local and remote pfSense create a new Group under System -> Routing -> Groups.  The group will define your dedicated connection as Tier 1 and OPT3 as Tier 2.  My trigger level is set to Member Down. 6.  On both your local and remote pfSense create a new Firewall Rule under LAN which has all traffic from all sources bound for the remote network use the new Gateway Group (under Advanced) you created in Step 5. 7.  Test - unplug the point-to-point connection, monitor things under Status -> Gateways, wait a minute or so, and hopefully you will still be passing traffic albeit through the VPN.

Eureka!  Thanks, Heper! Here's the guide I've promised: The scenario: You have two locations with Internet connections and a dedicated point-to-point connection between the two and two pfSense systems performing all routing at both sites.  You desire the two sites remain connected should the dedicated connection fail. The solution: Create a pfSense configuration with failover from the point-to-point connection to a site-to-site VPN utilizing the existing Internet connections at each site. Steps: 1.  Create an OpenVPN Server on the main pfSense and Client setup on the remote pfSense (I used pre-shared keys).  DO NOT set a route option in the Advanced box as most instructions for configuring OpenVPN will suggest nor should you have a static route to your remote network defined under System -> Routes.  Also note that  IPSec can not be used in this scenario as it doesn't create a new adapter that we can work with in the firewall rules and gateways. 2.  Check and see that the VPN turns on and connects via Status -> OpenVPN before proceeding.  If it does not then troubleshoot your Internet connectivity and OpenVPN settings. 2.  Go to Interfaces -> Assign and add Interface OPT3 with Network port ovpns1 on both the server and the client pfSense systems. 3.  On both your local and remote pfSense add a new Firewall Rule allowing all protocols from any source to to any destination under both OPT3 and OpenVPN. 4.  On both your local and remote pfSense add OPT3 as a Gateway under System -> Routing -> Gateways leaving the Gateway and other options blank. 5.  On both your local and remote pfSense create a new Group under System -> Routing -> Groups.  The group will define your dedicated connection as Tier 1 and OPT3 as Tier 2.  My trigger level is set to Member Down. 6.  On both your local and remote pfSense create a new Firewall Rule under LAN which has all traffic from all sources bound for the remote network use the new Gateway Group (under Advanced) you created in Step 5. 7.  Test - unplug the point-to-point connection, monitor things under Status -> Gateways, wait a minute or so, and hopefully you will still be passing traffic albeit through the VPN.

See http://forum.pfsense.org/index.php/topic,38882.0.html and make certain you have followed all of the steps.  The floating rule and "tcp_outgoing_address 127.0.0.1" are important pieces of the puzzle which are not immediately obvious.

Hi, See my other post, I've already found half of the solution : http://forum.pfsense.org/index.php/topic,54763.0.html Now my LAN can access my DMZ by public IPs.

Pretty sure Hulu uses a single TCP connection to stream, so no multi-WAN load balancing will improve that. Short of options like bonding the connections that require the cooperation of your ISP (MLPPP, BGP, etc.).

That's less than ideal, but that'll be fine if you have two NICs going to your ISP. With the normal ways of assigning multiple public IPs you only need one NIC, but not in that case.

THANK YOU, for that answer. If its correct, I trust it is, it will help me tremendously in my trouble shooting.

Seems solved!  ;D On the left side, CONSERVATIVE mode for firewall…. on the right side, MTU on the WAN interface needed to be configured at 1492 for a PPPoE ADSL line. Hope not to encounter new problems...  8)

Dear all, today I try to use different weight on gateways, 3 for DSL ( 12mbps down / 1mbps up ) and 1 for sdsl ( 4/4 mbps ). It's working, but not the best way to make a good load balance. We can't have any control on all users ( HTTP upload ) Thanks for your help Armel

As Metu69salemi said, your firewall rule is NATing your public IP block out your WAN interface.  Change the Source to your LAN subnet.

This instruction works (with squid):  :) http://forum.pfsense.org/index.php/topic,38882.msg200321.html#msg200321 I tried several other ones (like from DimitiS) but these did not work for me. Now it's up to dansguardian.

Exactly how did you set up the rule? Aren't you having a problem with rule order? Consider that rule has to be before any "allow all" rule that catches all traffic, otherwise just gets routed as per THAT default rule (which, if you didnt specify a gateway would be as per the system routing table) Anyway, in your particular case I would add a static route instead of using policy based routing. Regards!

There are others who have seen that (and some that see it every time their PPPoE goes down), I've seen it myself too but I can never replicate it in a repeatable way. I've tried unplugging the nic, unplugging the DSL line, power cycling a bunch of times, etc. But it never fails for me when I do any of that. I've had my DSL line be down (on the provider's side) for 24+ hrs and it comes back fine, but then some random time a month or two later it'll go down for a couple hours and never come back up. If you search around a bit here on the forum and on redmine you'll find several others talking about it. The typical subject is "PPPoE won't reconnect after link loss" (https://redmine.pfsense.org/issues/1943)

PS good idea to look at specific/dedicated hardware solutions too. I looked at Mikrotik Routerboard, you can get some amazing hardware for not very much money. However my testing of multi-wan on my PC didn't go well and I found it was complicated to use mikrotik Routeros, so I gave up on it … advantage of Routerboard is price and things like that it will run off a bog standard PSU 10-24v if I remember. They do cases to fit their stuff so would be compact also. Cheers

I've also been looking into this, but the confusion marketing on net takes a long time to find useful straightforward information. Locally in Thailand I have a number of public wifi hotspots (pay as you go), so I've setup my spare laptop with pfsense. It works fine for load balancing if I'm uploading in separate applications. Problem comes when I want to use the bandwidth for one application that doesn't support multiple routes/connections. Also it's hard to get any decent upload bandwidth here without paying a king's ransom, which is why I started looking into this in the first place. So something like this :- Hotspot 1 –-> laptop USB ---> WAN1 (pfsense VM) Hotspot 2 ---> laptop PMCIA ---> WAN2 (pfsense VM)                    -----> internet Hotspot 3 ---> POE station adapter wired ---> WAN2 (pfsense VM) etc With public hotspots my traffic is often on open air, and I have a little understanding of just how unsecure this is, which is very not good ! Solution from what I've read is to setup a VPN, multiple tunnels, but what VPN provider. I could use a VPN provider, assuming they will allow multiple connections - easiest solution. Or I could use a VPS service with pfsesne -- pfsense connection. More complex solution I'm not sending government top secret info, so don't need highest level of encryption, just something that fairly hard to crack and put off most attempts, so L2TP or PPTP would be OK I assume. Also I want to use my unjailbroken iphone so doesn't make sense to use openvpn. Also, to add another factor into the equation, I could use VPN in the OS on any of the machines that I connect with, which actually would be even simpler than setting up pfsense for VPN. Simplicity is better in my experience, make it too complicated and it's hard to maintain and difficult to maintain and operate. Any ideas ? Kind regards

Good job!  Thanks for the update and the words of advice.  Much appreciated. Aaron