Yes, I normally use gateway groups for the clients, because also that way you can load-balance the traffic, and/or send some particular traffic out WANA and other traffic out WANB. Default gateway switching is only good for dumping everything from primary-WAN to other-WAN. And as you say, if primary-WAN is down then often you do not care so much that the pfSense version check, package download etc do not work for that time.

In order to force it to change the default gateway back to OpenVPN, I had to mark the WAN gateway as down and back again as up (ofc marking it as up was not needed to make the default gateway change but I just wanted it to be up). But the same thing does NOT work for forcing it to change from the OpenVPN gateway to the WAN gateway.

Any hint in the right direction whould be great. Is my explanation of our configuration clear enough or can I do anything for others to better understand what I'm trying to achieve ? Best Regards, Daniel Bielefeldt

I guess you just repeat similar stuff on GW02. Make VIPs for some IPs in that middle subnet 10.0.1.0/24 and GW01 will be forwarding to those, then forward those onward on GW02 to the server(s) behind GW02. Firewall rules will need to be for the appropriate destination IP address at each stage, as the concerned router sees the packets after NAT.

@azekiel: no, never got any suggestion or solution :( Actually everithing in 2.2 is working well, with load balancing, squid and squidguard, here the procedure I follow to get everithing working: https://forum.pfsense.org/index.php?topic=88826.0 but other references are inside. Hope this will help.

@jaspras: I see that your TATA connection is on half dublex that's probably your problem. Normally it should be full dublex Check also if you need cross eth cable between your modem and FW WAN interface (or put a small switch between your modem and pfsense)

OK, here's the final update! After going through the VPN forum, the general consensus was the IPSec in version 2.2 is somewhat broken, so we changed the VPN settings to OpenVPN. We tested the failover, and it switched over to the failover port and re-established the VPN connection in about 15 seconds. All is good! (Now I'm regretting getting the dynamic dns service set up, but anyway) So I'll mark this as officially solved, thank you very much for all your support. Best regards.

If they obtain a range of public facing IP's from the ISP, Yes you can do exactly that (ISPs usually charge for a block of Public IP's to use.) Then you could in theory add the extra public IP's to the "Virtual IP's" section of PFSense, and then create 1:1 NAT Routes and Firewall rules to each tenant's router to break the IP down into the more common 192.168 or 10.10 style subnetworks. This would be of benefit if the tenants need to do anything that would require communication Back into their private networks (running servers, remote desktop access, etc.) You could technically also keep everything in a local network setting using a managed switch that supports VLANs Ex: Public IP's > PFSense with 1:1 NAT + Routes > Switch > Tenant Routers > Tenant Computers (*More Complex) OR Single Public IP > PFSense with VLANs > Switch w/ VLANs > Tenant Routers > Tenant Computers (*Less Complex) My knowledge is by far basic on this but either way is doable, And it depends on one factor, Do the tenants need to run a server or do they *Need a static IP thats public facing? Because if not, Just get a decent multiport managed switch and do VLAN's to isolate each tenant while still sharing the single public IP the landlord is already assigned from the ISP. As for traffic shaping, Yes, across the board… You can use Traffic Shaper to create limits that are applied via Firewall Rules. to throttle clients as much or as little as you would like. One last word of caution, Have the client make REAL sure that his ISP contracts allow him to sublease the connection to other parties. Paid or not... Some can be a real stickler about what you do with the connection you pay for. (its lame, but it happens...)

Unfortunately, i don't have simple "Pass all" rules - there are 10-20 rules per interface, which can lead to errors if either the Failover gateway isn't set on LAN1 / LAN2 (traffic would not fail over) or GW1 isn't set on LAN3 (traffic would leave the wrong interface). Make your rules correct and it'll work.  It's probably easier and less error-prone to have everything related to traffic from a LAN interface in one place than to have it in multiple places.

I'm going to assume from your post that you are not using VLANs and also have all of this equipment hooked up to a switch attached to your pfSense box via the same LAN interface. In that case, what I would do is create a container (Firewall > Aliases > IP) for your phone and security systems. Enter their IP addresses and save the alias as "t1devices" for example. Now create 2 new gateway groups (under routing) The first group can be called "T1primary" and should contain the T1 as Tier 1 and optionally your Opt1 gateway as a Tier2 if you want failover The second group could be called "Opt1primary" and would be Opt1 = Tier1 and again, the T1 as Tier2 if you want your PCs to be able to fail over to the T1 if opt1 goes down. Now, go to Firewall > Rules > LAN and change the default LAN>internet rule to use gateway "opt1primary" Next add an additional rule (put it At the Top) and for Source choose "Single host or alias" and put in the alias you created before (e.g. "t1devices") and select gateway "t1primary" under Advanced > gateway. Save and test – should work as you are asking. If in the future you have additional devices that you want to "force" to use the T1, all you have to do is add their IPs to the t1devices alias.

Which hardware configuration are you trying to get working: a) 3G stick directly in pfSense; or b) 3G stick in Asus router, pfSense going out through Asus LAN to internet as failover. ? (a) might have hardware support issues in FreeBSD/pfSense - I have never done 3G stick directly in pfSense myself. (b) I'm sure will work as long as the Asus router and stick are working - I have that with a Tp-link device and 3G stick in my home as failover. Post details of where you have got to and the various settings and rules you have for LAN, WAN and WAN2…

pfsense webgui might run on port 80 …. make absolutely sure this isn't an issue in your situation check the default route on the webserver ... you sure its pointed at the pfsense ?

you either need more physical interfaces on your pfsense or need to start using vlans (that require your routers/modems to be able to deal with them – unlikely). so its like johnpoz said ... don't try to work around a broken design ... start from scratch and do it properly also a 10W switch would cost you around $20 / year  .... (if you can afford 2 isp's for a home network, then i dont think the additional $2 / month will make a diff)

Thanks! This worked!

Concluding this utter waste of time: Go hire someone qualified to fix and maintain your network.

@doktornotor: Did you read this? https://doc.pfsense.org/index.php/Upgrade_Guide#LAGG_LACP_Behavior_Change No I did not see that link before. However I do appear to have solved the problem as I indicated in the edit of my previous post. On my Cisco 3750 under global config, I have set "port-channel load-balance src-dst-ip" & under my 4 LAG ports on the switch "channel-group 2 mode on", "switchport mode trunk", "switchport trunk encapsulation dot1q", & "switchport trunk allowed vlan X-X". Under interface "port-channel 2", set options also for "switchport mode trunk", "switchport trunk encapsulation dot1q", & "switchport trunk allowed vlan X-X". Packetloss looks to have cleared up. Though i'm confused on how if LACP isn't supported in the VMs without vCenter & a distributed switch as to how they had any connectivity at all. Here are the 2 official vmware related links about this. I switched my LAG interface mode to loadbalance in pfSense. Though I did try to read & am confused on difference between FEC and loadbalance options. I believe they both do the same thing. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034807 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006129 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2034277

Yup. it's that easy.  Note the negate route above the one with the gateway.  That's important. If it were me, I'd create two gateway groups.  One made up of WAN1+WAN2 with WAN1 preferred and one made up of WAN1+WAN2 with WAN2 preferred. Then I'd set a rule like the above but with the group instead.  That way if one of the WANs goes down both LANs still have internet.  Unless your situation absolutely requires LAN2 egress WAN2 and never anything else.

@Derelict: Someone else can correct me but I don't believe there's any way to tell pfSense to obtain a second IP via DHCP. I'm sure this seems too simple, but assuming you had multiple NICs. 2 inbound (from ISP connecting to your modem) & 1 or 2 out to your switch. You should just be able to add a new interface for your other NIC. Interface -> Assign. You should then be able to select that new interface coming in from modem & set the IPv4 config type to DHCP just like the regular WAN interface. Now you may need more than 1 outbound to your switch. You may need 2, one for each IP. I don't know if data from 2 public IPs can be piped over 1 NIC to your network.