Need help with setup of CARP + failover + multiWAN

docwho76 · Oct 16, 2008, 1:08 AM

I have used the MultiWANVersion 1.2 document and the CARP setup SWF to get a two system CARP setup created and have started to embark upon the multi-WAN aspect of the config and am running into a point where I am unsure of how to proceed.

Basically, I have a system with 3 WAN connections, 1 LAN, and 1 SYNC interface just for CARP. I have no problems with creating the VIPs for each WAN connection. My questions concern doing failover from WAN1 to WAN2 to WAN3, etc with NO loadbalancing and how to handle the firewall and NAT rules.

The CARP walk through has me specify the VIP of the WAN connection that the CARP is using in the NAT rules. What am I supposed to do if I have multiple VIPs, one for each WAN connection?

Additionally, how do I handle the failover between three WAN links without any load balancing? Any help would really be appreciated here, thanks! Its not exactly clear if I am making 6 pools in Load Balance, (i.e. WAN1failstoWAN2, WAN1failstoWAN3, WAN2failstoWAN1, WAN2failstoWAN3, WAN3failstoWAN1, WAN3failstoWAN2) or if I should do it some other way?

dotdash · Oct 16, 2008, 7:16 PM

@docwho76:

The CARP walk through has me specify the VIP of the WAN connection that the CARP is using in the NAT rules. What am I supposed to do if I have multiple VIPs, one for each WAN connection?

Just create one for each interface:
WAN 192.168.1.0/24 * * * (WAN CARP IP) * NO
WAN2 192.168.1.0/24 * * * (WAN2 CARP IP) * NO
WAN3 192.168.1.0/24 * * * (WAN3 CARP IP) * NO

As for the pools, what are you trying to do with them? You could use all three and make three failover pools:
WAN1failstoWAN2failstoWAN3, WAN2failstoWAN3failstoWAN1, WAN3failstoWAN1failstoWAN2

docwho76 · Oct 16, 2008, 9:55 PM

Ahh yes, I see what you mean about the NAT rules now. I should have seen that!

As for the failover pools, my understanding is limited at best. As it is I have been looking at the MultiWAN & loadBalancing document and I see how to create a load balancing pool between multiple WAN connections, but its not immediately clear how one modifies that for just failover with no load balancing.

That is to say, I have these current pools created, WAN1failstoWAN2 and the like, but in the load balancing document it has you use the Load Balance pool as your default last matching routing rule. I dont see how I would do something similar with the failover pools I have created? I hope this makes some sort of sense.

In other words, would I just create LAN firewall rules for each failover group like the following:

* LAN net * * * WANFailsToWAN2 Default LAN -> WAN1 or WAN2 if WAN1 fails

LAN net * * * WAN2FailsToWAN Default LAN -> WAN2 or WAN1 if WAN2 fails

Or just leave it at the current of:

* LAN net * * * * Default LAN -> any

Thanks!

dotdash · Oct 20, 2008, 4:27 PM

The failover pool is going to act like a default gateway, so how you use them depends on what you need to do.
A simple configuration would be to change the gateway on the default LAN rule to the failover pool. Then, your LAN clients would use WAN1. If WAN1 was down, then would use WAN2, if WAN1 and WAN2 were down, they would use WAN3 (for example). More complex configurations would route a particular machine, subnet, protocol, etc to a particular failover pool. For example, you could place a rule before the default LAN rule matching destination http and specify a pool that used WAN3 then failed to 1 and 2. That way web traffic would use WAN3 if it was up, failing to WAN 1 and WAN2, and the other traffic would use WAN1 first and fail to the other lines.

docwho76 · Oct 20, 2008, 10:16 PM

Ok, I now have CARP + multiWAN + failover setup successfully it looks like and I have discovered a few things I wasnt expecting. I've got two failover pools, WAN1-WAN2-WAN3 and WAN3-WAN2-WAN1. However, when I modify the LAN interface default traffic rule to use the WAN1-WAN2-WAN3 failover pool as a gateway it would appear that my second failover pool is never going to get used. So would I just create a second default rule for the LAN subnet like so:

* LAN net * * * WAN1-WAN2-WAN3 Default LAN -> any

LAN net * * * WAN3-WAN2-WAN1 Default LAN -> any

The second question/issue I ran into is that with this setup in my test lab if I disconnect WAN1 on the left CARP system, but not on the right CARP system that left goes from MASTER state to BACKUP state and the right CARP box is using its working WAN1 connection to route outbound traffic. Obviously this is a somewhat potentially artificial condition that is mimicking a cable cut, and not the actual WAN1 upstream router dying. If I cut both CARP systems WAN1 out I would expect that the current MASTER CARP system (normally left) would switch to WAN2 as its next in the failover pool. Is this a correct statement?

The third question/issue is a more concerning one to me, but I realize might be outside of the scope of what pfSense can do. I have noticed that when I fail from WAN-X to WAN-Y that sessions in progress dont gracefully transfer over. Or at least this is how it APPEARS to me, I might have a lack of understanding here. Basically, what I do for failover tests is I startup a streaming audio radio station in iTunes while running a continuous ping to www.yahoo.com. If CARP left on WAN1 fails to CARP right on WAN1 I see no real blip in the ping traffic or any disruption in the streaming audio.

However, if WAN1 fails to WAN2 my ping to www.yahoo.com just stops getting echo replies and the iTunes streaming audio runs out of buffer and stops playing. If I quit the ping and restart it, the ping gets responses just fine using the new gateway. Ditto for the iTunes, if I tell it to restart the audio stream it works fine. Is there a way to make the transition from WAN1 to WAN2, etc a seamless experience for my end users?

My current Outbound NAT rules look like this:
WAN 10.0.0.0/8 * * * 99.49.233.4 * NO Use WAN-CARP IP for LAN
WAN2 10.0.0.0/8 * * * 172.16.0.50 * NO Use WAN2-CARP IP for LAN
WAN3 10.0.0.0/8 * * * 192.168.1.4 * NO Use WAN3-CARP IP for LAN

The private IP blocks are a workaround for my test lab so I could have enough IP space to test the CARP VIPs with since our main DSL only has 5 useable public IPs, of which 2 are in use for business needs. WAN2 basically hooks into the LAN side of our office network, and WAN3 hooks into a Macbook Pro sharing its wireless connection to the Internet via a 3g cellphone acting as a data modem.

As usual, any help is appreciated.

dotdash · Oct 21, 2008, 4:45 PM

Ok. There are a lot of questions there. Let me see if I can address a few.

The rules process top down, so if you have a second default rule, it will never be processed. You would need something like this, for example, if you wanted to split the traffic:
* 10.0.0.0/9 * * * WAN1-WAN2-WAN3 1st half of LAN -> 1-2-3
* 10.128.0.0/9 * * * WAN3-WAN2-WAN1 2nd half of LAN -> 3-2-1

2,3) CARP and the failover/loadbalancing are two separate things. If one box drops an interface, the other will take over via CARP. Failover will occur when the monitored IP becomes unreachable. A CARP failover will preserve the existing states and be mostly transparent, when the load balancer switches to a new WAN, any existing states are dead and need to be re-established. This is normal. Users generally don't notice when they are just browsing, but streaming media, etc will drop.

docwho76 · Oct 21, 2008, 7:53 PM

Thanks for your help! I dont really want to split the LAN subnets between the two failover pools. I guess this is because I somehow got the idea that if WAN1 failed to WAN2, or if it failed all the way to WAN3, what dictates the connection moving back to WAN1 or WAN2 if those circuits come back up? I ask this because in the scenario I am going to be dealing with, WAN1 is going to be a significantly higher bandwidth connection then WAN2 or WAN3, and if WAN1 fails and the system goes to WAN2/3 will the system move back to WAN1 when it comes back up?

dotdash · Oct 21, 2008, 8:12 PM

Yes, once the monitor IP is responding again, the system will switch back to WAN1. Established connections on WAN2 will continue to use WAN2. (eg- streaming music)

docwho76 · Oct 22, 2008, 7:46 PM

Excellent! Now if only there was a way to sync FreeRADIUS between machines in a carp cluster. But I can solve that problem through other methods, aka a single system on the LAN that isnt one of the pfSense boxes.