CARP with multiple WAN subnets

dotdash

I'm working on setting up a cluster on a firewall with multiple subnets (non-contiguous) on the WAN.
Basically, as they needed more IPs, the provider assigned more blocks. So it looks something like this:
Wan subnet 12.13.14.16/29 WAN if=12.13.14.17 gateway=12.13.14.22 WAN carp=12.13.14.21 Second node WAN if=12.13.14.18 Additional subnets 12.13.20.80/29 12.13.30.32/28

After a review of the literature, I've come to these options. I was wondering if anyone had any feedback from a similar deployment.

Based on my interpretation (possibly wrong) of a solution offered by Bill M on the mailing list:
Add static routes on the WAN 12.13.20.80/29 and 12.13.30.32/28 gateway 12.13.14.21 (the WAN CARP)
Then add VIPs of type 'other' to BOTH nodes. This would be very slick, but I haven't tested it and may have got it wrong.
Hack the config.xml to add alias addresses to the WAN, then add CARP VIPs. I think this will work, but I don't like having to use a hack, and this also wastes two IPs on each of the subnets.
Looks like carpdev on FreeBSD is in ALPHA. (http://www.nabble.com/ifconfig-carpdev-t4309277.html)
Wait until it becomes stable and is rolled into pfSense. Problem is that in the meantime a bunch of hosts will be down when the primary node fails.
EDIT: Correction- carpdev for FreeBSD is BETA. (http://www.nabble.com/carpdev-…-t4704557.html)

UPDATE, In case anyone has a similar situation:
Finally had a lull to add a second firewall and reconfigure for failover. Solution one seems to be working fine. The VIPs for the additional subnets are added as 'other'. The 'other' VIPs sync'd over to the backup unit along with the CARPs. In initial testing, the servers using the other VIPs are fully reachable when the primary node is off-line.

youri

@dotdash:

I'm working on setting up a cluster on a firewall with multiple subnets (non-contiguous) on the WAN.
Basically, as they needed more IPs, the provider assigned more blocks. So it looks something like this:
Wan subnet 12.13.14.16/29 WAN if=12.13.14.17 gateway=12.13.14.22 WAN carp=12.13.14.21 Second node WAN if=12.13.14.18 Additional subnets 12.13.20.80/29 12.13.30.32/28

After a review of the literature, I've come to these options. I was wondering if anyone had any feedback from a similar deployment.

Based on my interpretation (possibly wrong) of a solution offered by Bill M on the mailing list:
Add static routes on the WAN 12.13.20.80/29 and 12.13.30.32/28 gateway 12.13.14.21 (the WAN CARP)
Then add VIPs of type 'other' to BOTH nodes. This would be very slick, but I haven't tested it and may have got it wrong.

Hack the config.xml to add alias addresses to the WAN, then add CARP VIPs. I think this will work, but I don't like having to use a hack, and this also wastes two IPs on each of the subnets.

Looks like carpdev on FreeBSD is in ALPHA. (http://www.nabble.com/ifconfig-carpdev-t4309277.html)
Wait until it becomes stable and is rolled into pfSense. Problem is that in the meantime a bunch of hosts will be down when the primary node fails.
EDIT: Correction- carpdev for FreeBSD is BETA. (http://www.nabble.com/carpdev-…-t4704557.html)

UPDATE, In case anyone has a similar situation:
Finally had a lull to add a second firewall and reconfigure for failover. Solution one seems to be working fine. The VIPs for the additional subnets are added as 'other'. The 'other' VIPs sync'd over to the backup unit along with the CARPs. In initial testing, the servers using the other VIPs are fully reachable when the primary node is off-line.

Hi how are configured the additional subnets ? Do you have any remote gateways from 12.13.20.80/29 12.13.30.32/28 on a remote router or are these blocks directly routed on your L2 link ?
–
YP

dotdash

The blocks are directly routed to me. I have static routes on the WAN pointing the blocks to the CARP WAN VIP. This is most likely unneeded, but it's working now, so I haven't tried removing the routes.

Eugene

Hello dotdash,

could you clarify please as I need to configure the same thing. Taking your configuration as an example I have to do the next to make it work.
We have two firewalls with carp-ip-address at WAN 12.13.14.21. Then at both firewalls I add as type=other the same VIP 12.13.20.80. Obviously I do some NATting with this VIP (I want to do outbound and port forwarding) and everything works when failover occurs (either of firewalls is down)?

Thanks,
Eugene.

dotdash

This setup has since moved to a new location with a contiguous block, so I'm working from memory here.
I added the 'other' VIPs singularly on the master, and I believe they sync'd to the slave automatically- perhaps after a reboot. So, using the example numbers, I added 12.13.20.81, 12.12.20.82, 12.13.30.33, etc as 'other' VIPs. Then I used the VIPs when setting up port-forwards, etc. The only problem I had was that the 'other' VIPs broke the failover logic for the DHCP server, so I had to make a frowned-upon hack to force the backup node to always be the secondary dhcp server.

Eugene

What do you mean 'failover logic for dhcp server'? We are talking about WAN interface, I suppose you run DHCP server at LAN. So, how the setup of VIPs at WAN can affect DHCP-server?
I am going to try this setup tonight, will provide feedback.
Also it would be great to know a little bit more internals. Everything is clear about carp-VIPs as there is good documentation about FreeBSD's CARP. But what is 'proxy ARP' VIP, what is 'Other VIP'? Do you know equivalents of CLI commands to configure these types of addresses?
Another my concern… Everything is understandable when we have CARP: IP packets have always virtual MAC-address of CARP-interface as destination and only master responds with its interface MAC as a source. But with 'Other VIP' - how it works? Which interface will answer to ARP-request? What algorithm works here?

Probably I did not search thoroughly but so far can not find comprehensive answers to all these questions :-(

dotdash

@Eugene:

What do you mean 'failover logic for dhcp server'? We are talking about WAN interface, I suppose you run DHCP server at LAN. So, how the setup of VIPs at WAN can affect DHCP-server?

Yes, I was running failover DHCP for the LAN on the cluster, and somehow the type-other VIPs made both servers to think they were the primary. pfSense runs a check which looks at the CARP skew to see if the node is the master, then sets the DHCP as primary or secondary accordingly. I didn't have time to debug past finding an ugly workable hack.
@Eugene:

Also it would be great to know a little bit more internals. Everything is clear about carp-VIPs as there is good documentation about FreeBSD's CARP. But what is 'proxy ARP' VIP, what is 'Other VIP'? Do you know equivalents of CLI commands to configure these types of addresses?

IIRC, proxy arps are handled by choparp, a proxy-arp daemon which responds with the MAC of the interface.
I'm not sure how the mysterious other VIPs work- that was actually the only time I used them.

Eugene

Ok, thank you, choparp was a very good hint. The only question remains - what is 'Other IP'.
If I configure 'Other VIP' at master it automatically appears at slave.
But neither master nor slave responds to ARP requests. So, I can not see how I can use this VIP. -(((

dotdash

The best information on other VIPs I've seen was from hoba:
http://forum.pfsense.org/index.php/topic,3987.msg24632.html#msg24632
In my case, the other VIPs just worked and were available even when one box was rebooting.

Eugene

Perfect. Now I know what 'other VIP' is. It worked for you because your provider did not care about L2 address and just forwarded these addresses to you.
Unfortunately I can not use it.
Anyway it is very interesting… Suppose you are server and you have IP packet arriving at your interface... how L2 decides that this packet is intended to you and deliver it to L3 in ethernet environment? I suppose L2 has to deliver every packet and L3 will decide whether this packet yours or not. Obvious overhead but good opportunities to handle this traffic. Very interesting... I would like to read more how FreeBSD handles this.

Thanks again.