CARP Setup with HAProxy and VIPs

ddepaolis · Sep 12, 2023, 9:31 PM

@planedrop Thanks so much for your reply; of course I'm planning a lab so do I.
Let me explain better my load balancing configuration through HA proxy.

I set a public alias VIP on WAN interface (1.2.3.4/28) and a private alias VIP on LAN interface (192.168.1.50/24). Alias VIP on WAN interface is configured with a NAT port forward rule for inbound connections, I explain later. Alias VIP on LAN interface is configured as "external address" in HA Proxy (frontend). HA Proxy, in turn, has two server configured in the backend

Connections from inside (LAN) to those two servers go, of course, directly to alias VIP 192.168.1.50/24 in frontend HAProxy configuration.
Connections from outside (WAN) to those two servers, follow this NAT Port Forward rule:
Interface: WAN - Protocol: TCP - Source IP: Any - Source port: 1024-65535 - Destination address: Alias VIP 1.2.3.4/28 - Destionation Port: 443 - NAT IP: Alias VIP 192.168.1.50/24 - Nat Port: 443

Then my main question: in a High Availability scenario using CARP, the slave PFSense node may have same Alias VIPs on WAN and LAN interfaces (1.2.3.4 / 192.168.1.50) or other ones ?

With a single PFSense node my load balancing configuration through HAProxy works really good. But adding a new layer of High Availabilty with a second PFSense node will it continue working so fine ?

viragomann · Sep 13, 2023, 8:26 AM

@ddepaolis
You can do exactly the same with an HA setup.
However, consider that you need one IP in each network segment for each node and additionally a CARP VIP. So you need 3 IPs in each subnet.

After configuring CARP HA, you have to hook up the other VIPs on the CARP VIP, so that they are available on both nodes and fail-over with the CARP.
But you can also use the CARP VIP itself for a HAproxy frontend.

ddepaolis · Sep 13, 2023, 8:26 AM

@viragomann Ok, so the actual Alias VIP 1.2.3.4/28 on WAN interface should become a CARP VIP. Then I need two other Alias VIPs on WAN interface: 1.2.3.5/28 on master PFSense node and 1.2.3.6/28 on the slave PFSense node (they're like "physical IP").

Same way, the actual Alias IP 192.168.1.50/24 configured in frondend HAProxy should become a CARP VIP. And I should continue using it in my HAProxy configuration.

So is it all correct ? (I hope so).
Thanks !

viragomann · Sep 13, 2023, 9:17 AM

@ddepaolis said in CARP Setup with HAProxy and VIPs:

so the actual Alias VIP 1.2.3.4/28 on WAN interface should become a CARP VIP. Then I need two other Alias VIPs on WAN interface: 1.2.3.5/28 on master PFSense node and 1.2.3.6/28 on the slave PFSense node (they're like "physical IP").

You need to set physical IPs indeed on the NICs of both nodes in Interfaces > Assignments.
So on the primary you configure the WAN interface with a static IPv4 1.2.3.5/28 and the proper gateway. And on the secondary 1.2.3.6/28 with proper gateway.

Ensure you have configured a sync network. A separate NIC is not really necessary, but it's highly recommended. Add a firewall rule to allow access on the sync interface.

Then add the CARP VIP on the primary. If you use an existing VIP you have to remove this before.

Also on the LAN you have to configure physical IPs on the interfaces and a VIP, which is used as default gateway on the the LAN devices then.

See also High Availability Configuration Example in the pfSense docs.

ddepaolis · Sep 13, 2023, 3:28 PM

@viragomann Yep. that's all clear in a normal scenario. However on my real enviroment, my actual PFsense node has the "main" public IP of subnet /28 set in the WAN NIC. Then all the other ones in that /28 subnet, are set as alias IPs on same WAN interface. So to explain better:

1.2.3.4/28 WAN NIC
1.2.3.5/28 Alias IP WAN
1.2.3.6/28 Alias IP WAN
1.2.3.7/28 Alias IP WAN
etc...

Splitting this situation on a High Availability scenario, the first 3 public IPs become 1.2.3.4/28 VIP CARP, 1.2.3.5/28 PFsense1 WAN NIC, 1.2.3.6/28 PFSense2 WAN NIC. The other following public IPs can remain as Alias IP on WAN interface ? Then Alias IPs can set the same on each PFSense node, without having addresses' conflict ?
Of course thanks so much for all infos I'm now collecting to develop my new enviroment !

viragomann · Sep 13, 2023, 4:21 PM

@ddepaolis said in CARP Setup with HAProxy and VIPs:

Splitting this situation on a High Availability scenario, the first 3 public IPs become 1.2.3.4/28 VIP CARP, 1.2.3.5/28 PFsense1 WAN NIC, 1.2.3.6/28 PFSense2 WAN NIC.

Yes, for instance. Though it doesn't matter which IP out of your public range you assign to the interfaces.

The other following public IPs can remain as Alias IP on WAN interface ?

No, as managened, they VIPs have to hook up on the CARP VIP.
After assigning the CARP VIP, on the primary go to the virtual IP settings and change the interface to the CARP VIP.

Then Alias IPs can set the same on each PFSense node, without having addresses' conflict ?

The only you have to do on the secondary is to assign the interface IPs for all your network segments (WAN, LAN, VLANs) and configure the sync interface and allow access on it.
If the sync is working all VIP settings are synced from the primary.

Edit:
Should add, on the primary your have to configure the sync in System > High Availavility.

ddepaolis · Sep 14, 2023, 1:52 PM

@viragomann Ok, now I'm setting my lab test enviroment. In this case I'm using, for easier mode of couse, private IPs on WAN and LAN interfaces. So This's my actual scenario:

PFsense-1 WAN IP: 192.168.41.20/24 (VHID 80 - Advertising frequency 1 base; 0 skew)
PFsense-2 WAN IP: 192.168.41.21/24 (VHID 80 - Advertising frequency 1 base; 100 skew)
PFSense WAN VIP (CARP): 192.168.41.22/24

PFSense-1 LAN IP: 192.168.42.20/24 (VHID 56 - Advertising frequency 1 base; 0 skew)
PFSense-2 LAN IP: 192.168.42.21/24 (VHID 56 - Advertising frequency 1 base; 100 skew)
PFSense LAN VIP (CARP): 192.168.42.22/24

Replica from master to slave goes fine. Now I aspected to be able to ping the WAN VIP 192.168.41.22 through a client from same subnet 192.168.41.0/24 but this doesn't happen: why ? Both PFSense nodes have a rule to permit ICMP "any to any" on WAN interface.
I also already set NAT Outbound in hybrid mode, as indicated in howto page. And I added an explicit rule to allow connections from LAN network through my WAN VIP as "traslated" address.

viragomann · Sep 14, 2023, 1:52 PM

@ddepaolis
Since you access from private IPs to the WAN VIP now, did you remove the check "Block private networks" in the WAN interface settings on both nodes?

ddepaolis · Sep 27, 2023, 2:30 PM

@viragomann Yes I confirm, both options "Block private networks and loopback addresses" and "Block bogon networks" are uncheched on both WAN and LAN NICs on both firewalls. But any ICMP echo reply comes from VIPs on WAN or on LAN. I noticed another unusual behavior. Immediatly after I set CARP VIP on LAN interface on master firewall, it goes immediately in "master" status also in the backup node. As in the pic I added. This behavior always happens when I replicate this action. Actually my lab is inside a VMWare VSphere server, so both PFsense firewalls are virtual machines. I don't know if it could have any effect about this situation.

PS: syncronization now goes on a third NIC /28.

viragomann · Sep 27, 2023, 5:58 PM

@ddepaolis said in CARP Setup with HAProxy and VIPs:

Actually my lab is inside a VMWare VSphere server, so both PFsense firewalls are virtual machines. I don't know if it could have any effect about this situation.

Yes, it could.

Read Troubleshooting High Availability Clusters in Virtual Environments.

ddepaolis · Sep 28, 2023, 8:56 AM

@viragomann I see, and this could be the explanation because I haven't ICMP echo replies on VIP CARP too ?
Then I plan another lab in another different virtualization environment.
Always thanks and regards !