CARP WAN Failover not working

mkrbrazier

Hello,

I've been using PFSense ever since Monowall development stopped and I've been impressed with it meeting all my needs.

Two PFSense firewalls are virtual instances, one on an ESX server the other on Xen (currently updating both to Xen but have a legacy application that cannot be moved yet).

I have two PFSense firewalls in active/backup mode using CARP for failover. Both instances are aware of each other and entering persistence maintenance mode or disconnecting network causes failover as planned however, only LAN traffic fails over. CARP IP for lan does not miss a single ping nor is there any interruption to MySQL replication.

Two IP addresses have a 1:1 NAT to web servers, a shared NAT IP address for misc inbound services is also present.

The problem I have is when failing over all WAN traffic inbound and outbound stops and I'm struggling to work out why.

I have a manual outbound NAT rule set for my LAN subnet to the LAN CARP IP, is this correct or should I have more/different rules? Could there be something in the data center network that is required for PFSense? I can supply packet captures on request to help track down what is missing unless there is something more fundamental that I have missed.

See attached image of topology for mentioned services above.

Regards,
Matt.
![pfsense_carp_basic (1).png_thumb](/public/imported_attachments/1/pfsense_carp_basic (1).png_thumb)
![pfsense_carp_basic (1).png](/public/imported_attachments/1/pfsense_carp_basic (1).png)

dotdash

I don't understand why your diagram shows a separate LAN under each firewall, the standard practice is to have both connected to the same LAN segment. If you are not doing that, you're probably going to be on your own for troubleshooting.
That being said, check the usual suspects- promiscuous allowed on ESXi vswitch, no idea on Xen… make sure status shows master on master and backup on slave, etc.
I don't know what you mean about NAT to the LAN CARP, outbound should be the WAN CARP.

mkrbrazier

Hi,

Thanks for the reply. Bad diagram on my part (updated original post). There is a single LAN under the firewalls routing all servers via a CARP IP.

ESXi has promiscuous mode configured and all recommended tweaks have been made to Xen to make PFSense function correctly.

Matt.

jacknelton

It seems to me that your DCGW (or a switch between your FWs VMs and the DCGW) is not updating is L2 forwarding table.

Can you please specify the "upward" physical connections?

Cheers

J

Derelict

Does WAN stay MASTER/BACKUP or is it always MASTER/MASTER?

There's not a lot to happen there. Setting CARP maintenance mode hard sets the advskew to 254 and if the primary receives a more recent advert from the backup it will go into a BACKUP state on that VIP, likewise if the backup does not receive a more recent advert from the primary it will go MASTER.

Running a mismatched pair can be challenging and is not recommended. But this should work. State sync is another matter.

How are the WAN ports and the DCGW physically and virtually connected? Do both WAN ports see the CARP adv traffic like they should? (Packet Capture on CARP.)