CARP failover causes default route on master to go missing

[michael_d]

Hi all,

I have recently been attempting to set up a HA pfsense cluster using CARP. The two firewalls are SG-8860's running pfsense 2.4.0-RELEASE.

I am having an issue where when failing over the default route on the master firewall goes missing and doesn't return when failing back to the master (using "Persistent maintenance mode" to fail over).

The testing setup is as follows:

• To route to the internet we use a /31 from our ISP. One of these IPs is a CARP IP on the firewalls, the other IP is the default gateway for the firewalls (Set in "System -> Routing -> Gateways").
• The two WAN interfaces have IPs in 172.16.0/24 so they can hear each others CARP traffic. We tried having no IP configuration on the interterfaces and just associating the CARP IP with the interfaces but the firewalls could not hear each others CARP traffic in this case.
• The LAN interfaces have IPs in our internal /24 and a CARP address also in this /24.
• pfsync and XMLRPC sync is enabled using a dedicated interface.
• The upstream is simulated by another SG-8860 using the default gateway IP of the firewalls on its LAN interface and outbound NAT'ing any traffic from the LAN side of the firewalls to the WAN interface, which has internet connectivity.

With this setup we are observing:

• When the firewalls come up for the first time things are as expected. We can route to the internet via a machine on the LAN side of the firewalls (ping 8.8.8.8 is used to test connectivity).
• Entering "persistent maintenance mode" on the master correctly fails over to the secondary, again routing to the internet from the LAN side works fine
• Leaving "persistent maintenace mode" resutls in "Destination host unreachable" reponses from the master.
• Investigation of the master shows that there is no longer a default route (Despite there still being a default gateway in "System -> Routing -> Gateways").

I am unsure why the default route goes missing and it is obviously making the failover quite useless, since we can never fail back (short of rebooting the master so the default gw comes back!)

Any help would be appriciated, feel free to task for more information on the configuration and I'll do my best to provide it.

Thanks in advance.

[Polderdijk]

Go to Firewall > NAT > Outbound:

Make sure you have 'Manual (or Hybrid) Outbound NAT' and create an extra rule:

WAN - This Firewall - * - * - * - (WAN CARP IP) - *

Also i think you need to reboot so the apinger is refreshed.