CARP failover not routing properly

mcampbell

So I'm trying to deal with an issue that I have with my CARP set up, where when the carp fails over to the secondary box, site-to-site VPNs get disconnected and refuse to reconnect, and certain outgoing traffic no can longer go out.

This smacks of a routing problem on the second node to me, but I'm at a loss, as all of that should be synced with the pfsync settings between the two boxes. And I've checked manually between the two boxes' settings, and am unable to find any differences.

Can anyone give me any tips or pointers to look at in my settings?

cmb

What type of VPNs?

What traffic no longer works? Specific to certain NATed IPs, or?

mcampbell

To give a little bit better background on our network layout (as it is complex), I'm posting a diagram of it.

The site-to-site VPNs are IPSec (there's 5 of them). The IPSecs are bound to the CH VIP. They provide additional network legs, listed in the chart. When the failover occurs, the site-to-sites go down, giving a yellow icon in the IPSec status window. Additionally, the outgoing traffic that doesn't seem to leave right are web traffic, and VOIP phone traffic.

Edited to update revised image

![pfSense CARP layout.png](/public/imported_attachments/1/pfSense CARP layout.png)
![pfSense CARP layout.png_thumb](/public/imported_attachments/1/pfSense CARP layout.png_thumb)

cmb

Likely from those IPs not working in general on the secondary, assuming they're CARP IPs or IP aliases with a CARP parent. While failed over if you go to Diag>Ping on the secondary, source from one of the affected IPs, and ping out to something on the Internet, does it work?

Are these physical boxes, or VMs? Most common reason that comes to mind is VMware without appropriate vswitch config to allow the CARP virtual MACs to be used on the secondary system.

mcampbell

@cmb:

Likely from those IPs not working in general on the secondary, assuming they're CARP IPs or IP aliases with a CARP parent. While failed over if you go to Diag>Ping on the secondary, source from one of the affected IPs, and ping out to something on the Internet, does it work?

Are these physical boxes, or VMs? Most common reason that comes to mind is VMware without appropriate vswitch config to allow the CARP virtual MACs to be used on the secondary system.

These are physical boxes. I haven't actually tried the Diag>Ping on the secondary when the failover occurs. I'll do that next time it fails over. But at least right now, I can ping from an external source both WANs of both pfsense boxes, in addition to the CARP VIP shared between them on each WAN. If it were a problem from the IPs not working in general, would I not be able to ping the secondary's?

For reference, the IPs are set up like so (and as of right now, I can ping all of them externally):

BR network:
pfsense01: 208.xxx.xxx.171 (NIC's actual address)
pfsense02: 208.xxx.xxx.172 (NIC's actual address)
BR VIP: 208.xxx.xxx.170 (CARP VIP shared between the two IPs above)

CH network:
pfsense01: 71.xxx.xxx.19 (NIC's actual address)
pfsense02: 71.xxx.xxx.20 (NIC's actual address)
BR VIP: 71.xxx.xxx.18 (CARP VIP shared between the two IPs above)

@nikkon:

@cmb:

What type of VPNs?

What traffic no longer works? Specific to certain NATed IPs, or?

OpenVPN only.

In our case, we don't use OpenVPN currently, our site-to-sites are IPSec.