This seems related to the other issues secgeek and I have seen.
In my case, the working behaviour (when triggered by the problem we're noticing) can be restored by triggering carp failover (disabling on primary) - the setup will continue to work after failback.
When I do static set ups, I don't use a single DNS - even if using carp - I use the DNS of each router.
The GW does have to be the floating IP though.
It might be helpful to those trying to help if you posted more info about the config? Or did you resolve already? Thanks!

I am presently using two separate physical machines.
I believe they sync interface is connected by a cross over cable.
The issue doesn't affect the alias ip's - only the main carp virtual IP.
So that's my work around at the moment is to not put any services on the main floating IP.
I've seen enough posts to suspect this is a real problem though - not one of our misconfiguration - what do you think?

Hi jimp,

check your outbound NAT. You should be doing manual outbound NAT to a CARP VIP, or else you'll also get cut off like you see there.

Thanks for your help, this one did it! I was NATing using the firewall IP instead of the virtual IP. Once I did a manual outbound NAT as suggested, the problem is fixed and the downloads continue through the failover with only a few packets dropped in between.

Enjoy your weekend and thanks again!

Kind Regards,

Jason.

I am abandoning this question and forum, since I've gotten no feedback on how to resolve this. I decided to decommission the PFsense firewall and use an alternative product.

I feel a little dumb… I missed to get the correct Base advertising frequency on both nodes of one carp VIP config...

Greets!

@Derelict:

I'd probably start here.: https://doc.pfsense.org/index.php/How_can_I_forward_ports_with_pfSense%3F

Thats perfect.  thanks.

I've seen this behavior under several circumstances:

when there is a carp misconfiguration; be very careful about the VHID, it must be unique for each virtual IP

when there is something filtering CARP traffic between the nodes

when there is leakage between the virtual IPs (eg: lan and dmz can see each other on layer 2).

@JeGr:

@Rob Hate to disappoint you, while the main problem is indeed fixed (no aliases were created with 2.1.4 anymore), there still is a bug with deleting said aliases. They won't get deleted on the backup node, thus bringing chaos to the CARP stack on that interface leading to a split-brain (master/master) situation on that interface (can be resolved by rebooting the standby node or manually deleting the aliases on the VIP interface in a root shell on console).

So my advice: be careful.

Interesting! Do you know how long this bug has stood for, we've always had interesting behaviour with CARP + VIPs and failovers; we've always ended up rebooting the secondary for "random" problems like these.

@secgeek: Don't know what you are trying to say, but it doesn't make sense.
@pedreter: we run a datacenter firewall cluster with HA and carp on multiple physical interfaces (6) with about a dozen VIPs on our second WAN link and minimum one on the remaining links (WAN, XFER, MGMT, etc.). No problem with that. Seems like a configuration issue to me rather than any problem with CARP or IPs.

Greets

Good news! Everything is working. The settings were correct as you mentioned, except for one thing under Virtual IPs: The source interface was set to WAN instead of LAN. Changing it to LAN let me have both 20.0/24 and 11.0/24 subnets on the same LAN.

You will need an IP for each physical box, so you are only going to have three IPs that will fail over.
Lets say .249 is the gateway, 250 could be one firewall, 251 the second, leaving you with 252, 253, and 254.
You might be able to share IPs using port forwards and have enough. You can terminate multiple IPSec tunnels on one CARP VIP.

If you use CARP VIPs, you will have to add an Alias IP on each firewall or you will get an error about no matching subnet.
Another option is to use OTHER VIPs, which may work depending on how the provider is routing the block to you.

You likely need to enable MAC address spoofing on the two VMs.

Welp, as it turns out, the Motorola/Arris box that they gave me doesn't require sticky statics - It's strictly optional.  So I can continue using PFSense as a filtered bridge without any worries. :)

This patch to rc.carpmaster and rc.carpbackup is still necessary on 2.1.4 to stop/start quagga on carp role change right?

It looks like some logic was added to determine whether or not to start quagga based on role at startup, but the patch is still required for failover.

Do you have your interfaces assigned in the same order on both pfSense in Interfaces > assign? This is essential for syncing correctly.

For the record, I've reproduced this on the firewalls in the office here; the patch appears to solve it. However I'm still totally confused as to how it's working at all

When the bug is present the IP address of the Alias (lets say .21) isn't assigned to any of the interfaces at all. Yet if I do curl https://x.x.x.21 then I can get through the NAT and the firewall to the webserver underneath. Why is PFSense responding on an IP it doesn't own!?