cmb, Thank you for the nudge!  :D In my frustration I forgot to troubleshoot the obvious, so far I bypassed the switch the boxes were connected to and hooked up straight to the edge router, so far for yesterday's evening and today's testing, it's going very well.  Speeds are being maintained and data transfers (FTP, HTTP, etc) are coming in at max rate.  Seems that intermediate switch was not playing nice when assigning the VIPs to the CARP interface.

Thank you for your reply. I am still not sure of the statement about having the distinctive interface - is there a way to bind a user to login only through specific interfaces, that I am unaware of?  As far as I see, a configured user can login through any allowed interface. HTTPS is good for encrypting the traffic, but exposing the system to yet another full admin user is what I need to secure. If a configured user can login through any interface, it would be nice to know what minimum privileges are needed for the CARP user. Thanks ahead of time for your replies.

Hi guys - you have exactly the problem I have! And you've found the same fix. Forcing failover / failback works as a work around. I'm using the latest 2.1.5 with the same results. Here's a "sort of" solution too. Don't use the main carp vip - use another ip (an alias ?) for your services? They don't have the same issue even when the issue occurs on the main IP. I think these articles are related: https://forum.pfsense.org/index.php?topic=81050.msg451115 https://forum.pfsense.org/index.php?topic=81709.msg451363

I would check that your firewall rules should allow the ping of course, but then try failing over (disable carp on primary to force a transfer) - pings might start working then - if so, you might be seeing the same problem secgeek and I are noticing.

This seems related to the other issues secgeek and I have seen. In my case, the working behaviour (when triggered by the problem we're noticing) can be restored by triggering carp failover (disabling on primary) - the setup will continue to work after failback. When I do static set ups, I don't use a single DNS - even if using carp - I use the DNS of each router. The GW does have to be the floating IP though. It might be helpful to those trying to help if you posted more info about the config? Or did you resolve already? Thanks!

I am presently using two separate physical machines. I believe they sync interface is connected by a cross over cable. The issue doesn't affect the alias ip's - only the main carp virtual IP. So that's my work around at the moment is to not put any services on the main floating IP. I've seen enough posts to suspect this is a real problem though - not one of our misconfiguration - what do you think?

Hi jimp, check your outbound NAT. You should be doing manual outbound NAT to a CARP VIP, or else you'll also get cut off like you see there. Thanks for your help, this one did it! I was NATing using the firewall IP instead of the virtual IP. Once I did a manual outbound NAT as suggested, the problem is fixed and the downloads continue through the failover with only a few packets dropped in between. Enjoy your weekend and thanks again! Kind Regards, Jason.

I am abandoning this question and forum, since I've gotten no feedback on how to resolve this. I decided to decommission the PFsense firewall and use an alternative product.

I feel a little dumb… I missed to get the correct Base advertising frequency on both nodes of one carp VIP config... Greets!

@Derelict: I'd probably start here.: https://doc.pfsense.org/index.php/How_can_I_forward_ports_with_pfSense%3F Thats perfect.  thanks.

I've seen this behavior under several circumstances: when there is a carp misconfiguration; be very careful about the VHID, it must be unique for each virtual IP when there is something filtering CARP traffic between the nodes when there is leakage between the virtual IPs (eg: lan and dmz can see each other on layer 2).

@JeGr: @Rob Hate to disappoint you, while the main problem is indeed fixed (no aliases were created with 2.1.4 anymore), there still is a bug with deleting said aliases. They won't get deleted on the backup node, thus bringing chaos to the CARP stack on that interface leading to a split-brain (master/master) situation on that interface (can be resolved by rebooting the standby node or manually deleting the aliases on the VIP interface in a root shell on console). So my advice: be careful. Interesting! Do you know how long this bug has stood for, we've always had interesting behaviour with CARP + VIPs and failovers; we've always ended up rebooting the secondary for "random" problems like these.

@secgeek: Don't know what you are trying to say, but it doesn't make sense. @pedreter: we run a datacenter firewall cluster with HA and carp on multiple physical interfaces (6) with about a dozen VIPs on our second WAN link and minimum one on the remaining links (WAN, XFER, MGMT, etc.). No problem with that. Seems like a configuration issue to me rather than any problem with CARP or IPs. Greets

Good news! Everything is working. The settings were correct as you mentioned, except for one thing under Virtual IPs: The source interface was set to WAN instead of LAN. Changing it to LAN let me have both 20.0/24 and 11.0/24 subnets on the same LAN.

You will need an IP for each physical box, so you are only going to have three IPs that will fail over. Lets say .249 is the gateway, 250 could be one firewall, 251 the second, leaving you with 252, 253, and 254. You might be able to share IPs using port forwards and have enough. You can terminate multiple IPSec tunnels on one CARP VIP.

If you use CARP VIPs, you will have to add an Alias IP on each firewall or you will get an error about no matching subnet. Another option is to use OTHER VIPs, which may work depending on how the provider is routing the block to you.