CARP, sync and failover working but no internet connection through failover

Vorkbaard

Hi, I have CARP and sync set up in a test environment. Everything gets synced nicely, WAN virtual IP and LAN virtual IP, outbound NAT from WAN virtual IP working fine, both firewalls working fine on their own and CARP status is changed correctly when putting the primary firewall in persistent CARP maintenance mode.

From the secondary firewall (which now is the CARP master) I can ping out to the internet and I can ping an internal server. However from that server I can not ping the virtual ~~WAN~~ LAN IP and I have no internet connection.

DHCP is set to the virtual LAN IP (verified it).

It looks like the LAN clients are still trying to use the primary firewall as their gateway.

More info:

I'm using VirtualBox to test this setup, using separate physical network interfaces for the WAN connections.
CARP members can ping each other.
Using * * * firewall rules - not blocking anything
Exact same PfSense versions
Log files show no interesting entries

I think I'm missing a piece of the puzzle. Any suggestions are appreciated.

/edit
Created a NAT Outbound manual rule WAN - This Firewall - * - * - * - <my carp="" wan="" ip="">- * and rebooted both routers but unfortunately that didn't solve the problem. (Removed it afterwards.)

Adding some screenshots.

For the outbound nat I deleted everything, set it to Automatic, returned to Manual and changed them to the WAN CARP VIP.

Also, I am really confused by this: the docs say:

Edit the automatically added rule for LAN
Select a shared CARP virtual IP address on WAN as the Translation address
Change the Description to refer to the rule's use of the CARP VIP if desired
Click Save
[…]
NOTE: Never add outbound NAT rules that could match the WAN/Public IP addresses of the cluster.

So should I use my LAN CARP IP? (I tried but that made things much worse and seems illogical.)

Furthermore, the picture in that document uses a 127.-address as a public ip address. Am I supposed to just pick a random loopback address as WAN CARP IP?

</my>

Vorkbaard

Well, I started over. It worked then, without much trouble. Only this time WAN CARP would be master on both nodes and LAN CARP would behave. Tried different virtual network adapters, different promiscuity settings. Kept getting strange and unpredictable problems.

At that point I trashed my VM lab and built it physically. Then it worked instantly. I suppose CARP doesn't play nicely with virtualisation.

kinch

Many thanks.
I was also forgiving because CARP Interface stayed on Master for both. I am using Microsoft Hyper V on a Windows 2019 server. Will probably be the same problem.
Thanks, has saved me a lot of time.

Derelict

It plays fine with virtualization. I would switch that and say that some hypervisors don't play nicely with CARP. You don't have to do anything special in XenServer or Proxmox/KVM.

Some tweaks need to be made to some others:

https://docs.netgate.com/pfsense/en/latest/highavailability/configuring-high-availability.html?highlight=carp%20esx#vmware-esx-users

It's kind of a moving target.

kinch

Thank you for your answer.

Thanks for the input. I have examined it further. It needs to be enabled in Hyper-v vswitch MAC-spoofing, then it works.

Thank you!