@SteveITS Thank you!
Unfortunately there does not seem to be a backup option for users and/or certificates only. So looks like i'm going to have to copy those sections of config over manually.

@jeffsmith82 said in Sync not working:

used to force you to use the admin account until a relativity recent version

Oh, good to know, thanks.

@Kajetan321 the vip just points to one of them, whoever is the master. So yeah to it the name is not correct.

What you would want to setup is alternative name..

systemadv.jpg

so if you pfsense1.home.arpa, and pfsense2.home.arpa on the 2 boxes. Here for the vip name you would want pfsense.home.arpa

This is located under system / advanced / admin access

@SteveITS Thank you very much. I appreciate immensely your input. It clarified my misunderstanding.

@jimp , thanks for the directions.

@mi8088 The firewall was sending traffic out, but the cable modem was dropping it.

There's really only two fixes I can see:

The cable modems need to change their behavior to accommodate changes in MAC addresses. pfSense's CARP IP and all associated traffic needs to use the same MAC address that doesn't change when failing over.

I ended up disabling CARP on the WAN IP and haven't had any issues with the connection going down since.

@SteveITS Solved the issue.

After reboot works on both devices.
Thanks a lot for your support!

@mi8088 said in No traffic on a WAN CARP IP from outside, working internally and for Virtual IP:

Do you mean this behaviour?

The behavior of not allowing MAC changes on the router in front of pfSense.
I don't know any device, which doesn't let you change this.

I don't know if we can get the CPE configured somehow, our provider is claiming they can't do anything with it.

This is required for CARP, however.

Is there a way to get around it with an extra switch? (Which of course introduces another point of failure...)

Not with an L2 device. You can put an L3 switch (router) in between and nat the traffic to pfSense as its best.

However, pfSense send the response packet back from the hardware MAC, not the virtual.

Can I change this somehow?

No, pfSense will use the interface MAC, when responding. You can spoof this MAC though, but you cannot spoof the CARP vMAC, and both must be different naturally.
So the only option to make CARP work is to allow this on the connected devices.

@Gabri-91 I used the CARP Interface IP of the other pfsense box as TIER 2, not the "modem" interface IP.

First check if CARP Interface Firewall allows Traffic
Second, try to ping each other box over carp interface
Third, take a look at your NAT rules

Also check ESXi vSwitch config. For a few days, I also had problems with ESXi. Promiscuous Mode wasn't enabled anymore on vSwitch for the "modem" interface. But in this case, it was related to HAProxy on a CARP IP.

In my case, I only used physical devices as pfsense firewall, no vms. So I cannot reproduce issues related to vms and esxi.

@vizi0n Gotcha, so the goal is to spoof the MAC address on a CARP event so that the ISP router, which is handing IPs out via DHCP, gives the new primary the same IP as the previous (or ideally no DHCP even has to happen again).

Not sure if there is a way to do this, nothing is immediately coming to mind, definitely not a "supported" config but I'm sure you knew that haha.

Of course pfSense does support spoofing the MAC on an interface, but I don't know of a good way without doing a LOT of custom work to program it to do so based on a CARP event trigger.

Sorry, since saying IDK isn't really that helpful lol.

P.S. sorry for late reply, was on a work trip so didn't really have time to check forum stuff.

@viragomann Thank you very much for the reply and the references.
I am very much interested on how to best set up outgoing routing from one of the switches (ex. CS-VC1 in my diagram) towards the firewalls. Would you do a priority based routing configuration or something else?

@UserCo
How will you get the public IPs?

CARP is basically only supported with static interface IPs. I don't think, the provider will give you multiple PPPoE IPs.
And yes, if both come over a single cable you need a switch in front of the pfSense boxes.

There are some threads in the forum discussing "PPPoE as CARP VIP" though. You can use the search, maybe there are possibilities. I don't know.

@viragomann I see, and this could be the explanation because I haven't ICMP echo replies on VIP CARP too ?
Then I plan another lab in another different virtualization environment.
Always thanks and regards !

@NobleKangaroo said in Dropped inter-VLAN connections to backup CARP node and backup CARP node cannot reach internet through primary:

When doing inter-VLAN connections to the backup node (such as an SSH connection), the connection eventually times out. It doesn't matter if I set the LAN device's gateway to the primary or the backup CARP node, inter-VLAN connections to the opposite side eventually drop.

Since you should have configured an IP on each node in each VLAN, there is basically no need to access the secondary with an IP in another network segment.
If you do this however, the request packets have to pass the primary, but the secondary is sending responses directly to the client. So this ends up into an asymmetric routing and in dropping connections.

If you want to access it still this way, you can masquerade the traffic as explained in the docs: Troubleshooting VPN Connectivity to a High Availability Secondary Node

Solved. (Although I still have temporary outages after each DHCP configuration change.)

TLDR; I had the skew on my VIP addresses set to 100 and 200 instead of 0 and 100. Unfortunately I hadn't noticed this since the DHCP failover was working as expected prior to the config change. The notes in troubleshooting HA DHCP failover are worth careful study.

The smoking gun in my post above is that all the communication is on port 519 and not a mix of 519 and 520. This was caused by both of my pfSense units believing they were secondary (since the skews were high). This was found by browsing /var/dhcpd/etc/dhcpd/conf and looking under the clause marked "failover peer".

Thanks! Daryl

The opt number for the interface is probably the most important value. Everything else references that, firewall rules, NAT rules etc.

That's why you can reassign the interface to a new NIC or rename it and all the config will follow it. If you change the opt number you have to change everything that references it or recreate everything else.

The opportunity for typo'ing something here is large!