@mi8088 The firewall was sending traffic out, but the cable modem was dropping it.

There's really only two fixes I can see:

The cable modems need to change their behavior to accommodate changes in MAC addresses. pfSense's CARP IP and all associated traffic needs to use the same MAC address that doesn't change when failing over.

I ended up disabling CARP on the WAN IP and haven't had any issues with the connection going down since.

@SteveITS Solved the issue.

After reboot works on both devices.
Thanks a lot for your support!

@mi8088 said in No traffic on a WAN CARP IP from outside, working internally and for Virtual IP:

Do you mean this behaviour?

The behavior of not allowing MAC changes on the router in front of pfSense.
I don't know any device, which doesn't let you change this.

I don't know if we can get the CPE configured somehow, our provider is claiming they can't do anything with it.

This is required for CARP, however.

Is there a way to get around it with an extra switch? (Which of course introduces another point of failure...)

Not with an L2 device. You can put an L3 switch (router) in between and nat the traffic to pfSense as its best.

However, pfSense send the response packet back from the hardware MAC, not the virtual.

Can I change this somehow?

No, pfSense will use the interface MAC, when responding. You can spoof this MAC though, but you cannot spoof the CARP vMAC, and both must be different naturally.
So the only option to make CARP work is to allow this on the connected devices.

@Gabri-91 I used the CARP Interface IP of the other pfsense box as TIER 2, not the "modem" interface IP.

First check if CARP Interface Firewall allows Traffic
Second, try to ping each other box over carp interface
Third, take a look at your NAT rules

Also check ESXi vSwitch config. For a few days, I also had problems with ESXi. Promiscuous Mode wasn't enabled anymore on vSwitch for the "modem" interface. But in this case, it was related to HAProxy on a CARP IP.

In my case, I only used physical devices as pfsense firewall, no vms. So I cannot reproduce issues related to vms and esxi.

@vizi0n Gotcha, so the goal is to spoof the MAC address on a CARP event so that the ISP router, which is handing IPs out via DHCP, gives the new primary the same IP as the previous (or ideally no DHCP even has to happen again).

Not sure if there is a way to do this, nothing is immediately coming to mind, definitely not a "supported" config but I'm sure you knew that haha.

Of course pfSense does support spoofing the MAC on an interface, but I don't know of a good way without doing a LOT of custom work to program it to do so based on a CARP event trigger.

Sorry, since saying IDK isn't really that helpful lol.

P.S. sorry for late reply, was on a work trip so didn't really have time to check forum stuff.

@viragomann Thank you very much for the reply and the references.
I am very much interested on how to best set up outgoing routing from one of the switches (ex. CS-VC1 in my diagram) towards the firewalls. Would you do a priority based routing configuration or something else?

@UserCo
How will you get the public IPs?

CARP is basically only supported with static interface IPs. I don't think, the provider will give you multiple PPPoE IPs.
And yes, if both come over a single cable you need a switch in front of the pfSense boxes.

There are some threads in the forum discussing "PPPoE as CARP VIP" though. You can use the search, maybe there are possibilities. I don't know.

@viragomann I see, and this could be the explanation because I haven't ICMP echo replies on VIP CARP too ?
Then I plan another lab in another different virtualization environment.
Always thanks and regards !

@NobleKangaroo said in Dropped inter-VLAN connections to backup CARP node and backup CARP node cannot reach internet through primary:

When doing inter-VLAN connections to the backup node (such as an SSH connection), the connection eventually times out. It doesn't matter if I set the LAN device's gateway to the primary or the backup CARP node, inter-VLAN connections to the opposite side eventually drop.

Since you should have configured an IP on each node in each VLAN, there is basically no need to access the secondary with an IP in another network segment.
If you do this however, the request packets have to pass the primary, but the secondary is sending responses directly to the client. So this ends up into an asymmetric routing and in dropping connections.

If you want to access it still this way, you can masquerade the traffic as explained in the docs: Troubleshooting VPN Connectivity to a High Availability Secondary Node

Solved. (Although I still have temporary outages after each DHCP configuration change.)

TLDR; I had the skew on my VIP addresses set to 100 and 200 instead of 0 and 100. Unfortunately I hadn't noticed this since the DHCP failover was working as expected prior to the config change. The notes in troubleshooting HA DHCP failover are worth careful study.

The smoking gun in my post above is that all the communication is on port 519 and not a mix of 519 and 520. This was caused by both of my pfSense units believing they were secondary (since the skews were high). This was found by browsing /var/dhcpd/etc/dhcpd/conf and looking under the clause marked "failover peer".

Thanks! Daryl

The opt number for the interface is probably the most important value. Everything else references that, firewall rules, NAT rules etc.

That's why you can reassign the interface to a new NIC or rename it and all the config will follow it. If you change the opt number you have to change everything that references it or recreate everything else.

The opportunity for typo'ing something here is large!

@viragomann TNX, i forgot it was there.

@Igor-Moura Happy to help.

I'm still not coming up with any reason this should be happening, quite odd if I'm being honest. It sounds like a configuration thing but I'm not sure what would actually cause that, if it were a bug though I would imagine my test or prod HA environments would be seeing it too.

I'll keep thinking on this and come back if I have any other ideas.

@stephenw10 any thoughts on this thread here? Nothing is immediately coming to mind that would cause traffic to end up on both nodes.

@SteveITS Yeah this is correct, no need for a managed switch, personally though I usually use an existing switch, setup the VLAN with tags at the port level, this way I don't need a different switch for WAN. So that's why I suggested that, but only works if you have available ports on an existing managed switch, otherwise an unmanaged one makes more sense.

Hi,

Already found out that there was a VLAN misconfiguration on slave pfSense, even if this shouldn't affect the wan and gateway functionality.
I'll check up all of the ports/cables as soon as I can.