WAN CARP IP stops responding - requires cable modem reboot

drees

I suspect this is an issue with my upstream provider (Cox Business cable internet in this case), but thought I'd reach out to see if anyone else has seen this before.

I'd been running pfSense for a long time with a single IP for a while using a 3100 with no issues. The 3100 locked up one day, power cycling brought it back up and started planning to replace it with two 4100s in a HA setup so that in case there are future hardware issues with pfSense, the internet would remain up.

Of course, since implementing this, we've now experienced two different outages that have been resolved by rebooting the cable modem.

The first time, it just looked like a typical routing issue and didn't think much of it.

But this second time I figured it was atypical and found out that each devices' external IP still responded and were accessible. Also looking at the firewall logs, the external WAN CARP IP was getting traffic, but didn't seem to be getting back out. There is just a dumb 5-port GigE switch on the WAN ports between the firewalls and cable model (I suppose it could be a switch issue in theory?). I rebooted both firewalls to see if that would resolve the issue, but it didn't.

Anyone see this type of issue before?

Also as an aside - the setup has 3 different LAN interfaces - it wasn't super clear to me, but is using the same VHID (1 in this case) on all 3 OK? It does seem to be working fine.

mi8088

@drees
When you say this:

Also looking at the firewall logs, the external WAN CARP IP was getting traffic, but didn't seem to be getting back out.

does this mean that the firewall was not sending traffic back out, or that the firewall was sending traffic out, but it wasn't getting back to the source?

mi8088

If it's the second case, I may be experiencing something similar:
https://forum.netgate.com/topic/183528/no-traffic-on-a-wan-carp-ip-from-outside-working-internally-and-for-virtual-ip

drees

@mi8088 The firewall was sending traffic out, but the cable modem was dropping it.

There's really only two fixes I can see:

The cable modems need to change their behavior to accommodate changes in MAC addresses.
pfSense's CARP IP and all associated traffic needs to use the same MAC address that doesn't change when failing over.

I ended up disabling CARP on the WAN IP and haven't had any issues with the connection going down since.