Switches getting wrong MAC for CARP interface
-
Has any one seen this.
Aruba and Meraki switches. Two pfSense+ 24.11 devices running CARP/VIP. VIP set up properly. Using subnet mask of subnet, not /32 like some people do, no VHID over laps, and nothing else on the network running VRRP or CARP.
All works fine until it does not. Everything works and does for weeks and then it seems that all of a sudden, all devices on the network loose routing to the gateway, aka the VIP. It seems to happen when load goes up, basically lots of packets, but low throughput. To "fix" this, you remove the VIP, assign it directly to the interface, and walla. Everything works. While looking in the Meraki switches, I see two MACs for the VIP. One is the physical NIC, the other is the VRRP NIC. This was before the GARP that is sent when you apply a new IP on the interface.
The Arubas have a very basic config, just VLANs. IGMP snooping on both the Arubas and Merakis are disabled.
We have never seen this before with any other installs, but this is the only install that have Meraki and Aruba switches.
Thoughts?
-
@beloc said in Switches getting wrong MAC for CARP interface:
While looking in the Meraki switches, I see two MACs for the VIP. One is the physical NIC, the other is the VRRP NIC.
It's by design of CARP that you can resolve the VIP to the CARP MAC, but if the device responses it uses its hardware interface MAC.
Some devices don't like this though and you have to allow MAC spoofing to enable communication with the CARP VIP.
-
So spoofing on the switches or the pfsense? I am assuming the switches.
-
@beloc
The connected devices (switches) have to allow MAC spoofing.
There is nothing you can do on pfSense. -
Strange that it works for weeks, then suddenly stops and requires spoofing to be enabled on the switches. Everything is possible, of course... But in theory, it shouldn't work from the beginning. What’s in the pfSense logs at that moment?
-
@w0w I don’t remember anything specific that jumped out being in the logs, but I will say they have a ton of flapping Mac addresses out there. They also are using Shortel phones and the mini switches within the phones for some of the desks. Now, whether or not this causes an issue is unknown, but I know I’m not a big fan of using those switches and have seen them cause issues on other networks. Primarily with a device that’s connected to it, but this is a strange issue.
-
@beloc said in Switches getting wrong MAC for CARP interface:
but I will say they have a ton of flapping Mac addresses out there.
I’m not sure, but this might be triggering the MAC-spoofing issue, so try enabling it.
-
Just for reference: https://docs.netgate.com/pfsense/en/latest/highavailability/index.html#switch-layer-2-concerns
"switch must...Allow traffic to be sent and received using multiple MAC addresses"
