CARP/NAT issue on cable modem
I am having an issue with Manual Outbound NAT not working correctly on a CARP WAN setup over a cable modem.
Provider has assigned a static /29 on the WAN. I created a new CARP WAN and LAN configuration, followed the usual setup steps from the documentation, have had this working well many times before. CARP status shows correct (master/backup) on both WAN and LAN on the pfsense status screen.
Cable modem is wired to the switch, and then each pfSense unit is connected to the switch. Pfsense units are also connected to each other for CARP SYNC.
The problem is that when Manual Outbound NAT is configured (route all LAN traffic to the CARP WAN VIP), there is significant packet loss (50-90%+) pinging the upstream WAN gateway or beyond (both from the LAN side, or from Diagnostics > Ping, when choosing the CARP WAN VIP as the source IP). I suspected this was due to the cable modem disliking seeing multiple MACs with regard to the CARP virtual IP.
The provider suggested enabling what they call "multihoming" on their side. We provide a MAC address that is permitted for each static IP which they whitelist on their side. Presumably, so ONLY those provided MAC addresses can communicate on behalf of the assigned IPs.
- CARP WAN VIP X.X.X.106 MAC 00:00:5E:00:01:05 (VHID 05)
- FW1 WAN X.X.X.107 MAC 00:08:a2:0f:98:xx
- FW2 WAN X.X.X.108 MAC 00:08:a2:0f:95:xx
But now that they enabled this option, there is 100% packet loss when Manual Outbound NAT is enabled.
Is this because the CARP/VRRP specs allow the router to use its local interface MAC to send/reply with the CARP VIP and only uses the CARP MAC for receiving?
I suspect the provider needs to whitelist the FW1 and FW2 MAC on the .106 IP (in addition to the CARP MAC) but I am not sure if they are technically capable of doing this... waiting on their response.
Has anyone else had a similar experience and do you have any suggestions?
The provider advises that due to "security protocols" on the CMTS (cable modem termination system), they do not allow multiple MACs to send on behalf of a single IP. So the packets being sent by the firewalls with the CARP VIP, originating from the firewall interface MACs and not the CARP MAC, are being filtered out. They advise there is no way to override this in their system.
I found a mention of the system tunable net.link.ether.inet.carp_mac which was available in pfSense versions prior to 2.2 which looks like it may haved solve the problem by force replies sent from the firewalls to use the CARP MAC. But apparently as of >2.2 this no longer has any effect.
Is there any other solution here? If there was a way to force the firewalls sending on the CARP VIP to use the CARP MAC, it would work...
It seems others have encountered this issue as well. See bug 9476.