CARP outgoing traffic black hole
-
Hey all,
I'm having trouble with an HA set up with CARP VIPs. I have two firewalls, each with a static WAN IP assigned to the interface, and two CARP VIPs configured (in the same /26 subnet as the interface IPs) on two separate VHIDs.
What isn't working is outbound packets from CARP VIP->WAN. I'm inspecting all packets between the cable modem and the firewalls, and everything looks good there:
-
Pinging one of the VIPs from an LTE phone, I can see traffic to and from the firewall as I'd expect (see below) but I get 100% packet loss on the ping. Wireshark packets:
- [modem]->[master firewall] ICMP request packet
source: [phone IP/ISP gateway MAC]
destination: [VIP/multicast VHID MAC] - [master firewall]->[modem] ICMP response packet
source: [VIP/master firewall interface MAC]
destination: [phone IP/ISP gateway MAC]
- [modem]->[master firewall] ICMP request packet
-
With outbound NAT set up to route traffic via a CARP VIP, I send a ping to an external WAN destination IP (Google), and I can see the traffic leaving the firewall as I'd expect, but no reply coming back in. I get 100% packet loss on the ping. Wireshark packets:
- [master firewall]->[modem] ICMP request packet
source: [VIP/master firewall interface MAC]
destination: [Google IP/ISP gateway MAC]
- [master firewall]->[modem] ICMP request packet
The firewall interface IPs work great, inbound and outbound using the same tests above. I spent an hour on the phone with Cox Business support trying to ascertain the problem. They claim their equipment wouldn't drop any packets for any reason, but due to privacy policy they are unable to do any deep packet inspection to see what traffic looks like on their side. We tried clearing ARP cache, and ARP pinging with no success/reply.
Best guess right now: the cable modem (Arris SB6141) is unhappy about seeing two MAC addresses for one IP - VHID MAC for inbound, firewall interface MAC for outbound? Really hoping someone might have some ideas!
-
-
@meyerds said in CARP outgoing traffic black hole:
but due to privacy policy they are unable to do any deep packet inspection to see what traffic looks like on their side.
They don't need to do any such thing. All they need to do is look at the ARP and the IP headers. That is complete and utter nonsense.
Not sure about any ideas. It doesn't sound like it's pfSense.
Connect everything to something that will mimic the ISP gateway. Does it work?
If so, it's something upstream at the ISP.
-
Thanks for the reply.
Connect everything to something that will mimic the ISP gateway. Does it work?
Since my LAN interface is working with a CARP VIP, that seems to prove my networking infrastructure is compatible. I agree with you, it's something upstream from the pfSense firewalls.
The cable modem is a bridge device, so in theory it should just pass traffic without filtering, but I've read a few reports of weirdness here, dropping packets on mismatch in the ARP/route table. Unfortunately I don't know of a way to inspect that. An older topic here, which you were actually on @Derelict, seems to support my mismatch theory, but there's no detail there as to where the mismatching packets were being dropped or how to stop packets from getting dropped.
Have you heard of ISP-side issues with CARP before? Any tips on what questions to ask them to get them to find the issue on their side?
-
@meyerds said in CARP outgoing traffic black hole:
two CARP VIPs configured (in the same /26 subnet as the interface IPs)
Perhaps I'm misunderstanding but are you saying you have two CARP IPs on the WAN? (and presumably another on the LAN side) Perhaps the pings are going out one and replies being routed back on the other? Two shouldn't be needed.
Did you try getting rid of CARP temporarily and setting that IP as the WAN IP of one router? If that works it seems like it's not a routing issue with the ISP...?
-
@teamits said in CARP outgoing traffic black hole:
Perhaps I'm misunderstanding but are you saying you have two CARP IPs on the WAN? (and presumably another on the LAN side) Perhaps the pings are going out one and replies being routed back on the other? Two shouldn't be needed.
I do have two CARP VIPs on the WAN, they route to separate web servers on the LAN. Packet inspection hasn't revealed any problems with the routing. I did try with just one CARP VIP like you recommended here... switched one of the CARP VIPs to a Proxy ARP VIP just to see what would happen... the CARP VIP still doesn't work but the Proxy ARP VIP does work.
Did you try getting rid of CARP temporarily and setting that IP as the WAN IP of one router? If that works it seems like it's not a routing issue with the ISP...?
Yep turning off CARP and using one of those IPs as the interface IP works just fine. I'm thinking it's not a routing issue, but rather an ARP mismatch/packet filtering issue. But don't know how to test this since it's happening either in the modem/bridge or in the ISP CMTS/gateway.
-
Ah I understand now. We have a similar setup in our office for a VIP, though not with CARP. Did you set up a specific outbound NAT rule for the CARP IPs? (Firewall/NAT/Outbound...source=LAN-IP/32, translation/address=WAN IP) We have Manual Outbound NAT set.
-
@teamits yep manual outbound NAT so traffic coming out of each web server exits the firewall via its respective VIP. Works for the Proxy ARP VIP, but not the CARP VIP.
I'm hoping to get both working as CARP for the purpose of HA.
-
Just FYI I got an answer to this, just not the one I wanted. See my response in https://forum.netgate.com/topic/134297/cox-and-the-carp-mac/17
