Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP outgoing traffic black hole

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    8 Posts 3 Posters 992 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      meyerds
      last edited by

      Hey all,

      I'm having trouble with an HA set up with CARP VIPs. I have two firewalls, each with a static WAN IP assigned to the interface, and two CARP VIPs configured (in the same /26 subnet as the interface IPs) on two separate VHIDs.

      What isn't working is outbound packets from CARP VIP->WAN. I'm inspecting all packets between the cable modem and the firewalls, and everything looks good there:

      1. Pinging one of the VIPs from an LTE phone, I can see traffic to and from the firewall as I'd expect (see below) but I get 100% packet loss on the ping. Wireshark packets:

        1. [modem]->[master firewall] ICMP request packet
          source: [phone IP/ISP gateway MAC]
          destination: [VIP/multicast VHID MAC]
        2. [master firewall]->[modem] ICMP response packet
          source: [VIP/master firewall interface MAC]
          destination: [phone IP/ISP gateway MAC]

      2. With outbound NAT set up to route traffic via a CARP VIP, I send a ping to an external WAN destination IP (Google), and I can see the traffic leaving the firewall as I'd expect, but no reply coming back in. I get 100% packet loss on the ping. Wireshark packets:

        1. [master firewall]->[modem] ICMP request packet
          source: [VIP/master firewall interface MAC]
          destination: [Google IP/ISP gateway MAC]

      The firewall interface IPs work great, inbound and outbound using the same tests above. I spent an hour on the phone with Cox Business support trying to ascertain the problem. They claim their equipment wouldn't drop any packets for any reason, but due to privacy policy they are unable to do any deep packet inspection to see what traffic looks like on their side. We tried clearing ARP cache, and ARP pinging with no success/reply.

      Best guess right now: the cable modem (Arris SB6141) is unhappy about seeing two MAC addresses for one IP - VHID MAC for inbound, firewall interface MAC for outbound? Really hoping someone might have some ideas!

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        @meyerds said in CARP outgoing traffic black hole:

        but due to privacy policy they are unable to do any deep packet inspection to see what traffic looks like on their side.

        They don't need to do any such thing. All they need to do is look at the ARP and the IP headers. That is complete and utter nonsense.

        Not sure about any ideas. It doesn't sound like it's pfSense.

        Connect everything to something that will mimic the ISP gateway. Does it work?

        If so, it's something upstream at the ISP.

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        M 1 Reply Last reply Reply Quote 0
        • M
          meyerds @Derelict
          last edited by

          Thanks for the reply.

          Connect everything to something that will mimic the ISP gateway. Does it work?

          Since my LAN interface is working with a CARP VIP, that seems to prove my networking infrastructure is compatible. I agree with you, it's something upstream from the pfSense firewalls.

          The cable modem is a bridge device, so in theory it should just pass traffic without filtering, but I've read a few reports of weirdness here, dropping packets on mismatch in the ARP/route table. Unfortunately I don't know of a way to inspect that. An older topic here, which you were actually on @Derelict, seems to support my mismatch theory, but there's no detail there as to where the mismatching packets were being dropped or how to stop packets from getting dropped.

          Have you heard of ISP-side issues with CARP before? Any tips on what questions to ask them to get them to find the issue on their side?

          1 Reply Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire
            last edited by

            @meyerds said in CARP outgoing traffic black hole:

            two CARP VIPs configured (in the same /26 subnet as the interface IPs)

            Perhaps I'm misunderstanding but are you saying you have two CARP IPs on the WAN? (and presumably another on the LAN side) Perhaps the pings are going out one and replies being routed back on the other? Two shouldn't be needed.

            Did you try getting rid of CARP temporarily and setting that IP as the WAN IP of one router? If that works it seems like it's not a routing issue with the ISP...?

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            M 1 Reply Last reply Reply Quote 0
            • M
              meyerds @SteveITS
              last edited by

              @teamits said in CARP outgoing traffic black hole:

              Perhaps I'm misunderstanding but are you saying you have two CARP IPs on the WAN? (and presumably another on the LAN side) Perhaps the pings are going out one and replies being routed back on the other? Two shouldn't be needed.

              I do have two CARP VIPs on the WAN, they route to separate web servers on the LAN. Packet inspection hasn't revealed any problems with the routing. I did try with just one CARP VIP like you recommended here... switched one of the CARP VIPs to a Proxy ARP VIP just to see what would happen... the CARP VIP still doesn't work but the Proxy ARP VIP does work.

              Did you try getting rid of CARP temporarily and setting that IP as the WAN IP of one router? If that works it seems like it's not a routing issue with the ISP...?

              Yep turning off CARP and using one of those IPs as the interface IP works just fine. I'm thinking it's not a routing issue, but rather an ARP mismatch/packet filtering issue. But don't know how to test this since it's happening either in the modem/bridge or in the ISP CMTS/gateway.

              1 Reply Last reply Reply Quote 0
              • S
                SteveITS Galactic Empire
                last edited by

                Ah I understand now. We have a similar setup in our office for a VIP, though not with CARP. Did you set up a specific outbound NAT rule for the CARP IPs? (Firewall/NAT/Outbound...source=LAN-IP/32, translation/address=WAN IP) We have Manual Outbound NAT set.

                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                Upvote 👍 helpful posts!

                M 1 Reply Last reply Reply Quote 0
                • M
                  meyerds @SteveITS
                  last edited by

                  @teamits yep manual outbound NAT so traffic coming out of each web server exits the firewall via its respective VIP. Works for the Proxy ARP VIP, but not the CARP VIP.

                  I'm hoping to get both working as CARP for the purpose of HA.

                  1 Reply Last reply Reply Quote 0
                  • M
                    meyerds
                    last edited by

                    Just FYI I got an answer to this, just not the one I wanted. See my response in https://forum.netgate.com/topic/134297/cox-and-the-carp-mac/17

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.