No traffic on a WAN CARP IP from outside, working internally and for Virtual IP

mi8088

Hi,

our aim is to set up a cluster of two Netgate 4100s in HA mode at one site. For now, one is active, but we are having trouble with getting the WAN IPs working with CARP.

We have a set of WAN IP addresses from our ISP, call them:

x.111
x.112
x.113
x.114
x.115

The intention is to use x.111 and x.112 for the two devices (interface address), and use the others as IPs for services, ie have them as CARP or Virtual IPs over CARP.

Right now, we are using only one of the firewalls.

For now, we have set one of the WAN IPs as CARP. CARP is enabled in multicast mode, showing "Master" status and one sync host ("this node"). I guess this is what is expected when there is only one host for CARP.

The other two WAN IPs, x.114 and x.115, are still set up as virtual IPs on device A.

When we do this, we are not able to contact x.113 from external sources.
Contacting x.111, x.114 and x.115 all work. By contact, I mean mainly ping, but similar things seem to happen on other protocols.

The firewall is connected to a CPE switch which we have no control over. We have tried putting a dumb switch between our firewall and the CPE, not much result here.

From an external source

I run ping x.111 or the other WAN IPs from a host at a different site, I've called it Y.178. This traffic is allowed and not blocked by firewalls. This has been tested and worked perfectly before enabling CARP (ie with Virtual IPs only).

A packet capture shows that the ping request reaches the firewall, and that a reply is sent out, eg:

10:57:13.230572 IP Y.178 > x.113: ICMP echo request, id 16351, seq 7, length 64
10:57:13.230588 IP x.113 > Y.178: ICMP echo reply, id 16351, seq 7, length 64

I get no reply at Y.178.

This looks exactly the same as when I try to ping x.111 or ping x.115:

11:16:44.134888 IP Y.178 > X.115: ICMP echo request, id 12568, seq 3, length 64
11:16:44.134898 IP X.115 > Y.178: ICMP echo reply, id 12568, seq 3, length 64

In these cases, I get the expected reply at Y.178.

From the firewall, or an internal source

If I try the same ping of the WAN IPs from the firewall itself or a host on the LAN side, I get replies from all IPs.

Some other things we have tried

We have tried:

setting the IP x.113 as a virtual IP, not CARP. It then responds as expected.
disconnected the firewalls from the CPE, and connected them to a laptop acting as a router. In this case, all the IPs respond as expected externally.
We have tried setting up the second firewall as well, with CARP both in multicast and in unicast mode. This doesn't make anything better yet.
CARP on the LAN Interface: this works as expected, we have not seen anything weird going on here.

Questions

It seems that the traffic from the CARP IP disappears somewhere en route back.

Main question: is it possible that the CPE or the service provided is doing something that blocks traffic because it is from a CARP IP?

Any other suggestions of things to try out are welcome.

viragomann

@mi8088
Seems the device in front of pfSense WAN does not allow MAC address changes. This is necessary for CARP to work though.

In fact, if the device wants to send a packet to the CARP VIP (or any other VIP hooking up on it), it resolves the IP to the virtual CARP MAC via ARP and sends the packet to this virtual MAC then. However, pfSense send the response packet back from the hardware MAC, not the virtual.
Some devices do not accept such responses from a different MAC by default. But this behavior is mostly changeable.

mi8088

@viragomann thanks for the comment, that could be the explanation!

But this behavior is mostly changeable.

Do you mean this behaviour?

Some devices do not accept such responses from a different MAC by default.

I don't know if we can get the CPE configured somehow, our provider is claiming they can't do anything with it.
Is there a way to get around it with an extra switch? (Which of course introduces another point of failure...)

or did you mean this behaviour can be changed:

However, pfSense send the response packet back from the hardware MAC, not the virtual.

Can I change this somehow?
The docs say the following:

or when having a unique MAC address is a requirement. In rare cases a provider requires each unique IP address on a WAN segment to have a distinct MAC address, which CARP VIPs provide.

viragomann

@mi8088 said in No traffic on a WAN CARP IP from outside, working internally and for Virtual IP:

Do you mean this behaviour?

The behavior of not allowing MAC changes on the router in front of pfSense.
I don't know any device, which doesn't let you change this.

I don't know if we can get the CPE configured somehow, our provider is claiming they can't do anything with it.

This is required for CARP, however.

Is there a way to get around it with an extra switch? (Which of course introduces another point of failure...)

Not with an L2 device. You can put an L3 switch (router) in between and nat the traffic to pfSense as its best.

However, pfSense send the response packet back from the hardware MAC, not the virtual.

Can I change this somehow?

No, pfSense will use the interface MAC, when responding. You can spoof this MAC though, but you cannot spoof the CARP vMAC, and both must be different naturally.
So the only option to make CARP work is to allow this on the connected devices.