CARP Single WAN IP Odd Behavior and Issues

CaptRyco

Hello. Let me just say that I am not a newbie to pfSense, I've previously configured CARP with dual WAN failover with a public /27 subnet.

I'm working on a new CARP setup with a single WAN connection and one single static IP address. For the sake of simplicity, forget the 2nd pfSense box and assume it's in carp maintenance mode.

Problem: When using a 'CARP' type virtual IP for the WAN I am unable to reach my public gateway. The second I change from CARP to 'IP Alias' type everything works as expected and I can reach my public gateway and ping my gateway. Of course this defeats the purpose of CARP fail over.

Firewall A: (SUPERMICRO MBD-A1SRi-2758F-O)

WAN: 172.16.146.2/24
LAN: 192.168.4.2/21
CARP: 172.16.145.1/24 (dedicated interface)

CARP LAN IP: 192.168.4.1
CARP / IP Alias WAN IP: 100.100.100.66 (fake)
WAN Gateway: 100.100.100.1

Some following things I've tried:

Changing WAN CARP IP to a 'local' 172.16.146.1 and then adding the real WAN IP as a virtual IP using the virtual CARP WAN interface.
Changing gateway settings (disabling monitoring, use non-local gateway option, adding gateway to WAN interface entry, etc)
Making sure extra services are disabled such as squid, snort, openvpn, etc.
Restarts, basic stuff, removing the switch in front of the WAN connections, etc

I've looked at some posts regarding CARP with a single IP and I have done everything correctly, but cannot figure this one out. Any help would be greatly appreciated.

dotdash

I have this running at a site, here are some notes. Don't know if any of these are relevant/necessary, but they work for me.
Note that the secondary box is not available from the public side. I haven't upgraded this one to 2.3.2 yet, still on 2.2.6

Subnet on WAN matches the real subnet mask.
Block private is unchecked on WAN.
Have an outbound NAT rule:
WAN 'This Firewall' * * * (public IP CARP) * NO

CaptRyco

Thanks for the reply. My subnet for my 'private' WAN address and CARP address are both /24 (255.255.255.0). Block private is of course un-checked as well.

I just added the rule 'WAN 'This Firewall' * * * (public IP CARP) * NO' and now I can use dpinger and not just disable the gateway monitoring which was on my to-do list.

Do you think this rule will affect my virtual CARP WAN issue though? I just find it so odd that 'IP Alias' works, but a CARP IP does not.

dotdash

@CaptRyco:

Do you think this rule will affect my virtual CARP WAN issue though? I just find it so odd that 'IP Alias' works, but a CARP IP does not.

The configuration is working for me with a CARP VIP, not sure what the issue is with your setup. You checked the usual stuff? /24 mask on the carp, shows as master in carp status, management rules reflect public ip, not 'wan address'…

CaptRyco

So after pulling my hair out for a few more hours, going back to a stock configuration, removing options in /boot/config.conf.local, and trying a few more things I finally figured out what's going on.

My setup is perfect. This is validated by changing my public IP address from 'CARP' to 'IP Alias'. Everything works great in 'IP Alias' mode, but of course this lacks failover for the WAN.

Everytime I switched to 'CARP' for my public virtual IP I would lose connectivity to my public gateway and I would see this in the system.log:

pfsense01 kernel: arpresolve: can't allocate llinfo for <public ip="" address=""> on igb0</public>

This led me to some more googling around and stumbling upon this: http://marc.info/?l=pfsense-support&m=129985175716097

Here's an update on this issue that may prove helpful for other
customers that reside on a passive optical network and want to use
CARP. The service provider, service provider hardware vendor, and I
have performed extensive troubleshooting on why CARP was not working
over the passive optical network, and we have gotten it working.
Basically, there were 2 issues on the Service Provider side that was
preventing the CARP IP's from being accessible from beyond the ONT.

First the typology. The redundant set of pfsense boxes were plugged
into a managed layer 2 switch. The ONT was plugged into this switch
also. The ONT terminated at the service provider's local point of
presence into a managed layer 2 switch. Up-links from the service
provider's switch led to a router in their core network.

2. The Service provider, in order to make its standard residential-type
configuration work efficiently, had proxy-arp enabled on its router.
This being enabled was causing the original problem of the secondary
firewall not being able to ping the CARP IP's. Disabling proxy-arp for
the Pfsense and CARP IP's on the service provider's router fixed this
initial issue, but we could still not communicate with the CARP IP's
from a remote network or from inside the Service Provider's network.

3. The Service provider, as a security item, had its local point of
presence switch configured to not allow communication between switch
ports (even on the same VLAN). Any local traffic bound for other local
traffic on the switch had to first travel to the provider's core
network. Unfortunately, this broke CARP by preventing the core router
from asking for which MAC's belonged to the CARP IP's. The service
provider removed this restriction for the IP's in use by the PFsense box
and CARP, and things started working correctly.

I hope this helps others who may have Internet Service over a Passive
Optical Network and are having trouble getting CARP working.

This makes perfect sense as to why CARP fails for me. My ISP is a local telcom that just performed a fiber rollout previously to mostly residential customers. I'm now going to reach out to my ISP to see if we can get CARP working properly as I'm 99% sure the problem is due to my ISP.

carp_vs_alias.png_thumb