CARP without multicast

bbruun

I have a problem with how our ISP aka hosting partner where we have a rack handles internet traffic.
They do not allow multicast traffic onto their network (it is under storm control) so we are not able to setup PFSense in HA as CARP utilizes multicast for heartbeats.

We are in the process of getting a new full rack and need to have it isolated and have thought about getting two XG-7100 and set them up in HA to avoid going to the hosting center "all the time".

From what my hosting provider says (2 of them as one of them purchased the other) then it is normal for hosting providers to not allow/dissallow multicast on their network. I can sorta understand and respect that but it leaves me in a situation where I can't get HA, unless someone here knows how.

I've tried to setup PFSense HA our ESX environment (same provider) with CARP and it semi/sorta works. The WAN interface works and can get its IP address on the network just fine, but not always on the CARP addresses - and failover does not work at all.
I'm a networking noob on this level and would like to keep PFSense as our firewall technology in our company and not either run singleton using just WAN and IP Alias on it or switch entirely to a different firewall provider where they use different technologies to do failover.

I can get the hosting provider to deliver internet via unmanaged (and managed) BGP to two frontend switches on the outside, but I don't know if that will solve the problem with the multicast if that is how CARP utilizes network. Setting WAN and IP Alias and Proxy APR addresses work fine, but PFSense does not do failover on these.

Is it possible to setup PFSense HA without multicast and using a dedicated crossover cable between the boxes for SYNC ? Is there some CLI tool that can be used to do a manual fail over between the boxes if they have dedicated WAN IP and IP Alias where the IP Aliases are dynamic, aka are not setup/enabled by default but by a CLI/background script?

TL;DR; CARP uses multicast. Hosting provider does not allow multicast. How does one go about setting up HA?

Derelict

Put your own layer 2 gear that passes multicast traffic between the ISP and you?

bbruun

We are noobs when it comes to this kind of setup - where low level CARP workings rise to the surfact...

We've setup PFSense in HA multiple times, but that is apparently on networks that allow multicast.

We have come to the same conclusion - aka have a set of switches in front of PFSense that limits multicast, but ... now we are in a dual WAN (cross linked between the firewalls to the switches) so we are going to learn 2 things in one go (if that is even possible)

  Internet
   /     \
 sw-1    sw-2
 |   \  /  |
 |    /\   |
 |  /    \ |
pf-1     pf-2
 |   \  /  |
 |    /\   |
 |  /    \ |
lan-1    lan-2

That might resolve in a new question here (dual WAN to 2 switches using CARP), but for now I'd just like to thank you @Derelict for the answer.

Derelict

Just so you know it wouldn't work with VRRP or HSRP either. It's not just CARP.

As long as you are not trying to HA between two different ESXi hosts, you should be able to get it working in the vswitch between two pfSense guests.

https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability-virtual.html

bbruun

@derelict it is a physical setup, not virtual.

I have the core network setup with BGP routing working via FRR.
The 2 BGP legs are on separate subnets, so CARP multicast isn't working...

I've been working hard on this setup and have (via a lot of reading and fustration over the CARP functionality not beinging configurable to something else eg simple ping's and then sync the state over the SYNC interface) have ended up with something like the following:

The two BGP routes are connected to two switches (which are stacked for failover and LAGG/LACP) and then setup LACP on the WAN interfaces on the XG-7100's so they are connected to each BGP and then have a network specialist (I'm dumb when it comes to network equipment) configure the switches for me so LAGG/LACP is isolated to 3 ports and multicast is kept on the ports the WAN ports are located in to avoid network spam.

I hope this will work, but I need to read up on the XG-7100 to setup WAN LACP and find a way to test it without the actual switch(es) to avoid downtime. For some reason Netgate likes to use switches and obscure non ethX naming schemes for everything and not expose information about the physical layer before actually configured PFSense makeing i difficult for a (PFSense/FreeBSD) noob to get up 2 speed.

There is a first time for everything.

Thanks for your updates and sorry for the very long delay in the update - I had to get moving on this project faster than expected and have just finised to the above state where WAN LACP and stacking switch configuration is needed (even our hosting partner had issues with BGP due to PFSense not being Cisco and for some reason FRR had issues with the BGP password causing weired issues and one of the BGP CPE's fail on ARP refresh from time to time, so it has been a fustrating and slow process).