Hello,
I'm experiencing the same config and difficulties. Two PFsense (2.4.5p1) with CARP and two ISPs.

Have to annonce a network, with the "set nexthop" to external CARP IP.
But is it correct with openbgp to have the same network announced twice but with a different set nexthop ?

Eg (with RFC 1918 addresses) :

10.10.10.0/24 set nexthop 192.168.1.1
10.10.10.0/24 set nexthop 172.16.1.1

(or here https://forum.netgate.com/topic/51849/openbgp-with-carp-nexthop-carp-ip-carp )

Or will openbgp drop the second network announce ?

Thanks.

Alright man ofc that makes perfect sense. Many thanks for your reply.

@kom I'm using VMware ESXi v.7 (latest update).
I really don't understand why I'm experiencing this strange behaviour.

May be the problem related to the pfsense version I'm using (the latest one - v.2.5.2 community edition). I think that promiscuous mode should detect neighbours traffic, maybe it's normal. But something is wrong in pfsense that shows neighbours traffic also in STATUS->TRAFFIC GRAPH-> WAN using LOCAL filter.

Anyway, thank you for the time you spent for me.

@jokabo
So you had just to add it as type IP alias on the master, select the proper CARP address at interface, otherwise it doesn’t failover properly.

@ultrahkr the vSwitch shouldn't matter as it's not getting an IP, the ISP won't see it. What you need to do is set static MAC on the VM's, but unfortunately you cannot do that on the CARP. So, you may have to rely on DNSRR for external items coming in, if that is the concern.

@pfsenseuser1234 Why three? There's simply too much traffic for one (over 50k requests/minute, over 2Gbps of upload). Two is enough for now, third one is just to be safe.

No problem with that :)

I'm guessing pfsense can handle 50k req/min for now and I would be fine with simple active-standby approach. But what if I go to 100k by the end of the year? Or 150k. I'll probably hit some limit sooner or later. That's why I'm thinking about multi-master configuration.

I can understand the thought, but pfSense simply can't work in an active-active constellation.

*Not only volumetric DDoS (>10Gbps to kill one VM), but also SYN floods, DNS amplifications, slow HTTP requests… I get them all.

OK you could get various DDoS but the goal of all is clear: denial of service via massive flooding. Be it SYN, be it DNS, be it slow HTTPs. You can filter out some, but at the end your own firewall will give up or your upstream pipe is full. DDoS mitigation is nothing you can do on your end/site in a meaningful way. Yes you can somehow slow the process and try to mitigate the hit but with enough firepower / distribution you'll get nuked nonetheless. So if you get hit on a continuous fashion the only really helpful thing is bringing a CDN into play and (actually) try to hide the real endpoint addresses so they don't actually leak out (or the attacker will just ignore the CDN and hit the IP). That's the point where Cloudflare, Akamai or Stackpath etc. come into play.

Other then bringing a CDN into play and balancing your three proxies out for traffic reasons, I don't see a quick/simple way to bump that up further. But perhaps some other can chime in.

Cheers

@kom I run a ton of personal stuff at home as well
Mainly CCTV and file servers,
so for me its way more cost effective to run the game servers on the hardware I already have
(If I was making money from this then yes id put the money back into a VPS)
Also some of the game servers I run take a bit of grunt to run, so running on my own hardware is much cheaper

If you can tag a VLAN on that interface, traffic shaping does work with LAGG+VLANs since at that point traffic shaping is only on the VLAN, not the LAGG directly.

We hoped to have the updated pf code to let this work would be in 2.5.2 but it still needed some work and had to be backed out.

It's in 2.6.0 snapshots already but still needs work yet, may be a couple weeks before it's in a state were this would be testable in a viable way.

It resolved by using change default gateway in server. My server was using two gateways. (that uses two uplinks)

@kom
Thanks for your help, my understanding was incorrect. I got it now.

@steveits Thanks for the link, looks like that's my issue.

It's never been supported to sync between different versions, either numerically or CE vs Plus (formerly Factory). It may work by coincidence, but it's always been a gamble.

We (And FreeBSD) try hard to ensure that pfsync does not break between versions, so that isn't usually a concern. CARP is unlikely to break unless something major changes in the base OS between versions but that is also unlikely.

XMLRPC / Configuration sync is more prone to be incompatible. Primarily because of Plus vs CE releases happening at different times. They may end up on different configuration revisions and there isn't a way around that. See https://docs.netgate.com/pfsense/en/latest/releases/versions.html and look at the "Config Rev" column. So long as that matches between two HA nodes, they can do config sync.

Soon we'll have a way to run Plus on non-Netgate hardware and VMs, but it's still being worked on.

tl;dr: The type and version must always match between HA nodes, same as always.

This is the sort of thread that could be very useful for next guy.. I would assume a few people could run into such a problem... Wonder if might be useful to add note in the pfsense docs if running on proxmox on ovh.. Have to look to see what is in the offical docs if anything about running pfsense on proxmox, etc.

so today, i reloaded the HA config in the last state i left off in before my last roll back, and it turns out my issues were being caused by some typos in my CARP VIPs. this caused me to be unable to ping the expected VIP, as well as AD login was failing to find the SD since the DC needed that .1 gateway to get back to the firewall.

all good now, everything was suddenly as expected when i fixed the 2 typos in my config.

thanks everyone!

I don't mind having some level of automated failover on LAN side only. This HA idea only came to me after experiencing multiple failures of my SG-3100 and deciding to move to pfsense CE on an Ubuntu KVM as it was recreatable and servicable faster than buying a new Netgate appliance if it died (working from home).

I have multiple desktop PC's running Ubuntu Server with dockers and NAS but I really did not put my firewall into that mix so I resurrected the DL380 G6's. My employer had a "back door sale" and I got them cheap so I was running ESXi on them just to learn about it ESX.

Apparently I have a lot of toys, and sometimes don't know any better, so I tend to over complicate things, but ultimately this doesn't have to be a bullet proof enterprise class solution. I can do sneaker net and walk down to the basement and move a cable if a server dies.

If I can get two pfsense instances running on two identical servers and do some level of automated failover that would be cool. If that doesn't work out because of the ATT modem's restrictions and I have to copy the config from router1 to router2 daily and move cables during a failure, that's ok too.

I'll keep reading the comments until they make sense or my eyes blur, and probably switch back to the pfsense running on the Ubuntu KVM either tonight, or this weekend while I play with HA on the G6's. Thanks everyone for the help and the great ideas.

@viragomann I had misread this page here https://docs.netgate.com/pfsense/en/latest/firewall/virtual-ip-addresses.html got it mixed up with the CARP docs.

@matthewdaniels both are set to "HTTPS"

The error I'm getting says that the "software configuration version of the other member could not be determined." Would setting them both to use http/80 help in this case?

Hi,

after some testing I can confirm, that this still is being caused bei DHCP-Sync. After I disabled this in the HA-sync config, the firewall do sync again. So this is still a problem.

Kind Regards,
Jens

@rle I have no issues with pfBlockerNG but I'm on 2.5.1 / 3.0.0_16 + patch. I can only suggest you check the logs after having run a full reload on both nodes. Be sure that the unbound service is running without issues and that the DNSBL webserver config has no conflicting ports on the LAN interface.

@jhorne
As mentioned, on each WAN you have to set up a CARP VIP. This can then be used for any services like forwarding to an internal server later.

Additional virtual IPs on an interface have to be added as type IP alias by selecting the CARP VIP from the interface drop-down.
So if the primary firewall is going down, the VIP moves over to the secondary, cause it's hooking up on the CARP address.

@jakemurray That's a big subnet for 6 IPs. You might want to check with them on that. And yes, your WAN static config should be the IP address and the mask of the subnet, so /23 in your case.

@derelict BIG Thanks! After Your answer I can find in documentation this info about "advskew 254". In 2 topics "2.2 New Features and Changes" and "Troubleshooting High Availability". And not in main topic "CARP Status" where "Maintenance Mode" is describerd. And where both buttons placed "Temporary disable CARP" and "Enter Persistent CARP Maintenance Mode" :( So incorrect association about persistently disabling CARP.

@kiokoman Thank you Sir. You're correct.

I can see from here - https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=29&cip=155.70.7.55&ctype=ipv4&printit=0&x=109&y=13 -
that the first usable is 155.70.7.49, which will be the ISP router (pfSense default gateway) set into the WAN interface. Can I rather use 155.70.7.48, the network address in a bid not to waste IP addresses?

Invariably, is this how to reuse IPs (network and broadcast addresses)?

Pardon me, it was indeed 155.70.7.56/29. And sorry, I'm trying to learn the IP addresses by heart. In this case, can I use 155.70.7.56 in the WAN as against 155.70.7.57, the first usable IP? I'm trying to maximize the IP addresses.

@wineguy This problem with VIP seems to be common.

I just reported a similar case here - https://forum.netgate.com/topic/163533/virtual-ip-consistently-loses-connection.

So to answer my own question the only real difference I can see in practical terms is -

/26 for WAN side and CARP VIPs ties you into using a /26 whether you end up using the IPs or not.

If you end up not using them then you are wasting IPs and to get them back (assuming you could depending on how many had been used) you would need to change subnet mask of WAN side and upstream devices etc.

/29 for WAN and /26 routed to the CARP VIP gives a lot more flexibility ie. you can reserve a /26 but actually route a /28 to the CARP VIP and if you run out of IPs you can simply change the route entries on the upstream devices to use a different subnet mask.

You are still reserving the /26 but if it turns out the demand for IPs is not there then you can reuse for other purposes.

In the environment I work in where public IPs are scarce this is quite useful because it means you never overcommit on IP address allocation.

@operations HA on dynamic WANs (DHCP, PPPoE) is generally unsupported.

@viragomann Thanks for your input!
Issue solved!
It was issue basically frame untagged on switch of particular VLAN, so after tagging it works and able to connect secondary and sync!

@hege Late reply on this topic, but relevant.

Hyper-V SR-IOV implementation does NOT support mac spoofing with SR-IOV

Technical;

Mac spoofing is required for CARP because the mac address is changed on outbound packets, that's part of CARP.

Hyper-V natively does not allow outbound packets through the virtual switch from a Hyper-V guest that does not have the exact same mac address as assigned to the virtual machine (unless you enable the "allow mac spoofing" checkbox.

SR-IOV technically can allow mac spoofing, this is all there in the IEEE specification for for this is to work, but, quite simply Microsoft Hyper-V doesn't implement it.

Therefore you need to enable "allow mac spoofing" and forego SR-IOV or VMQ network accelerate functions.

@derelict said in high availability w/ redundant layer 2 switches causing loop on my test network:

People call all sorts of things a "lagg."

Very true - its a kind of a catch all.. I was thinking lacp, which yeah you need a stack..