It's never been supported to sync between different versions, either numerically or CE vs Plus (formerly Factory). It may work by coincidence, but it's always been a gamble.

We (And FreeBSD) try hard to ensure that pfsync does not break between versions, so that isn't usually a concern. CARP is unlikely to break unless something major changes in the base OS between versions but that is also unlikely.

XMLRPC / Configuration sync is more prone to be incompatible. Primarily because of Plus vs CE releases happening at different times. They may end up on different configuration revisions and there isn't a way around that. See https://docs.netgate.com/pfsense/en/latest/releases/versions.html and look at the "Config Rev" column. So long as that matches between two HA nodes, they can do config sync.

Soon we'll have a way to run Plus on non-Netgate hardware and VMs, but it's still being worked on.

tl;dr: The type and version must always match between HA nodes, same as always.

This is the sort of thread that could be very useful for next guy.. I would assume a few people could run into such a problem... Wonder if might be useful to add note in the pfsense docs if running on proxmox on ovh.. Have to look to see what is in the offical docs if anything about running pfsense on proxmox, etc.

so today, i reloaded the HA config in the last state i left off in before my last roll back, and it turns out my issues were being caused by some typos in my CARP VIPs. this caused me to be unable to ping the expected VIP, as well as AD login was failing to find the SD since the DC needed that .1 gateway to get back to the firewall.

all good now, everything was suddenly as expected when i fixed the 2 typos in my config.

thanks everyone!

I don't mind having some level of automated failover on LAN side only. This HA idea only came to me after experiencing multiple failures of my SG-3100 and deciding to move to pfsense CE on an Ubuntu KVM as it was recreatable and servicable faster than buying a new Netgate appliance if it died (working from home).

I have multiple desktop PC's running Ubuntu Server with dockers and NAS but I really did not put my firewall into that mix so I resurrected the DL380 G6's. My employer had a "back door sale" and I got them cheap so I was running ESXi on them just to learn about it ESX.

Apparently I have a lot of toys, and sometimes don't know any better, so I tend to over complicate things, but ultimately this doesn't have to be a bullet proof enterprise class solution. I can do sneaker net and walk down to the basement and move a cable if a server dies.

If I can get two pfsense instances running on two identical servers and do some level of automated failover that would be cool. If that doesn't work out because of the ATT modem's restrictions and I have to copy the config from router1 to router2 daily and move cables during a failure, that's ok too.

I'll keep reading the comments until they make sense or my eyes blur, and probably switch back to the pfsense running on the Ubuntu KVM either tonight, or this weekend while I play with HA on the G6's. Thanks everyone for the help and the great ideas.

@viragomann I had misread this page here https://docs.netgate.com/pfsense/en/latest/firewall/virtual-ip-addresses.html got it mixed up with the CARP docs.

@matthewdaniels both are set to "HTTPS"

The error I'm getting says that the "software configuration version of the other member could not be determined." Would setting them both to use http/80 help in this case?

Hi,

after some testing I can confirm, that this still is being caused bei DHCP-Sync. After I disabled this in the HA-sync config, the firewall do sync again. So this is still a problem.

Kind Regards,
Jens

@rle I have no issues with pfBlockerNG but I'm on 2.5.1 / 3.0.0_16 + patch. I can only suggest you check the logs after having run a full reload on both nodes. Be sure that the unbound service is running without issues and that the DNSBL webserver config has no conflicting ports on the LAN interface.

@jhorne
As mentioned, on each WAN you have to set up a CARP VIP. This can then be used for any services like forwarding to an internal server later.

Additional virtual IPs on an interface have to be added as type IP alias by selecting the CARP VIP from the interface drop-down.
So if the primary firewall is going down, the VIP moves over to the secondary, cause it's hooking up on the CARP address.

@jakemurray That's a big subnet for 6 IPs. You might want to check with them on that. And yes, your WAN static config should be the IP address and the mask of the subnet, so /23 in your case.

@derelict BIG Thanks! After Your answer I can find in documentation this info about "advskew 254". In 2 topics "2.2 New Features and Changes" and "Troubleshooting High Availability". And not in main topic "CARP Status" where "Maintenance Mode" is describerd. And where both buttons placed "Temporary disable CARP" and "Enter Persistent CARP Maintenance Mode" :( So incorrect association about persistently disabling CARP.

@kiokoman Thank you Sir. You're correct.

I can see from here - https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=29&cip=155.70.7.55&ctype=ipv4&printit=0&x=109&y=13 -
that the first usable is 155.70.7.49, which will be the ISP router (pfSense default gateway) set into the WAN interface. Can I rather use 155.70.7.48, the network address in a bid not to waste IP addresses?

Invariably, is this how to reuse IPs (network and broadcast addresses)?

Pardon me, it was indeed 155.70.7.56/29. And sorry, I'm trying to learn the IP addresses by heart. In this case, can I use 155.70.7.56 in the WAN as against 155.70.7.57, the first usable IP? I'm trying to maximize the IP addresses.

@wineguy This problem with VIP seems to be common.

I just reported a similar case here - https://forum.netgate.com/topic/163533/virtual-ip-consistently-loses-connection.

So to answer my own question the only real difference I can see in practical terms is -

/26 for WAN side and CARP VIPs ties you into using a /26 whether you end up using the IPs or not.

If you end up not using them then you are wasting IPs and to get them back (assuming you could depending on how many had been used) you would need to change subnet mask of WAN side and upstream devices etc.

/29 for WAN and /26 routed to the CARP VIP gives a lot more flexibility ie. you can reserve a /26 but actually route a /28 to the CARP VIP and if you run out of IPs you can simply change the route entries on the upstream devices to use a different subnet mask.

You are still reserving the /26 but if it turns out the demand for IPs is not there then you can reuse for other purposes.

In the environment I work in where public IPs are scarce this is quite useful because it means you never overcommit on IP address allocation.

@operations HA on dynamic WANs (DHCP, PPPoE) is generally unsupported.

@viragomann Thanks for your input!
Issue solved!
It was issue basically frame untagged on switch of particular VLAN, so after tagging it works and able to connect secondary and sync!