@steveits them this is not for a mortal like me...thanks buddy.

@viragomann Thank you. Working perfectly now.😊

Solved it.

Diagnostic Method:

Review ALL the basics... Interfaces are same, same order (easiest for me: check the Interfaces menu item links :) ) XMLRPC Sync setup is correct: correct IP, login, pw on Master. NONE of those on Backup. Sync setup is correct in other packages (depends on pkg) Fix any errors Now make a change in the area(s) that were not syncing

In my case:

Oops: I had an IP still in "Sync Config to IP" Then, make small changes as needed... changing one static DNS assign-> All transferred changing one HAproxy item -> All transferred changing one Cert item -> all sync'd incl old/bad certs gone

etc.

@mrpete FWIW,
This effect has simply disappeared over time.

I have no idea what the root cause was.

I can't say failover is 100% reliable yet, but it "mostly works" and that's good enough for now. I need to move on to more urgent issues.

@honest_matt Not documented, but here are some hints that may help. I'm still in the process:

Convert your existing setup to use CARP VIP's (Virtual IP's - Firewall->Virtual IP) as the primary IP, and an alternate IP for direct access to that box. The CARP IP should be the gateway for any VLAN etc. It also should be what is provided as DHCP and DNS IP in DHCP server.

Change your WebGUI to specify a specific port for SSL instead of the default 443. You'll want this later with HAproxy. (It's set in System->Advanced->TCP Port)

Add an additional interface just for sync. I call mine HA. Give it its own subnet, and add a rule for HA that allows the HA net to talk to HA net freely.

Do NOT yet define HA sync stuff.

Do a backup. Save the XML file. Examine the XML. Record the interfaces assigned to WAN, LAN, and OPT1-n -- the new/mirror box must have the exact same interfaces assigned the same way.

Put the backup XML on a USB stick, name the file config.xml

Set up the mirror pfSense. Reboot with the USB stick in place, and NOT connected to your WAN or LAN. It should auto-configure itself with everything from the primary box.

Attach directly to the new box. Change the interface IP's to be different from the other box. Leave WAN undefined for now. Once that's done you should be able to attach the HA Sync ethernet.

Follow standard instructions to define HA Sync / XML-RPC on Primary and Secondary. At this point, any changes on primary should propagate to secondary.

You're on your way... there is still more to configure.

WAN DNS and DHCP sync/failover Any other failovers

It's a pretty big deal ;)

@jimp Yep. A password plugin was injecting a PW into one of the two fields.

Thanks.

@mrpete said in Shell/Cmd line for HA/CARP/VIP troubleshooting and config?:

I'm realizing that getting HA/CARP/VIP running properly is quite the detailed process, in which it as raaather easy to break any ability to access pfSense through the GUI / (V)LAN / WAN.

So (of course) I went looking for shell equivalents to the various GUI configuration and status screens related to HA/CARP/VIP.

At the very least, I want to learn how to accomplish the following from the shell:

List and Remove CARP IP's (if busted, they will conflict with the other box) List and change (or at least clear) sync settings Turn off HAproxy Create a straightforward way to keep a backup HA machine connected enough to access the Internet

Answers:

ifconfig inter.face ip.add.re.ss -alias (temporarily removes any IP, including CARP VIP) See below for a console script that can disable CARP No solution found for sync settings, but a workaround: with CARP IP's removed from the backup pfSense, I could again plug in the ethernet and access the web GUI A console "svc" script (see below) can start, stop and restart services.

Many console scripts are documented here

@yo-mismo CARP (a "variant" of VRRP) packets are sent using MAC 00:00:5e:00:01:vhid (unicast) to 01:00:5e:00:01:vhid (multicast). ARP of CARP VIP should be 00:00:5e:00:01:vhid.

Can you post a Wireshark capture that shows that every packets are sent to the multicast 01:00:5e:00:01:vhid ?

It shouldn't be multicasting traffic other than the CARP heartbeats. If it does, there may be issues with ARP proxies or something something that would interfere with it.

I have uninstalled unused packages and added via shellcmd afterfilter /erc/rc.start_packages even if it should be launched without aid. Much better, haproxy+net-snmp+ntopng autostart, but freeradius is the only one left not starting automatically.

Is this "rule" still valid, that the VIP should be the lowest IP?

Additional: without NAT, how do I attach OpenVPN to the CARP IP, doesn't it also have to be mapped/rewritten to the CARP IP?

I try to set up a CARP cluster and have issues assigning fw rules etc, because I don't see the CARP IP in the Destination dropdown.

@gpfsenser I've also added a 'gateway group' to see if this helps - seems to be a requirement to the fix. Even if it works, this certainly falls into the 'super ugly' category. Appreciate the quick reply. Everyone is using their home internet more than they ever expected - a good opportunity I hope for pfsense to polish off some of the rough edges.

@mecjay12
This solution only works for the secondary, but you still not able to access the primary, when the secondary is the master and runs the OpenVPN server.

What I suggested works for both.

@jegr Correct, it appears that pfsync/state sync was configured originally but they missed out the config sync. After that, all changes/additions were made on the secondary firewall for some reason.

Looking at it today, there are 5 virtual IPs/CARP IPs setup already but the secondary firewall has been put into "Persistent CARP Maintenance Mode" at some point too.