CARP/HA not working
-
-
@viragomann found it in the netgate manual. But CARP on the second FW still says:
-
@nick-loenders
Presumably the CARP VIPs were not synced over to the backup, since the sync wasn't allowed before.
Maybe you succeed when you edit the CARP VIPs and save them again.Otherwise check the system log on both for hints.
-
@viragomann I see this on one :
/rc.filter_synchronize: The Netgate pfSense Plus software configuration version of the other member could not be determined. Skipping synchronization to avoid causing a problem!but both run on
Version 21.05-RELEASE (amd64)
built on Tue Jun 01 16:52:56 EDT 2021
FreeBSD 12.2-STABLE -
@viragomann is the rest oke and alos the cable on ETH8 ? IS this a good connection? It is not a crosscable.
I also found this:
https://forum.netgate.com/topic/80092/resolved-carp-not-failing-back-and-other-weird-behaviour-on-pfsense-2-2/12where it says:
To add closure to this issue, the problem went away by resetting sysctl net.inet.carp.demotion from 240 to 0 with:sysctl net.inet.carp.demotion=-240
sysctl net.inet.carp.demotion is essentially a penalty against the advskew settings. Returning this to 0 made the VIPs stable and removed the warning from the CARP status page, though it would recur following a reboot.
According to https://forum.pfsense.org/index.php?topic=89132.msg496865#msg496865 the problem is caused when using CARP on a LAGG. When the LAGG is initialised it loses some CARP advertisements and causes net.inet.carp.demotion to be increased by the value of net.inet.carp.senderr_demotion_factor (240).
Setting:
net.inet.carp.senderr_demotion_factor=0
But where can I add this setting??
-
@nick-loenders said in CARP/HA not working:
@viragomann I see this on one :
/rc.filter_synchronize: The Netgate pfSense Plus software configuration version of the other member could not be determined. Skipping synchronization to avoid causing a problem!Ensure that you allow also access to the webconfigurator port (TCP) on the Sycn interface.
Maybe this can prevent this issue.Setting:
net.inet.carp.senderr_demotion_factor=0
But where can I add this setting??Never changed this setting. My sync is on a seperate dedicated interface.
-
@nick-loenders If I'm reading your picture right the "Synchronize Config to IP" is set to your sync interface IP, and it should be the LAN IP of router2.
-
@steveits
Nope, it's fine to use the sync ip of the other box.OP- I've always used a dedicated physical interface for sync. I'm not familiar with that switchport hardware, but could you use a straight vlan interface instead of a lag?
-
@dotdash said in CARP/HA not working:
fine to use the sync ip
Actually the docs say it "should use the Sync interface." Wonder why ours was set up otherwise, years ago.
@nick-loenders on router2 the Synchronize Config to IP field should be blank, you don't want the secondary syncing back to the primary. Also did you find the troubleshooting doc?
-
@steveits Hi, I changed it on the 2nd device, but still no luck :(
-
I just enable ipv4 any to any from any on the sync interface. I have mine configured as a /30 plugged directly from one box to the other.
-
-
@nick-loenders
I've never worked with a model with the switchport setup. There's a note in the manual about CARP limitations due to the switchport not going down. Do you have the expansion riser? I'd get a couple of quad port intel cards and use those ports. -
There are a number of issues here:
The Sync VLAN on the switch id configured incorrectly.
You need to add ports 9 and 10 as tagged members to VLAN 8 so that it is passed from the LAGG.Config sync should only ever be from the Primary to the Secondary (unless you have more than 2 nodes) otherwise you will create a loop. Remove all the settings from the XMLRPC Sync section on DC-FW2.
Leave the pfSync section though as state sync needs to be both directions.You should not have any outbound NAT rules for the SYNC subnet, that should never connect to anything but the other node.
Most importantly though is that when using the XG-7100 in an HA pair the failover interfaces should not be on the Eth ports. That is because you will not get full failover function using those.
In the event of the port losing link, a bad port or a bad cable or unintentional disconnect for example, it will not demote itself. The results in a split Master/Backup that will interrupt traffic.
It will still failover correctly if the full device fails or is upgraded though.To avoid that you should use the ix ports for WAN and LAN or add an expansion card with additional discrete interfaces and use those.
Steve
-
@stephenw10 Hi, we will use port ix0 as OPT1/SYNC port , that should work right?
Also at this moment the FW1 is connect to SWITCH1 and FW2 is connected to SWITCH2, but there is no link between the switches, apparently that needs to be done as well, so we'll do that too.
Also now FW1 is connected to WAN ip 1 and FW2 is connected to WAN ip 2 like:
But I guess I need to add a switch for this as well, for the 3rd WAN IP ? like:
??
-
ix0 will work for the SYNC interface, yes, but since it's doesn't use CARP SYNC can be on one of the Eth ports. You just need to configured the internal switch correctly.
Using ix0 as either WAN or LAN is a much better use if you don't have an expansion card.
You need to have a layer connection between the nodes on all interfaces that have CARP failover, yes. So, yes, you need a switch on the WAN side.
See: https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html
Steve
-
@stephenw10 so how do I configure the ETH8 then, so it is configured correctly for CARP SYNC ??
if I leave ETH1 for WAN and eth2 for LAN .
-
You have to add ports 9 and 10 as tagged members of VLAN 8 in the switch config as I said.
Currently you have VLAN 8 only using port 8 so nothing is ever passed to to the internal LAGG.
Make sure you can ping between the nodes on the SYNC interface IPs. It's not required for sync but adding rules to allow it makes troubleshooting much easier.
Steve
-
@stephenw10 said in CARP/HA not working:
ou have to add ports 9 and 10 as tagged members of VLAN 8 in the switch config as I said.
Currently you have VLAN 8 only using port 8 so nothing is ever passed to to the internal LAGG.
Make sure you can ping between the nodes on the SYNC interface IPs. It's not required for sync but adding rules to allow it makes troubleshooting much easier.
SteveSo like:
So, if I add 9 and 10 as tagged to vlan 8, even though 9 and 10 are not physical ports, it might start syncing to the other firewall??
(I have no access to the other FW at the moment as I fucked it up a little bit and I need to go to the datacenter ...)
-
Yes, that is the correct switch setup. You should be able to use port 8 for sync with that on both firewalls.
Steve
