Lots of possibilities. I would simplify the configuration down to the basics and see if you can get it working.
Key suspects for me would be network structure issue (have you accidentally introduced a loop and STP/RTSP is kicking in and disabling ports, causing weird routing? do you have a bad cable or ports that are auto-negotiating at the wrong speeds? etc). A CARP or Virtual IP configuration step you missed (wrong netmask on a virtual IP? left CARP in temporary maintenance mode, etc...)?
Check out the pfSense system logs, check that node is MASTER on both WAN and LAN (if they split then of course you have routing issues) and the other is BACKUP on both, check your switch port status and logs to see if it gives you any hints...
CARP/VRRP/etc. are using not only virtual IPs but also virtual MACs to make failover a smooth experience without clients or network equipment having to learn a new MAC address of a failover server like with only IP based configurations (early linux HA cluster for example).
The VHID setting is influencing which MAC is handed out for that CARP style VIP. All of them are (IMHO) using the failover MAC space of
so with changing the VHID you are also configuring the last "XX" segment of said MAC address. That's why it has to be unique on that network segment (L2) and you also have to watch out for other cluster/HA-grade setups, that are using VRRP or HSRP style VIP/MAC combinations. But if your pfSense cluster is the only cluster in that network segment, VHID 1 is commonly fine on all interfaces. We're using VHID 4 and 6 (for IP4 / IP6 VIPs on the same VLAN) over multiple VLANs just fine :)
No, only the WAN IPs of the two pfSense boxes have to be within the same subnet. A /29 is just the minimal network size for a common CARP setup with the two WAN IPs and the WAN VIP within a subnet. But it doesn't matter if the subnet is larger.
@arnold-assistant said in HAProxy Frontend ACL Limitation:
Perhaps try not using the 'var', i think now that it did not 'set' it yet when the advanced config acl is using it.. http-request rules are processed in the order they appear in the config.. so to avoid that change the acl like this:
Does the log on the master (the FW which is switching to backup status) show a "reloading filter" message just prior to the CARP state change? Seems to be a known issue which is causing CARP instability (for us, on physical hardware, but apparently the issue is more common on VMs). Will hopefully be fixed in 2.4.5-p1. Some discussion and possible temporary mitigation discussed here:
Trying something really stupid, I seem to have solved it. This seems highly possibly to be a bug...
pi_dev_server2 is the pre-existing server backend, just renamed, and it uses the hostname as mentioned in my first post, which gets correctly resolved to 10.0.0.235 by HAProxy and uses port 80. There's no obvious reason why this doesn't work, but it doesn't, I've disabled it here for the test. But if you look at the original post, it fails with L4OUT.
Now all of the servers are working, as a result of that one change (?) or did something else suddenly start working? I'm not sure, but having pi_dev_server using an IP instead of a hostname seemed to make all of the others work properly, despite the fact that they are still using hostnames... Very very curious...
@Izaac A much delayed update: reducing the VM instances to single CPU/core did indeed resolve the problem. The hardware gateways, which are not so easily so handled -- heh -- still have issues.
So. If you can "get by" with a single core, do that until a fix can roll.
As you both referenced info about IPv6 bogons I checked the table with pfctl -t bogonsv6 -T show and it was indeed quite large on the firewalls, being populated by the contents of /etc/bogonsv6. Since I have almost 80 interfaces (VLANs) defined on each firewall, I did not want to have to go to each one and uncheck Block bogon networks, so I did the lazy thing instead and:
on both firewalls and then applied an arbitrary config change to trigger a reload of the firewall rules.
From that moment on both firewalls performed much better and the lock ups and CARP ping-pongs disappeared. I've got the bogon updates set to Monthly, so I'll need to re-empty /etc/bogonsv6 again in a few days, but doing this once a month as a workaround is fine for me until I can upgrade to a release where the locks up are fixed.
We are using mutliple VLANS and the access of the Firewall was only allowed via their management VLAN. As soon as i created a rule to allow access via the IP of the Interface of the VLAN I'm connected to it worked fine.
Created Maintenance Interface class 3 address dhcp enabled on master (got sync with slave node)
Backup Full Configuration of Slave Node
unplug all interfaces (LAN, Wan, Sync)
Restored Slave config to Master node using maintenance interface
Changed all interface IP addresses of all wan, lan, vlan (Previous master node's Addresses)
Changed all virtual IP's Skew from 100 to 0
Changed all DHCP enabled Failover peer IP addresses
Enter persistent carp maintenance mode
Plugged-in lan (Lagg) interface to check
note: It worked and all carp interface status changed from INIT to Backup.
Plugged-in all cabled wan, sync
Master node's H.A enabled and cinfigured
Note: sync wasn't working not with admin account may be because i have changed sync password (for sync account). So changed Sync account password on both master and slave.
rebooted Master node and it worked.
Tested. All is Good now.
I'm still not sure what went wrong while upgrading Master node whereas slave node worked perfectly after upgrade. Clean installation and restoring previous saved configuration has also failed.
Thank you very much pfSense and netgate team for making such a wonderful firewall and keeping it open source.
@viragomann The outbound nat is the same on both as it NAT's to the VIP, and alias includes all the subnets of my LAN. Even with only allowing a single ping the secondary box is able to eventually do updates and what not so issue resolved for now. Thank you
Yes, I think that makes the most sense. Change outbound NAT address from interface to CARP IP, then replace all primary IPs and create old primary as CARP VIP; apply all at once. Outage should be brief to none, then firewall B can be configured as backup.
We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.
Subscribe to our Newsletter
Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.