After more test, the more balancer perf I can get are finally with the LRO offload check : it decrease my iperf with the firewall interface a lot (2-3Gb/s instead of 15-20Gb/s), but increase the iperf going throught the firewall, between A and B (2-3Gb/s instead or less than 500Mb/s).

I did all these test on the same ESX, so where are my speed ???

Hi Netblues,
Thanks for your fast respons and sorry for not answering sooner.
I figured out my troubles, after a while, and found that I needed to add 3 more vmnets (VNICS) on the Firewalls and to Configure those Firewalls, the Windows dhcp servers and the Client machine with the appropriate vmnets to them aswell in Workstation Pro. Because I broadcastet my dhcp renewal from the client into the same network and hadn't seperated the network with different networks, I simply got a messy DORA exchange on the firewalls, where they would send the discover, offer, request and acknowledge between FW1 and FW2, before sending it to the dhcp servers.

added to https://redmine.pfsense.org/issues/7132#note-7

Again I stress... not really a programmer, this kind of is a bit over my head. I will try to look into it with some spare time, but in the meanwhile I wonder if no one would also benefit from this and if there isn't someone who maybe could pick this up and wrap it quickly.

Is "Synchronize states" checked on both routers in System/HA Sync? Not having states synced would block existing connections but new connections should work.

Possibly something upstream isn't liking the IP changes? Did you look at https://docs.netgate.com/pfsense/en/latest/book/highavailability/high-availability-troubleshooting.html#other-switch-and-layer-2-issues

@johnpoz said in Can HAProxy pass OpenVPN:

Well for haproxy to pass it on it would have to meet your acls... So say some random guy hit your IP. Unless he was calling for your exact fqdn say host.domain.tld ha proxy would not pass it on to your server.

Or whatever other acls you setup on haproxy.

Thanks @johnpoz ... Do you know if HAProxy returns an error like a 401/403 or if it just drops/rejects the connection?

Hi @teamits hehe well actually I do because the SYNC network also has a few other clients behind it that require the VIP such as LAN.
Basically as I mentioned on the above posts, I chose an already existing network for SYNC that has two other clients beside the pfSense machines. This is a secured network and these are administration machines with restricted access and little traffic.
The documentation recommends a separate network, as I see it, for two factors:

network availability and load privacy and security (as passwords aren't really encrypted)

Since the chosen network complies with these requirements, it is a very restrict network with very low traffic, this network was used and hence the interface used for sync has a carp vip.
Anyway, all configurations: HA, Interfaces and DHCP server etc have the peer IP directly where it belongs, not the CARP VIP.
I expect this interface to work alike the other interfaces (LAN/DMZ7DATA) etc.

It does not matter what interface you are connected to when you issue that command. It demotes all CARP VIPs on that node to advskew 254.

If you run that (or use enter CARP Mantenance mode in Status > CARP - which does the same thing) and you still see MASTER VIPs on that node, your HA is not properly-configured in the first place.

Are all of your CARP VIPs MASTER/BACKUP before you try to put the master into maintenance mode?

Today I had an idea. I disabled the firewall with pfctl -d on the second device.
Access was possible again.

After syncing the config from the first to the second pfsense, I enabled the firewall again with pfctl -e. You might want to reboot your device at this point.

Now it works again.
I must have messed up something with the firewall rules, and it was applied to the second pfsense, and then I was locked out as well as my first firewall from the gui. I have no other explanation for my situation.

You can follow the guide from the docs (found that later): https://docs.netgate.com/pfsense/en/latest/book/config/what-to-do-when-locked-out-of-the-webgui.html#disable-the-firewall

also check, what @jgraham5481 said in Can't access Backup router after HA/CARP enabled:

Those carp addresses should be the same subnet mask as the network they live on, ie: should be /24 if the interfaces on the master and salve firewalls are /24/

Thanks @netblues

I worked out what my issue was, I made a rookie mistake when creating the VLANs on the secondary pfsense machine.

The VLAN subnet should have been 192.168.150.2, but I assigned 192.168.150.1 which meant .1 was duplicated twice on the network.

Once that was corrected, everything works fine now.

@jgraham5481
https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshooting-vpn-connectivity-to-a-high-availability-secondary-node.html

True, but I need to "prove" that they are in sync for a 3rd party audit. So I need to show the current status of the sync.

@benrichardson_insync So its is expected to have this behaviour. Carp interfaces must be on the same broadcast domain. The master sends regular advertisements to the backups.

See here for more details about the mechanism
https://www.netbsd.org/docs/guide/en/chap-carp.html

"Flush all states when a gateway goes down" has to be off, otherwise states are killed even when a gateway reaches the high watermark, that has been set to down.

what is the error message you received?
make sure noting broken in L2 level

It can depend on the switch/router on the other end of the cable. For instance with Comcast routers often when replacing a router in an office (inside the Comcast router) I've found it's fastest to power off or reboot the Comcast router so it learns the IP has a new MAC. If you have the second router on, and are just plugging in cables, I would wonder if restarting the second router (or just leaving it off and powering it on) would help.

But overall CARP set up properly works basically instantly so that would be preferred. https://docs.netgate.com/pfsense/en/latest/book/highavailability/index.html