Fixed by properly NAT-ing the requests:
pfsense active - IP x.x.x.2/24
pfsense standby - IP x.x.x.3/24
pfsense CARP - IP x.x.x.1/24

Create NAT Outbound:

Interface LAN_MGMT Source - my client LANs Destination - pfsense IP from above (subnet x.x.x.x/30) NAT Address - LAN_MGMT address

Works like charm.

Brilliant! Thank you very much!

Thanks for the tips! This resolved the issue!

We created a lab with two firewalls with very basic configs, and set up something similar to what I outlined above. I provided rules and config to @Derelict at the beginning of July and messaged him since to follow up but haven't heard back. So, I'm uploading them here.

Using these configs, the IP address in the test alias (from the outside/WAN interface of the firewall) can connect to the secondary firewall's LAN interface, even though the rules would seem to block it.

Additionally, any IP address on the LAN subnet can connect to the secondary firewall's WAN address, even though the rules would seem to deny it.

If you turn off state syncing, this stops being true.

I'm interested to see if anyone can find an issue with how the config is set up, or whether using "This firewall" in the rules just isn't enough to ever block access on the secondary.

fwa_rules.debug.txt
fwb_rules.debug.txt
config-fwa.localdomain-20190627095630.xml
config-fwb.localdomain-20190627100039.xml

@agterry said in HA Dual-WAN Issues with Packet Loss:

failover a VPN tunnel as well if possible

IPSEC or OpenVPN? And by failover that means from WAN1 to WAN2 or just from master node to slave node?

It is possible but you must use things like Spanning Tree to prevent loops. HA + Bridging is not a recommended configuration.

Much better is to have your ISP issue you a small WAN interface network (/29) and route the subnet to that.

Then you can put the public subnet on an inside interface, eliminate all bridging and NAT, and your network will just make money while you sip margaritas by the pool.

Solved. I use IKEv2 now.

@Derelict said in HA Sync problems since updating to 2.4.4:

explicit-exit-notify does not apply in that case because the OpenVPN instance does not exit on failover. That means the clients will have to time out and when they attempt to reconnect they'll get whatever system holds the CARP VIP at that time.

Ah thanks for adding that. So would only work on a VPN configured on a failover gateway group type of WAN?

Edit: Also we got the problem to appear less now by adding the "no monitoring at all" switch to all 15 VPN gateways. But:

that eliminates the possibility for a some grouping of VPN interfaces that blocks monitoring and status/availability checks on those gateways is irritating to the admin staff (as all VPNs are always "on" even if they won't work)

etc.
We can definetly track that down to something that changed since 2.4.4 as with 2.4.3 and ~6-7 VPNs active at the time (with their gateways configured active) that problem didn't happen at all.

So situation is:

with 2.4.4 - whysoever - VPN GWs assigned and configured are "tilting out" the standby node even without gateway monitoring, the standby node spits out ~300 lines of syslog with every config save on the master as every single VPN interface is set down, then up, then reconfigured, then detected with a "new IP" (even if that didn't change at all) etc. etc. all that starts after every simple config change

I understand the config sync to be non-trivial, but it strikes me odd, that changes to aliases or rules have to trigger that whole bunch of interface action just to reload and activate some simple ruleset or alias changes?

Thanks for any insight,
Jens

On a healthy system the primary would be showing skew 0, the secondary skew 100.

Check the system log for entries related to why it is changing the skew to 254.

Follow up!

Yeah i basically followed Derelicts info and moved to Manaual NAT rules, then changed the IP to my CARP VIP for my WAN.

Everything seems happier now.

Thanks for your help.

System log

162a0852-f100-49fe-81dd-58da8cc38e79-image.png

Silly me... I noticed that in the tier1 and 2 I had specified the interface address and not the carp ip. Changing it to the carp ip resolved the issue.

Glad you found it.

@Derelict
Very sad we will try to setup a vm hosted HA pfsense maybe just nobody tried it like we want.

Yeah. Because the only firewall that has a route back to the VPN clients is the primary that is hosting the VPN server.

https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshooting-vpn-connectivity-to-a-high-availability-secondary-node.html

That solution works.

ARGH! It's always the same, you think you try everything then you post on a forum, then you go back and read the doco and find what you've done wrong.

For my NAT rule i was setting it to the LAN CARP VIP, not the WAN CARP VIP. As shown in doco and videos. Apologises.

And as for my DNS not working, i had not set the LAN CARP VIP to listen for DNS queries.

Apologises folks. All done

Yep I'd like to see central management for my pfSense boxes too.
But self hosted, not somewhere in the Internet aka Cloud.
So you have all this fancy stuff with HA, MultiWAN and so on...unable to control anything because the central portal is down? Only marketing guys can think of this shit putting Firewall management in the Cloud. 😂

-Rico

Your clients need to be set to use the CARP VIP as their default gateway. This is usually done in the DHCP server settings.

You need to set outbound NAT so traffic sourced from clients leaving the nodes uses the CARP VIP.

Youtube: High Availability

Youtube: High Availability Part 2

https://docs.netgate.com/pfsense/en/latest/book/highavailability/index.html

@JeGr said in OpenVPN HA Sync failover:

Wouldn't that only happen if you manually shutdown the primary node? Just curious, as in a failure (e.g. uplink down) case, no one would get the exit notify as the line is down? But yeah in an update scenario where you shut down the primary for updating, it would definetly cut links quicker than a timeout.

Of course a hard failure will probably not send anything from the OpenVPN process. Those are actually pretty rare compared to setting maintenance mode. An interface down event would (unless the down interface was necessary to send the disconnect advisories)