@antoinef67 I registered an account just to say thank you sir, a life saver!

@gkuyuk Had it resolved after spending some time on it. Setting the MTU to 1420 and MSS to 1280 on the sync interface resolved the problem. The switch was set at mtu 9000 and with that setting and default mtu of 1500 things should have been working fine but appearently not. Wanted to put it here if anyone else have a similar problem.s

@ccMatze
Floating ip's in hetzner can be moved only via robot administration, or custom api by making calls.
If you need carp then you need to order a /29 subnet,
However I don't see any option for /29 (or any other subnet) for cloud hosts.

You need to rethink your approach. Hetzner cloud vm's are already redundant. So in case of failure, your pfsense instance will always be available.
If you really need such redundancy then you should consider using dedicated servers which of course creates its own set of issues and concerns.

@vinns said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave):

right. thats the same result we got too. so nothing new on that. and i agree on the fact that, it could very well be that the support of HA sync does not include the FRR, afterall that is an additional package. i mean its not the end of the world to copy 30-40 lines from the xml and add them to the second node if that is the case so be it. :) many thanks for looking into this man , appreciate your help :)

:) 👍

@hoba

This seemed to work for one of our sets, so THANK YOU!

However, for anyone else that might be in the same boat, our state table was colsossal and therefore this may be a treatment, rather that a cure. *Or may be indicitave of an issue on one of the local networks

*I waited to hit submit until I found the issue - Camera system, wide open on separate VLAN in this case

We had the luxury (!) to run this remotely, in non peak times, with alternative, remote access, on the local intefaces - rather than WAN.

It took about 16 minutes for both on Xeon 3.2 physical, 8 vCpu, 8GB RAM 128GB fixed.

In fact, if you have alternative remote access to the local network, I'd recommend the exact of above with the states, wait for the states to clear, reboot each, then reenable 'System > High Availability > Synchronize states

I don't recommend doing this if you will have to travel several hours to complete on-site, and you don't have alternative remote access to the site. *just my 2¢

@SteveITS Thanks for replying. Hetzner got back to me and they can't route a subnet behind another subnet - only behind a single IP. So, I'll try setting this up a single CARP WAN IP and test. If not, 1:1 NAT would work as you suggested - but tbh, I'd prefer it without NAT.

@plokker We're looking to do the same thing. All our servers are in the same rack and connected via a second 10G NIC to a managed switch.

What IPs did you use on the CARP WAN side? pfsense recommend a minimum /29 for this.

Thanks.

@viragomann
The network card configuration is correct, what surprised me is that only towards Virtual IP 192.168.3.1/24 I had the problem. If I create another one like 192.168.88.1/24 the problem doesn't exist, I solved it simply by running a reboot of pfsense. But I still don't understand why this happened. Thank you very much for helping

It turned out it's faulty switch...
Replaced it and all well..
Sorry for the trouble and thanks for all suggestions !

bump

@alkaid
So maybe your backend server is configured to use the secondary node as default gateway.

The default gateway on your local devices behind the HA pair should be the CARP VIP of the subnet.

@terrorbyte704
This is, what I was trying to tell you.

@planedrop FWIW I restarted our backup router just now. The "wait" counter never reset because our a/v was interrupting the "up?" check due to the self-signed cert. (this is not going to happen to most people, but is expected behavior in this case, with Bitdefender)

Turns out the web GUI was using a new "GUI default" cert that it created at the boot instead of the real cert I mentioned above. Not real sure of the path there. I thought when I posted above it had already been set to use the new cert, but I can't go back and look again, now.