@ccMatze Floating ip's in hetzner can be moved only via robot administration, or custom api by making calls. If you need carp then you need to order a /29 subnet, However I don't see any option for /29 (or any other subnet) for cloud hosts. You need to rethink your approach. Hetzner cloud vm's are already redundant. So in case of failure, your pfsense instance will always be available. If you really need such redundancy then you should consider using dedicated servers which of course creates its own set of issues and concerns.

@vinns said in FRR BGP over IPsec , when HA happens (slave-> master, master ->slave): right. thats the same result we got too. so nothing new on that. and i agree on the fact that, it could very well be that the support of HA sync does not include the FRR, afterall that is an additional package. i mean its not the end of the world to copy 30-40 lines from the xml and add them to the second node if that is the case so be it. :) many thanks for looking into this man , appreciate your help :) :)

@hoba This seemed to work for one of our sets, so THANK YOU! However, for anyone else that might be in the same boat, our state table was colsossal and therefore this may be a treatment, rather that a cure. *Or may be indicitave of an issue on one of the local networks *I waited to hit submit until I found the issue - Camera system, wide open on separate VLAN in this case We had the luxury (!) to run this remotely, in non peak times, with alternative, remote access, on the local intefaces - rather than WAN. It took about 16 minutes for both on Xeon 3.2 physical, 8 vCpu, 8GB RAM 128GB fixed. In fact, if you have alternative remote access to the local network, I'd recommend the exact of above with the states, wait for the states to clear, reboot each, then reenable 'System > High Availability > Synchronize states I don't recommend doing this if you will have to travel several hours to complete on-site, and you don't have alternative remote access to the site. *just my 2¢

@SteveITS Thanks for replying. Hetzner got back to me and they can't route a subnet behind another subnet - only behind a single IP. So, I'll try setting this up a single CARP WAN IP and test. If not, 1:1 NAT would work as you suggested - but tbh, I'd prefer it without NAT.

@plokker We're looking to do the same thing. All our servers are in the same rack and connected via a second 10G NIC to a managed switch. What IPs did you use on the CARP WAN side? pfsense recommend a minimum /29 for this. Thanks.

@viragomann The network card configuration is correct, what surprised me is that only towards Virtual IP 192.168.3.1/24 I had the problem. If I create another one like 192.168.88.1/24 the problem doesn't exist, I solved it simply by running a reboot of pfsense. But I still don't understand why this happened. Thank you very much for helping

It turned out it's faulty switch... Replaced it and all well.. Sorry for the trouble and thanks for all suggestions !

bump

@alkaid So maybe your backend server is configured to use the secondary node as default gateway. The default gateway on your local devices behind the HA pair should be the CARP VIP of the subnet.

@terrorbyte704 This is, what I was trying to tell you.

@planedrop FWIW I restarted our backup router just now. The "wait" counter never reset because our a/v was interrupting the "up?" check due to the self-signed cert. (this is not going to happen to most people, but is expected behavior in this case, with Bitdefender) Turns out the web GUI was using a new "GUI default" cert that it created at the boot instead of the real cert I mentioned above. Not real sure of the path there. I thought when I posted above it had already been set to use the new cert, but I can't go back and look again, now.

@doni49 Sadly all screenshots are lost. If the browser doesn't show a certificate, either HAproxy does not deliver any, because it's not assigned correctly, or you are connected to the wrong host.

Thank you very much for your reply. I've managed to get it to work - thanks for your help. A couple of points: • I needed also to add a rule specifically to allow DNS traffic from the DNS Resolver in the firewall across the 4G WAN, otherwise DNS doesn't work (because it doesn't hit on the LAN rule) • In addition to changing the gateway manually (which is fine), I also need to tweak the DNS Resolver setting so that outbound requests go across the 4G WAN and not the normal WAN. Not sure if there's a way around that? If I enable both outgoing interfaces in DNS Resolver, then it seems to distribute DNS traffic even when the gateway doesn't need to failover.