@stephenw10 I sent you a chat

@Globaltrader312 I have now also removed the firewall rules under NAT

SOLVED! on my test rig I tried a state-killing option that had NOT solved the problem on my live box, but on the test rig it worked. The setting is in System/Routing/Gateways, "State Killing on Gateway Failure". After changing that from the default to "Kill states using this gateway when it is down", subsequent failover events created a few arpresolve errors in the log, but within 1 second they stopped, after an entry in the log showing a state killing action:

/rc.filter_configure_sync: GW States: Killing states for dynamic down gateway: WAN_DHCP, XX.XX.XX.1

After that worked, I had to figure out why this solved the problem with my test rig but not my live box. Eventually I traced it to a setting in System/Advanced/Miscellaneous in the Gateway Monitoring Section, "Skip rules when gateway is down". In my live box, which has some traffic that needs to be routed only through a VPN, I had enabled the setting "Do not create rules when gateway is down" years ago to make sure, if the VPN was down, that pfSense wouldn't route the traffic through the non-VPN WAN. But as soon as I cleared that check box, my failover arpresolve problem went away. So apparently that setting interacts with the failover in a way that prevents the state-killing action from working properly.

Next job is to figure out a different way to kill VPN-bound traffic if the VPN is down... Googling that now.

@Sperber said in Dual OpenVPN-Setting, CARP & Failover (HA, MultiWAN):

(Vorkbaard hat das bereits beschrieben: https://vorkbaard.nl/openvpn-in-a-pfsense-carp-cluster/ )

Die Info ist aber relativ alt und nicht zutreffen. Wir haben da sehr verschiedene und komplexe Services laufen und keiner braucht irgendwelche seltsamen Settings mit "local <extIP>" o.ä. - das sollte heute überhaupt nicht mehr nötig sein. Macht im CARP Setup auch keinen Sinn, da die CARP VIPs alle auf dem Master laufen und man diese so nicht ansprechen kann. Split CARP mit Master/Backup auf dem selben Node ist in der FreeBSD Variante von CARP/pf nicht enthalten, das ist leider nur in OpenBSD enthalten.

Mich interessiert allerdings auch wie @viragomann wie man überhaupt auf der 2. pfSense im CARP die Annahme von OpenVPN erlauben will. Der Traffic kommt ja nicht bei ihr an, weil der via CARP IMMER zur primären läuft, nicht auf den sekundären Node. Und wenn man das forwarden sollte auf Node 2, würde der Node versuchen asymmetrisch zu antworten (oder es läuft alles wieder über Node1), was auch wieder nicht sehr schön ist.

Wie ist das also realisiert, dass die Clients sich auf Node2 connecten und das auch funktioniert, wenn Node2 mal aktiv wird und Node1 passiv weil vlt. gerade gewartet wird o.ä.?

Ansonsten wäre mir schleierhaft wie das im Redundanzfall wirklich sauber funktionieren sollte ohne dass manuell eingegriffen wird?

Cheers
\jens

@marl_scot
The networks on different interfaces must not overlapping.
And I don't know any router which is capable to route with that settings.
Maybe the ISP can give some recommendations.

Two IPs within the same subnet with the same gateway is not a real failover set up for my understanding.

If the ISP refuses to change one of the subnets your only one option might be to put a router between the ISP and pfSense and nat the traffic to a different subnet.

@luckyh_de said in Multi-WAN with Backup down:

So i have to prevent any Packet to the LTE-router AS Long as primary ist okay

Hi,

The failover mechanism does not allow this, you definitely need something that, which tells the firewall that the connections are alive
(minimum GW pinger ICMP traffic)

@luckman212 Has this been integrated into a subsequent release or is this patch still valid? I'm having the same issue on 23.05.1-RELEASE.

@yacud With failover and multiple tiers, it will use the Tier1 gateways until it meets the criteria of a failure (specified packet loss or latency).

Then it will route all traffic on the Tier2 gateway until Tier1 gateway is back within acceptable limits.

If you want to load balance you could set multiple gateways as Tier1 and it will split traffic between them, you can set a "weight" in the gateway options to have it balance the traffic unevenly (e.g. put 2x as much on WAN1 vs WAN2)

As far as I know, there is no way for it to know what the maximum throughput of your link is - just trying to split it evenly if you want load balancing.

The best way to do an HA deployment is it invest in the gear necessary to build it correctly. Bridging like that is generally incompatible with pfSense HA.

https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.html

It can depend on the switch/router on the other end of the cable. For instance with Comcast routers often when replacing a router in an office (inside the Comcast router) I've found it's fastest to power off or reboot the Comcast router so it learns the IP has a new MAC. If you have the second router on, and are just plugging in cables, I would wonder if restarting the second router (or just leaving it off and powering it on) would help.

But overall CARP set up properly works basically instantly so that would be preferred. https://docs.netgate.com/pfsense/en/latest/book/highavailability/index.html

Добрый

В закладки https://docs.netgate.com/pfsense/en/latest/ Раздел "Routing and Multi-WAN"
И не забывайте на пф в General добавить явно каждому WAN-у по ДНС. Это важно.

на PfSense недавно, в этой сфере разбираюсь не очень хорошо

Коллеги.
Давайте не начинать каждый 2-й пост с "нытья" (
Как на вокзале, чес. слово, "Деньги украли, не могу 3-й год до Воронежа доехать, спасите-помогите". Просто пишите ТЗ. Этого достаточно.

Got it all up and running thanks! Eddie

Thank you @jimp

That helps clarify quite a bit. There really isn't much available about the "Skip Rules" setting.

Based on the documentation, your answers are what I expected, but I wanted to be sure.

Thanks again,
Josh