Thanks for the insight.

The other traffic on that physical interface will be negletible (only management data), so we will go for redundancy with VLAN.

OK, so I didn’t manage to work out what was specifically causing the problem. I was using a relatively old version of pfsense (2.3.3). I downloaded the latest version and redid the setup from scratch, and it just worked!

@Yathus said in Additionnal Subnet /29 over a PPPOE Connection:

I tried with /32 one by one, not working too.

Finally i re-add all IP from my block, one by one, /32 over Locahost interface and now i can ping !

Over WAN interface it's not working...

I captured on all interfaces (not only the one connected to ISP) and both firewalls. Only my firewalls advertised VHIDs. Could not spot any foreign MACs...
Moreover beside VHID=3, also 5 caused the same issue.

If the advskew is 254 it is almost certainly in maintenance mode. The unit will not fail over unless it loses an interface on link down. It will not fail over on "OpenVPN service, VM or WAN failure." I am not sure what that means exactly.

@Pavel88 You know that this Screenshot is about OPNsense, not pfSense and that you're probably in the wrong forums?

Besides that, deleting any automatically created NAT rules and replacing them with "any" isn't recommended on either platform. Why did you remove them? It has a reason, why we differentiate localhost 127.0.0.1 and the LAN network when doing outbound NAT as @viragomann is absolutely right above. Without the right outbound NAT for 127.0.0.1 (-> has to be WAN address) there will be no internet on the standby node as it can't translate to your WAN VIP without being master.

https://docs.netgate.com/pfsense/en/latest/book/highavailability/example-redundant-configuration.html#setup-sync-interface

Closing this. Thanks for pointing me into the direction of testing the Ping on the CARP VIP. That ended up being the issue. Turns out somehow ISP took back one of our 3 IPs, we got them to put it back on our account and now we are back to normal. Can ping off that CARP VIP as well as port forwarding works now using the CARP VIP as Destination Address.

Thanks again @Derelict

Thanks @JeGr . I've now installed Filer and I can definitely see the use in it for restoring/syncing my script files. I can see that I can also probably use it for /etc/pfSense-devd.conf. But that brings the next problem of what happens when the Netgate team updates this file? The "latest" and correct version would get overwritten by my file in Filer. Out of curiosity I've checked the file on GitHub and it was indeed updated 2 months ago and those changes are in the file on my routers. So that means it will definitely change with an upcoming upgrade.

Is there no other/better way to force the maintenance mode or execute the devd actions without modifying a system file?

Thanks for that clarification @jimp. It helps.

@Derelict said in CARP failover events triggered for no obvious:

those
Hallo,

thanks for your reply.

I have some layer 2 errors on the switch (spanning-tree). I will try to fix the errors and provide feedback as soon as possible, but I only have "downtime" at Friday to test my configs.

Thanks for your help.

@awebster that was exactly what i tought too!!!

The failover itself works fine by entering to maintenance mode but the VPN tunnels don't want to bring up. They should and it works when tunnels are terminated with other vendors. This situation is only with AWS cloud. Moreover, the pfsense should initiate the connection. The AWS never brings the VPN tunnels up. In case when I use the policy based VPN (the traffic initiated behind the firewall) it works fine. Moreover, the same setup as I have now such as VTI interfaces, routed-based VPNs were configured on VyOS which switchover the tunnels automatically in case of failover.