That's what the plugin system is for – if you want to take advantage of it, make a small/simple package that does nothing but run your code. I highly doubt we'll be adding multiple ways of running custom code in the same area, so either keep patching it or take advantage of the plugin system made for that purpose.

@viragomann:

@mitch2k:

What I'm not sure about, am I right that I need 2 extra public IP now? so 3 in total for the WAN interface; 1 for WAN pfsense1 (which is allready there), 1 for WAN pfsense2 and 1 WAN IP that fails over on the WAN interfaces (which would be the VPN IP). Or is there a way to do this without the need to buy extra public IP's?

As you say.

After the CARP setup is done, you can add further IP Alias to master, which are also shared.
Services like VPN have to listen on CARP IP or IP Alias.

There are thread in this forum where guys wrote, CARP also works with IPs in another subnet (private IP) assigned to WAN interfaces, but it have some disadvantages.
https://forum.pfsense.org/index.php?topic=87546.msg507885#msg507885

Great, thanks for the info!

Don't pay attention to the VDS Config part?  These are the important bits:

Enable promiscuous mode on the vSwitch

Enable "MAC Address changes"

Enable "Forged transmits"

@jimp:

If a gateway goes down, it is logged in the gateways log and you'll see evidence in the main system log as well.

To stop the states from being cleared for that, uncheck the box from both sides, but that may not be the case here.

Worth unchecking on both to be certain though. Also check the gateway status on both, make sure they all show up. If a gateway is marked down and stays down it could do that as well, though it wouldn't necessarily always show a transition since it's down and staying down.

Just to confirm, there were no such logged gateway failures (only thing in the gateway log is about how apinger has no targets and is exiting), and gateways both show up (and no network events…looking at upstream traffic graphs, this must have happened almost right at 2 AM last night, but nothing corresponds to 2 AM in the logs of either firewall or the upstream switches).

So, I'm left to wonder:

Could the state resetting even get triggered if a gateway isn't being monitored? What is the mechanism here?

Is the mere fact that the options were different on the two servers enough to cause this problem, even if no gateways went down or were monitored to begin with? I would hope not, but my confidence is a bit lower at this point.

Is this possibly a memory corruption issue or other hardware related problem?

At this point I'm left with the unfortunate reality of advising that they are better off with state sync disabled, and the consequence of reset TCP sessions in the rare event of a failure, given that pfsync has caused several problems for them since the 2.2 release.

VRRP is a common method for an device based fail over scenario and if you will have luck only over VRRP
it would perhaps working.

The CARP itself is VRRP, but has anyone tried this before?

Like  jimp was explaining is something alike VRRP, but not so common and won´t work
with other devices.

And ARPbalance over CARP is at this days a OpenBSD only thing such it will not running completely
on other systems. There fore you will be able to balance the entire load over more then one device
actual at the same time, with an automatic roll over effect.

So by having the aliases as URL's on a webserver, anyone could download your open ports and what ever if they wanted to? If they had the URL in question?

Had the same issue with the setup. Turned out a fat fingered one of the CARP vips. Make sure they are the same on both pfsense boxes :)

You can do it in 2.2.2 I had it in production for a bit, but you can't do failover properly- apinger sources from the bogus IP. I had to mark it up and manually fail over. Ended up getting more IPs and putting in a feature request to be able to point apinger to the CARP IP.

@Derelict:

What good is a /29 if you can't use the addresses?

Guessing it's not really a /29, it's a /30 from the ISP that he made into a /29.

I've put the scripts here

https://github.com/deasmi/pf_interface_pin
At some point I might try to make this into a package with a UI if anyone is interested.

Please note you need to re-install after an upgrade.

Yeah that should apply to all vswitches on the host. Is it all the CARP VIPs on one interface, or just one on that interface that has others that work fine?

Yes, this is normal if the firewall is the VPN server and you sync all settings from master to backup. This way the backup box has equal VPN setup and the same tunnel network exists on both, master and backup. So if the backup replies to a request from a VPN IP it sends the packet to its own VPN interface which is down though.

To resolve this, you can use outbound NAT. Add an outbound NAT rule for the LAN interface, which translates IPs of packets coming from VPN tunnel network and have one of the LAN addresses of the boxes as destination (you may use an alias here or add a second rule for the other box) to its LAN address.
So if you connect to the backup box over VPN, the packets get the LAN address of master and replies from backup box are sent back to the masters LAN IP and the master will route the packets to the VPN client.

I leave it enabled at defaults. It shouldn't need DPD to fail to the second node- the secondary should take over the existing IPSec connections.

This issue was fixed in 2.2.3.

One more thing I've noticed - the behaviour seems to be the same when adding new CARP VIPs. When you click save to add a VIP, it is immediately synced and applied to the secondary node, and only gets applied on the primary after clicking 'apply'. It's not so much of a problem in that case of course, because it's a new VIP, and doesn't matter if it's MASTER on the secondary initially.

Ok,…Interface Order was different on the two nodes, so the Virtual IP Synchronization was incorrect, and so it didn't work at all,...same order on both nodes, everything fine.