I did try just bringing the interfaces up/down with ifconfig, but this didn't seem to work correctly… I also tried bringing down the physical interface rather than the ppp interface, but that just caused pppoe to stall and never reconnect. There's options in the webui to connect and disconnect a ppp interface, is there some way to trigger this from the cli?

Sorted… Had to set the VIPs interface to Localhost for whatever reason.

There are several problems with that: HA nodes with CARP must have identical interface setups. You can't have three different ISPs across two nodes and have it work properly. Failover signaling happens via CARP VIPs not the sync interface and those VIPs decide to fail over based on multicast heartbeats on each segment with a CARP VIP (e.g. LAN) Using HA for "Multi-WAN" is not viable. There is no way to signal node failover based on a WAN failure. For proper HA, all nodes must be connected to all the same ISPs, though that isn't always possible, without that you can't have a setup that will cover both HA and WAN failover.

I have migrated to IKEv2 because of the strongswan's L2TP implementation does not work for the clients behind their firewall. IKEv2 works with CARP without any problem! Best regards yarick123

In fact the behaviour is totally erratic, sometimes I can get full MASTER on master node and BACKUP on slave nodes, but then some of my servers are unable to reach the network… Other time i delete a Virtual IP and recreate it and i can reach my server again but the interface is MASTER on both nodes... My last option is to change vSwitchs options as said in troubleshooting (https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting) but I can't do it right away.

Here is the ifconfig output with CARP re-configured. We'll see how long it lasts before the LAN interface hangs… sk0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500         options=8000b <rxcsum,txcsum,vlan_mtu,linkstate>ether 00:90:7f:3e:38:29         inet6 fe80::290:7fff:fe3e:3829%sk0 prefixlen 64 scopeid 0x1         inet a.b.c.243 netmask 0xffffff00 broadcast a.b.c.255         inet a.b.c.241 netmask 0xffffff00 broadcast a.b.c.255 vhid 2         inet a.b.c.242 netmask 0xffffff00 broadcast a.b.c.255 vhid 3         inet a.b.c.240 netmask 0xffffff00 broadcast a.b.c.255 vhid 1         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)         status: active         carp: MASTER vhid 2 advbase 1 advskew 0         carp: MASTER vhid 3 advbase 1 advskew 0         carp: MASTER vhid 1 advbase 1 advskew 0 sk1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500         options=80009 <rxcsum,vlan_mtu,linkstate>ether 00:90:7f:3e:38:28         inet6 fe80::290:7fff:fe3e:3828%sk1 prefixlen 64 scopeid 0x2         inet e.f.g.4 netmask 0xffffff00 broadcast e.f.g.255         inet6 2001:5a8:4:70a0::4 prefixlen 64         inet e.f.g.1 netmask 0xffffff00 broadcast e.f.g.255 vhid 10         inet6 2001:5a8:4:70a0::1 prefixlen 64         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)         status: active         carp: MASTER vhid 10 advbase 1 advskew 0 sk2: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500         options=8000b <rxcsum,txcsum,vlan_mtu,linkstate>ether 00:90:7f:3e:38:27         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none)         status: no carrier sk3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500         options=8000b <rxcsum,txcsum,vlan_mtu,linkstate>ether 00:90:7f:3e:38:26         inet6 fe80::290:7fff:fe3e:3826%sk3 prefixlen 64 scopeid 0x4         inet 172.16.4.1 netmask 0xffffff00 broadcast 172.16.4.255         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (none)         status: no carrier pflog0: flags=100 <promisc>metric 0 mtu 33172 pfsync0: flags=0<> metric 0 mtu 1500         syncpeer: 224.0.0.240 maxupd: 128 defer: on         syncok: 1 enc0: flags=0<> metric 0 mtu 1536         nd6 options=21 <performnud,auto_linklocal>lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384         options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet 127.0.0.1 netmask 0xff000000         inet6 ::1 prefixlen 128         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8         nd6 options=21 <performnud,auto_linklocal>sk1_vlan192: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500         ether 00:90:7f:3e:38:28         inet6 fe80::290:7fff:fe3e:3828%sk1_vlan192 prefixlen 64 scopeid 0x9         inet 192.168.42.4 netmask 0xffffff00 broadcast 192.168.42.255         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)         status: active         vlan: 192 vlanpcp: 0 parent interface: sk1 gif0: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1280         tunnel inet a.b.c.240 --> 208.201.234.221         inet6 2001:5a8:0:1::e15 --> 2001:5a8:0:1::e14 prefixlen 128         inet6 fe80::290:7fff:fe3e:3829%gif0 prefixlen 64 scopeid 0xa         nd6 options=21 <performnud,auto_linklocal>bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500         ether 02:08:2c:92:bd:00         nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15         maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0         member: ovpns2 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 13 priority 128 path cost 2000000         member: sk1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 55 ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500         options=80000 <linkstate>inet6 fe80::290:7fff:fe3e:3829%ovpns1 prefixlen 64 scopeid 0xc         inet i.j.k.17 --> i.j.k.18 netmask 0xfffffff0         inet6 2001:5a8:4:70a0::43:1 prefixlen 64         nd6 options=21 <performnud,auto_linklocal>Opened by PID 34443 ovpns2: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500         options=80000 <linkstate>ether 00:bd:2f:e2:00:02         inet6 fe80::2bd:2fff:fee2:2%ovpns2 prefixlen 64 scopeid 0xd         nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect         status: active         Opened by PID 20343</performnud,auto_linklocal></linkstate></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></up,pointopoint,running,multicast></full-duplex></performnud,auto_linklocal></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></performnud,auto_linklocal></promisc></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,linkstate></up,broadcast,running,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,linkstate></broadcast,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast>

No. There is no way to change that.

I just noticed this and put a fix in for it yesterday: https://redmine.pfsense.org/issues/4903

Select Network and type in the alias in the address field below.

That's what the plugin system is for – if you want to take advantage of it, make a small/simple package that does nothing but run your code. I highly doubt we'll be adding multiple ways of running custom code in the same area, so either keep patching it or take advantage of the plugin system made for that purpose.

@viragomann: @mitch2k: What I'm not sure about, am I right that I need 2 extra public IP now? so 3 in total for the WAN interface; 1 for WAN pfsense1 (which is allready there), 1 for WAN pfsense2 and 1 WAN IP that fails over on the WAN interfaces (which would be the VPN IP). Or is there a way to do this without the need to buy extra public IP's? As you say. After the CARP setup is done, you can add further IP Alias to master, which are also shared. Services like VPN have to listen on CARP IP or IP Alias. There are thread in this forum where guys wrote, CARP also works with IPs in another subnet (private IP) assigned to WAN interfaces, but it have some disadvantages. https://forum.pfsense.org/index.php?topic=87546.msg507885#msg507885 Great, thanks for the info!

Don't pay attention to the VDS Config part?  These are the important bits: Enable promiscuous mode on the vSwitch Enable "MAC Address changes" Enable "Forged transmits"

@jimp: If a gateway goes down, it is logged in the gateways log and you'll see evidence in the main system log as well. To stop the states from being cleared for that, uncheck the box from both sides, but that may not be the case here. Worth unchecking on both to be certain though. Also check the gateway status on both, make sure they all show up. If a gateway is marked down and stays down it could do that as well, though it wouldn't necessarily always show a transition since it's down and staying down. Just to confirm, there were no such logged gateway failures (only thing in the gateway log is about how apinger has no targets and is exiting), and gateways both show up (and no network events…looking at upstream traffic graphs, this must have happened almost right at 2 AM last night, but nothing corresponds to 2 AM in the logs of either firewall or the upstream switches). So, I'm left to wonder: Could the state resetting even get triggered if a gateway isn't being monitored? What is the mechanism here? Is the mere fact that the options were different on the two servers enough to cause this problem, even if no gateways went down or were monitored to begin with? I would hope not, but my confidence is a bit lower at this point. Is this possibly a memory corruption issue or other hardware related problem? At this point I'm left with the unfortunate reality of advising that they are better off with state sync disabled, and the consequence of reset TCP sessions in the rare event of a failure, given that pfsync has caused several problems for them since the 2.2 release.

VRRP is a common method for an device based fail over scenario and if you will have luck only over VRRP it would perhaps working. The CARP itself is VRRP, but has anyone tried this before? Like  jimp was explaining is something alike VRRP, but not so common and won´t work with other devices. And ARPbalance over CARP is at this days a OpenBSD only thing such it will not running completely on other systems. There fore you will be able to balance the entire load over more then one device actual at the same time, with an automatic roll over effect.

So by having the aliases as URL's on a webserver, anyone could download your open ports and what ever if they wanted to? If they had the URL in question?

Had the same issue with the setup. Turned out a fat fingered one of the CARP vips. Make sure they are the same on both pfsense boxes :)

You can do it in 2.2.2 I had it in production for a bit, but you can't do failover properly- apinger sources from the bogus IP. I had to mark it up and manually fail over. Ended up getting more IPs and putting in a feature request to be able to point apinger to the CARP IP.

@Derelict: What good is a /29 if you can't use the addresses? Guessing it's not really a /29, it's a /30 from the ISP that he made into a /29.