It works for me in Chrome here on 2.3.3. Make sure you clear the browser cache (ctrl+F5, or shift+reload) between tests. I tried switching between each of the possible VIP modes and the correct fields were enabled each time.

Solved! On the controller's firewall i had to disable "Prohibit IP Spoofing".

@Elegant You already have Dial on Demand. What I have read here http://sirlagz.net/2014/12/22/pfsense-carp-and-pppoe/ and here http://theartofservice.com/pfsense-carp-and-pppoe.html is that you also need to disable the Gateway monitoring. Then PPPoE will only be created on 2nd Firewall when 1st one goes down. I have not done this myself though yet.

@KOM: A port-forward / 1:1 NAT must be mapped to something.  In your case you would map it to the local IP of the CentOS box.  That's how it works.  And as I mentioned earlier, if you're uncomfortable forwarding traffic to a host on your LAN, then create a DMZ via new interface or VLAN and move your CentOS box there. Ok, so I will run a physical ethernet cable between my OPT1 interface and a physical interface on my ESXi Server. I'll assign that interface to CENTOS within the ESXI Controller. What will my configuration look like in PFsense?

Thanks viragomann  ;)

Does WAN stay MASTER/BACKUP or is it always MASTER/MASTER? There's not a lot to happen there. Setting CARP maintenance mode hard sets the advskew to 254 and if the primary receives a more recent advert from the backup it will go into a BACKUP state on that VIP, likewise if the backup does not receive a more recent advert from the primary it will go MASTER. Running a mismatched pair can be challenging and is not recommended. But this should work. State sync is another matter. How are the WAN ports and the DCGW physically and virtually connected? Do both WAN ports see the CARP adv traffic like they should? (Packet Capture on CARP.)

I too am receiving this error that I believe is related to PHP-FPM Sometimes if you restart php-fpmd service in the secondary node the SYNC completes. I sometimes need to reboot the FW in order to get the config syncronized.

My issue of the unexplained ignoring of the MAC address bypass list was solved by upgrading to a development snapshot (2.3.4 something) but in a failover to the backup router, which still had 2.3.2-Release-p1, the problem still existed, so we upgraded that to whatever snapshot was available that day. As soon as the 2.3.3 maintenance release is out we'll try that and see whether the problem comes back or not. The problem of CARP LAN addresses not responding to workstation ARP requests has not been solved. (Summary: we change static IP address on interface to something else, create CARP virtual IP on LAN with the original IP address, at which point workstations on the network cannot see the CARP IP address. Using a packet sniffer we observe that the router's LAN interface is not responding to ARP requests by workstations for the gateway IP address. Rebooting the router and every other device on the network, including switches and workstations, did not solve the problem.)  We will try that again after the maintenance release as well. thanks

Does it see the advertisements from the primary before you add the VIP? Does the primary see those advertisements from the secondary? It is not generally correct to add a CARP VIP to the secondary. You add it to the primary and it XMLRPC syncs over to the secondary with the proper advbase/advskew. If you add it to the secondary manually and there is not a 1/0 skew VIP already on the network, of course it will assume MASTER. Tested what you reported on a fairly-current 2.4-BETA VM pair: Added VIP 172.25.236.65 on Secondary only: xn0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 12:77:26:96:5d:a3 inet6 fe80::1077:26ff:fe96:5da3%xn0 prefixlen 64 scopeid 0x5 inet6 2001:470:f00e:7e01::3 prefixlen 64 inet6 2001:470:f00e:7e01::1 prefixlen 64 vhid 239 inet 172.25.236.3 netmask 0xffffff00 broadcast 172.25.236.255 inet 172.25.236.1 netmask 0xffffff00 broadcast 172.25.236.255 vhid 236 inet 172.25.236.65 netmask 0xffffff00 broadcast 172.25.236.255 vhid 241 nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual status: active carp: BACKUP vhid 236 advbase 1 advskew 100 carp: BACKUP vhid 239 advbase 1 advskew 100 carp: MASTER vhid 241 advbase 1 advskew 100 Deleted same: xn0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 options=3 <rxcsum,txcsum>ether 12:77:26:96:5d:a3 inet6 fe80::1077:26ff:fe96:5da3%xn0 prefixlen 64 scopeid 0x5 inet6 2001:470:f00e:7e01::3 prefixlen 64 inet6 2001:470:f00e:7e01::1 prefixlen 64 vhid 239 inet 172.25.236.3 netmask 0xffffff00 broadcast 172.25.236.255 inet 172.25.236.1 netmask 0xffffff00 broadcast 172.25.236.255 vhid 236 nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual status: active carp: BACKUP vhid 236 advbase 1 advskew 100 carp: BACKUP vhid 239 advbase 1 advskew 100</performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast>

@Smoothrunnings: a. If what you are saying the WAN on both firewalls have their own public IPs, then how does the secondary assume the role of the primary when fail-over occurs? Thanks The secondary gains control of the Virtual (CARP) IPs, the LAN side and the Public side. If this isn't clear, please review the CARP man page, the HA documentation, etc. I feel like this discussion is going in circles.

Found my issue… I had to put the VIP ip as the dns server in my dhcp server option.

@wickeren: I'm having a hard time making my virtual ip's available to be able to used by a service like openvpn or haproxy. I have just a single PPPoE WAN with a /29 subnet. On the interface itself i got a .97/32 assigned. In the past the virtual ip's (.98 - .102) were added as PROXY ARP, working perfectly for NAT. However, they are not listed as an interface option in e.g openvpn or haproxy. Switched to IP alias, same story. Then i found some hints suggesting for PPPoE the additional IP's should be assigned to the localhost interface instead of WAN, but that didn't help either. https://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses has nice info, but I couldn't resolve the issue with it. It might have to do something with PPPoE WAN. How can I make a service running on a additional IP different from the default assigned WAN IP? I do it this way: WAN = pppoe on say igb0 WANNIC = igb0 WAN will get itself an address via DHCP as now Set the IP for WANNIC and your PPPoE modem's "internal" address, for example a Draytek 120/130 will default to something like 192.168.2.1/24 so put 192.168.2.11/24 on WANNIC Put an outbound NAT on WANNIC to the modem, assuming the modem has no default gateway. You should be able to access it's web interface from LAN now. Add the IP aliases or CARP addresses to WANNIC for .98-.102 The extra IPs will appear at the end of the lists for things like IPSEC, OpenVPN etc Inbound rules go on WAN and not WANNIC Outbound NAT rules happen on WAN and not WANNIC apart from teh one I mentioned if there is a web interface on the modem WANIC should not have any firewall rules apart from a reject/block rule with logging You can put the IP aliases on localhost but creating the extra WANNIC interface allows access to the modem and makes life a lot easier when there is more than one WAN to deal with .

Hello David, many thanks for answering! It's still a little bit abstract for me, so I think I will 1st configure the existing firewall to also have LTE access fallback and then look into the failover. I will probably follow up with some more specific questions. Best

Good morning forum, I'm just suffering the same question as Andrew M. Robinson. The schema he's proposing seems to be the best one when HA is required both at filtering level (pfSense) and routing level (switches behind pfSense, L3 maybe?). After looking at this thread, it seems that it's posible to create a LAGG link (2 links from pfSense box1 to switch box1, 1 link from pfSense box1 to switch box2 - and same for pfSense box2, 2 connections from pfSense box2 to switch box2 and another one to switch box1), but apparently you would need to have stacking kit between those Catalyst. Question is: is really stacking kit needed here or is it possible to do cross-stack LAGG by just creating an LACP trunk link between the switches? (simulating the stacking kit). Thank you very much, kind regards David

One Interface can only have one MAC address. All VIPs except CARP hooking up on it have the same MAC. But since it is a virtualized pfSense, you may add multiple virtual WAN Interfaces to it. On each one you can set another MAC after.