So here's what I ended up doing. Since the setup was basically the same as another setup I had done that was working, I figured I'd just mimic that and hope for the best. I don't have the failover configured yet, but here's what I have.

ISP Gateway (x.x.157.17/29) <-> pfSense WAN (CARP x.x.157.18/29) <-> pfSense LAN (CARP 10.205.154.66/21)

1:1 NAT
x.x.154.66/26 -> 10.x.154.66/26
x.x.210.0/23 -> 10.x.210.0/23

NO VIPs (outside of the CARP WAN and CARP LAN)

I guess since my ISP routes the two subnets (above) to x.x.157.18 as the "next hop", pfSense automatically handles those requests since they match the external IP address of the 1:1 NAT entries. This works out nicely, since I can't see any reason why the subnets wouldn't fail over to the secondary server because neither server is "advertising" that they control the subnets; they just utilize the 1:1 NAT to map them after they receive the packet.

Before I finalized this setup, I took a look at the "Other" and "IP Alias" VIPs, and I noticed those can only be done on individual IP (same thing for CARP). This obviously wouldn't have worked for my setup, because I have well over 500 IP addresses.

Hi,

Thanks for that clue podilarius, after looking at the state table I noticed ICMP packets from pfsense to the 192.168.3.1 WLAN router were going via the LAN 192.168.0.1 vip.

This led me to the Manual Outbound NAT rules and I had a rule there saying:

"WLAN 192.168.3.0/24 * * * 192.168.0.1 * NO"  (WLAN to vip1 LAN)

Removing this fixed it!

I had to remove stale states from the state table manually too for changes to take effect immediately as the gateway status still showed the 192.168.3.1 router as being down (through apinger).

I also didn't have a default gateway set on the backup so setting that fixed the routing tables.

And on a slightly different note, my 192.168.2.1 adsl modem/router didn't pass multicast over it's switch (they're just getting too clever and locked down these days!) so I had to put in another unmanaged switch inbetween to allow vip2 interfaces to switch from master->backup properly, (as stated in the sticky, but I had to read that more than a few times before it sank in!)

Looks like it's ok now, thanks for the help!

Best Regards,

Vent

Yes, MASTER/BACKUP status changes are logged (and mailed to me \o/). I am using these network cards: Intel Pro/1000 PT Quad Port LP Server Adapter. PfSense finds them as em0, em1, em 2 and em3.

Thanks again :)

I had a couple physical boxes I was able to try this on and it worked OK.  So I am guessing it is an issue in the way VMware is configured…  I do have 2 hosts each with 2 physical NICs for vmguest networking going to 2 physical switches (trunked between them).  I'm using a VDS with a separate port group that has promiscuous enabled on the VLAN that has my CARP VIP and have configured the hosts with Net.ReversePathFwdCheckPromisc = 1.  Is there anything else I am missing?

Thanks, I initially thought Net.ReversePathFwdCheckPromisc = 1 was for a DVS. Changing this did the trick!

With this rules, only https does not pass from vip (192.168.3.0/24)

I am still struggling to wrap my head around this configuration. Ultimately what I am looking to do is prevent hosts within the same subnet from seeing each other and have the firewall rules enforced as if the host was external from the other system. I understand how to accomplish this with ASAs but not with PFsense. We're also utilizing carp so the solutions must failover. I have seen many posts suggesting to stay away from carp and bridging.

We currently have 2 pfsense boxes with 6 interfaces and we're looking to split our subnet in to about 10 separate security contexts.

Any insight is greatly appreciated.

@termvrl:

how i can do NAT for it??

Just select your vip wan ip on destination, just like you do with wan address  ;)

Slightly off topic, but related.

How could I achieve the following example

there are 2 sites connected via a low latency wireless bridge (100Mb over 15KM)

I want to do this without having to have a seperate subnet for each site, and a seperate pfSense clusters at each site

Trying to achieve a single broadcast domain I suppose would be the best term.

This would allow me to have VLANs across the two sites without having to have seperate IP ranges for all the VLANs (reducing admin overhead as theere are 20+ VLANs) which will then turn into 40+ subnets If I have to route it all between them.

I was initially thinking active/active could have been used to achieve that

Site 1. pfSense-1 and pfSense-2
Site 2. pfSense-3 and pfSense-4

All of these 4 pfSense could have then formed a single VIP in the same broadcast domain

is there another way around this?

any other ideas would most be appreciated.

In 2.0.x that's done via routing protocol, most commonly OSPF. In 2.1 you can bind OpenVPN instances to a gateway group as another option.

You might even be able to do this with only one private network by connecting 192.168.1.2-50 to the pfsense LAN port and then connecting 192.168.1.51-254 to an OPT interface that is bridged to the LAN interface.

How can I set up this option sir??

It's more CARP, not pfsync. Our base OS doesn't have that functionality. It's not exactly all it's cracked up to be really, which is true of all active/active firewalls, commercial and open source. For instance on Cisco ASA's there are massive restrictions, like you cannot use any VPNs with active/active for one. We'd likely also have to enforce similar restrictions in a number of areas including VPNs. The restrictions rule out things more than 99% of the HA installs I've worked on (likely upwards of a thousand in the last 8 years) require. Hence, it's not really all that attractive. We may implement it at some point, but it'll almost certainly come with restrictions like no VPN usage. It also may not actually increase performance, by the nature of how it works and where bottlenecks exist that define the maximum throughput on a given combination of hardware. It's something that would have to be tested.

Hello,

May be it is interesting for anybody who has the same challenge. Finally i got it to work, when i additionally configured a vip with the same
address as the secondary to have the possibility to select it in the outbound nat configuration as the nat address.

best regards
daniel

@GruensFroeschli:

I would just update to 2.0.1 (or even 2.0.2, search the forum) and use what's available there directly in the GUI.

(Yes it's theoretically possible but not through the GUI).

Thanks! I appreciate the lightning-fast answer.

So maybe I do the 2.0 upgrade first, and the other later.  That makes sense.

@vegaslaptop:

Hi everyone,

I am baffled at this setup. I have VHID on physical NICs working with no problem (WAN, LAN, SYNCing). Past pfSense I want to have the trunk from the primary and the trunk from the backup to go into Cisco 3560 without "flapping". Something like HSRP on Cisco but I cannot configure it or I think I am missing something.

Any comments or guides will be highly appreciated.

Does Cisco support CARP now?  If I recall correctly CARP was a response to HSRP/VRRP being proprietary, and Cisco not being willing to make adequate commitments that it would be open and raising issues of a patent suit over it.  If that's still true, as I think it is, then there isn't any interoperable failover protocol.  See http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

If you're seeing "flapping" on the trunk ports, this might be a spanning-tree-protocol issue, which normally affects bridged configurations into switches.  I don't know how VHID, CARP, and pfSync interact with STP so I'll let someone else speak to that; nonetheless, you might need to turn off STP on your trunk ports on the switch, if you know they're not going to bridge, to keep the switch from shutting one or the other down as a potential loop. (If they might bridge and loop, believe me, you're better off with it flapping.)
  – Clifton

Yeah that's fixed on 2.0.2 and 2.1. Forgot that was broken on 2.0.1.

I guess to prevent this, I would boot the master and break the sync relationship on it followed by configuring it as the new slave?

Perfect friend, thanks very much. ;D