I am still struggling to wrap my head around this configuration. Ultimately what I am looking to do is prevent hosts within the same subnet from seeing each other and have the firewall rules enforced as if the host was external from the other system. I understand how to accomplish this with ASAs but not with PFsense. We're also utilizing carp so the solutions must failover. I have seen many posts suggesting to stay away from carp and bridging.

We currently have 2 pfsense boxes with 6 interfaces and we're looking to split our subnet in to about 10 separate security contexts.

Any insight is greatly appreciated.

@termvrl:

how i can do NAT for it??

Just select your vip wan ip on destination, just like you do with wan address  ;)

Slightly off topic, but related.

How could I achieve the following example

there are 2 sites connected via a low latency wireless bridge (100Mb over 15KM)

I want to do this without having to have a seperate subnet for each site, and a seperate pfSense clusters at each site

Trying to achieve a single broadcast domain I suppose would be the best term.

This would allow me to have VLANs across the two sites without having to have seperate IP ranges for all the VLANs (reducing admin overhead as theere are 20+ VLANs) which will then turn into 40+ subnets If I have to route it all between them.

I was initially thinking active/active could have been used to achieve that

Site 1. pfSense-1 and pfSense-2
Site 2. pfSense-3 and pfSense-4

All of these 4 pfSense could have then formed a single VIP in the same broadcast domain

is there another way around this?

any other ideas would most be appreciated.

In 2.0.x that's done via routing protocol, most commonly OSPF. In 2.1 you can bind OpenVPN instances to a gateway group as another option.

You might even be able to do this with only one private network by connecting 192.168.1.2-50 to the pfsense LAN port and then connecting 192.168.1.51-254 to an OPT interface that is bridged to the LAN interface.

How can I set up this option sir??

It's more CARP, not pfsync. Our base OS doesn't have that functionality. It's not exactly all it's cracked up to be really, which is true of all active/active firewalls, commercial and open source. For instance on Cisco ASA's there are massive restrictions, like you cannot use any VPNs with active/active for one. We'd likely also have to enforce similar restrictions in a number of areas including VPNs. The restrictions rule out things more than 99% of the HA installs I've worked on (likely upwards of a thousand in the last 8 years) require. Hence, it's not really all that attractive. We may implement it at some point, but it'll almost certainly come with restrictions like no VPN usage. It also may not actually increase performance, by the nature of how it works and where bottlenecks exist that define the maximum throughput on a given combination of hardware. It's something that would have to be tested.

Hello,

May be it is interesting for anybody who has the same challenge. Finally i got it to work, when i additionally configured a vip with the same
address as the secondary to have the possibility to select it in the outbound nat configuration as the nat address.

best regards
daniel

@GruensFroeschli:

I would just update to 2.0.1 (or even 2.0.2, search the forum) and use what's available there directly in the GUI.

(Yes it's theoretically possible but not through the GUI).

Thanks! I appreciate the lightning-fast answer.

So maybe I do the 2.0 upgrade first, and the other later.  That makes sense.

@vegaslaptop:

Hi everyone,

I am baffled at this setup. I have VHID on physical NICs working with no problem (WAN, LAN, SYNCing). Past pfSense I want to have the trunk from the primary and the trunk from the backup to go into Cisco 3560 without "flapping". Something like HSRP on Cisco but I cannot configure it or I think I am missing something.

Any comments or guides will be highly appreciated.

Does Cisco support CARP now?  If I recall correctly CARP was a response to HSRP/VRRP being proprietary, and Cisco not being willing to make adequate commitments that it would be open and raising issues of a patent suit over it.  If that's still true, as I think it is, then there isn't any interoperable failover protocol.  See http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

If you're seeing "flapping" on the trunk ports, this might be a spanning-tree-protocol issue, which normally affects bridged configurations into switches.  I don't know how VHID, CARP, and pfSync interact with STP so I'll let someone else speak to that; nonetheless, you might need to turn off STP on your trunk ports on the switch, if you know they're not going to bridge, to keep the switch from shutting one or the other down as a potential loop. (If they might bridge and loop, believe me, you're better off with it flapping.)
  – Clifton

Yeah that's fixed on 2.0.2 and 2.1. Forgot that was broken on 2.0.1.

I guess to prevent this, I would boot the master and break the sync relationship on it followed by configuring it as the new slave?

Perfect friend, thanks very much. ;D

Thanks for the reply.  That's another thing all together.  From the OpenBGPD package info - the filter rules are evaluated in sequential order, from first to last.  But, in the Raw config tab the top line of the config says # This file was created by the package manager.  Do not edit!  And I'm fairly certain those rules were created for me.  I might delete all config entries and start from scratch to see if those come up.  I'm mostly concerned about my advertisements right now and making sure everything goes to the right places.

Aaron

We typically suggest editing /etc/rc.carpmaster and /etc/rc.carpbackup so that the OSPF daemon gets stopped while in backup mode, and started when it's in master mode.

Given the dynamic nature of OSPF, it really doesn't matter if it's advertising the CARP VIP, if the adjacency changes it'll need to update anyhow. Might have a few seconds of downtime as the neighboring comes back up during a switch, but afaik there's no way to have the OSPF daemon actually advertise the CARP VIP as the router address.

Alternately, you can also just setup OSPF as normal on the secondary box but with higher costs than the primary box has, so it would fail to that one quicker, perhaps. I can't remember if I've tried that one before.

Thanks for your answer cmb.

I will consider switching to IP aliases. Seems the right move specially having the advantage of log notifications in case of IP conflict.

You can't do active/active CARP nodes in that way. Only one router node can/should be active at a time for handling any traffic. You can't spread the load between multiple firewall units in an effective way at this time.

Some people have hacked things up manually to make some VIPs master on the primary box and some VIPs master on the secondary box but that's really not something I would suggest.

As you noted, multiple WANs should be connected to all carp nodes to be used properly. Anything else is just ugly and asking for trouble…

Shame on me!
I forgot to enable the first 3 options in the carp setting page on the slave firewall. I didn't traslate well page 393 of pfsense book!