After doing some more digging, I figured it out.  It was a VMWare thing. I had to set the virtual adapter with a security policy exception to allow promiscuous mode.

There seems to be another issue though - it seems as though there is another client out there on the WAN (albeit, on a different VLAN) using a pfSense box, because I see the same MAC address as what my pfSense box is using for my CARP MAC Address.

Is there a way to change the CARP MAC address so I can differentiate my MAC address from this other person's?

There is a lot in your post, but I'll try to answer some of your questions.
@Izinyoka:

Does the firewall use its own WAN IP for outbound connections, or does it use the VIP?

You should use AON and specify your CARP address for outgoing.
@Izinyoka:

2. Can it also provide redundancy for ISP failure?

CARP is generally used to provide failover if your firewall has a hardware problem. ISP redundancy is a separate issue. You can use multi-WAN failover as one solution.
@Izinyoka:

3. Im sure I read somewhere that CARPDev can provide redundancy like in case 1. with only one public IP, is that correct?  because it looks like CARPDev is included in version 2 and I tried it but I don't see a difference (inbound packets are still distributed between both boxes randomly).

I have to check out the newest 2.0 snaps, but AFAIK, CARPDEV is not yet stable on FreeBSD and not in 2.0.
If you were using CARPDEV, the WAN interfaces would have private IPs and they would share the public CARP IP.

@purdue512:

I'm feel for you here. You are doing the right thing..

I did the same (put two switched on the private (lan) side to avoid a single point of failure). But I did not have any CARP problems. I was under the impression that the carp traffic was through the synch link…  At least for me..

Do you have your incoming WAN links setup this way or just the lan side?

Andy

I think the issue on your internet LB is that you are NATing on the same subnet. I know that we've never been able to do this in two different implementations that are similar to yours…  Just ain't gonna work.

@blak111:

You could also accomplish this using static only DHCP entries for the machines on each firewall if you don't have VLAN capable switches.

Yeah.
We did that at the last LAN party i helped organize.
But if you cannot get your guests to register their MAC before the party it's a pain in the ass…
People check in; someone has to go to their place and get their virus-check and their MAC, go back to the checkin, add their MAC to the list at the correct place....
Maybe in the end too much of a hassle.

Hi,

I have a simpler setup with just WAN and LAN.  The WAN has 5 static IPs.  I wanted to forward the ports to the servers internally.  The primary IP assigned responded very fast.  But the other VIP have very slow response.  To make a valid test, I forwarded the http port of each static IP to an internal IP of the same internal server (with differnet LAN IP respectively) using virtual host, serving the same exact content.  I've tried using VIP as CARP, PARP, and other.  All have performance issue on the VIPs.  Does any one know the causes to this?

Thanks,
Tommy

BTW:  I'm using pfsense version 1.2.1 RC2. I have 0 In/Out errors on status > NICs.

FWIW, there is some discussion of this here: http://forum.pfsense.org/index.php/topic,7039.0.html

CARP supports active/active setup, but pfsense doesn't (at least not out of the box).
If you want such advanced configuration i would advise you to study CARP and fix pfsense to be in active active (switches are important too in this case). If you do it, post a detailed description on how you did it, so others can follow.

The easier option would be just to upgrade hardware of your firewalls. If you have such traffic you have to have enough money. You could even buy support from Sullrich i'm sure he would help you set up active/active freebsd install if you paid for it. :-)

I haven't used virtual server pools and other related stuff with pfsense yet, but it should be nothing more that a bunch of DNAT rules. For those you need VIP (virtual IPs). I would recommend you to go and create one CARP VIP then try to use then in your load balancing options.

Af far as i know pfsense doesn't have to be BGB aware (i could be wrong thou).
You would set it up the same way if there were no BGP, just make sure that both firewalls are connected to all networks.

I guess noone understands your post.
How can you still access (measure the pings) if you remove the VIP and NAT rule? If you remove those then you cannot access the user behind in LAN from internet, so you cannot measure it.

'cause no one responded i'll try.
What kind of load balancing do you do? Outbound internet load balancing or inbound load balancing to servers.

Hello

I think I posted to soon, i discovered that I had a slight misconfiguration in one of the firewalls interfaces settings, it was set as a /24 when it should have been /25 so there for the new subnet I was adding was encompased ino the /24 range.

I will post back if im still having the issue, I just reconfigured the firewalls and rebooting the backup one

So far it looks good I can actually route through that network onto the internet now

Sorry for wasting your time.

No the vlans are vlans. Each of our customers are on their own vlan behind the firewall with public address on their machines. We route through the firewall to the public addresses, no nat or port forwading. Everything in the forum seems to be with nating and port forwarding from the public ip's on the WAN to private ip's.

This is the current setup that we are trying to cluster.

WAN Gateway              FW1 WAN              Vlan1 interface          Customers machine
xxx.xxx.58.145<–-->xxx.xxx.58.148<------->xxx.xxx.53.1<--------->xxx.xxx.53.2

Thanks

Rick

For the people that have been reading this hoping for answers to my questions, maybe I can help you out.
From a newbie standpoint, I have this working:

my ignorance was in understanding that virtual ips, thus this forum being CARP/VIPs, is the key to using CARP/pfsync

Here's what I have that is working in a small test environment:
box1:
WAN: public IP
LAN: 10.2.1.1/16
OPT1: 192.168.10.10

box2:
WAN: public IP
LAN: 10.2.1.2/16
OPT1: 192.168.10.11

In the CARP settings I am sychronizing everything and using the 192 addresses as the peer sync addresses for each box respectively.  Box1 has 192.168.10.11 and the webGUI password entered at the bottom of the page on the CARP settings. Box2 has these boxes left blank.

Under virtualIPs I created a new local address of 10.2.1.4/16 associated with the LAN interface that is of type CARP.  Put in a VHID password and bahm, CARP up and running.
You can post beginner level questions and I can try to pass on the little bit I've learned.

Dotdash i sent you a private message with a clarification of how things are setup

Apparently it was it.. I had to simply reboot the two FW.

Weird..

Now it's working fine.. (have been for two days)

Thank you for your support! ;)

Create a VIP PARP with the Outside address you are wanting to use. .5 I assume.
Go over to the Load Balancer Tab/section

Create the Pool first, your web servers Create the Virtual Server this is the Same IP Address as the VIP

The VIP is going to be your Outside address and the Pool is going to be your in side address
Also don't forget to create Firewall Rules to allow the Web traffic from the outside interface to the inside interface.

Also When create the Virtual Server it will ask you for a "Pool down" server I used an old server matching my 2 production Web servers as my Pool down server it is Really slow so I didn't want it in the round robin but it is a nice fail back plan…

Hope this helps

Never mind. I was looking for load balancing on UDP but I learned that verion 1.2 only supports TCP load balancing.