It's a known issue and has been that way forever. CARP and bridging aren't really things you want to mix in general. (Search the forum, posts from myself and others over the years might convince you…)

Sounds like traffic isn't getting to the CARP IP for some reason - two most common would be an IP conflict, or a stale ARP cache upstream from where that IP was previously assigned elsewhere. Packet capture on WAN would confirm or deny that.

Hi again.

good news, I solved the problem…

I've looked at the PFTOP from my 2nd server, and I saw in the rules list only 3 rules... deny all...
I have a alias that include a URL list.  the path was http://127.0.0.1/list.txt (the list is local on the FW)
I put the list on the 2nd fw, reload filter, and  voila !

Ciao !

You have to change your interface IPs to CARP IPs, which requires having the interface IP (your default gateway internally, and destination IP on WAN-side traffic) removed briefly. Generally that can be done quickly enough that it doesn't impact any hosts that are already online, or any inbound traffic as long as it's shorter than the period of the ARP cache, which it'll easily be. I've done it many times on production networks on the fly without dropping a packet, but you need to be careful to be ready to add the CARP IP immediately upon changing the interface IP or it's possible you'll create an outage until the CARP IP is added. I always have the CARP IP ready to save, change the interface IP, save the CARP IP. In that case it's just a matter of how quickly you can apply changes on one tab, and click Save and Apply on another.

Some more details/ideas here after reading the book again:

Since I don't appear to have a true routed subnet, it looks like I could connect a switch to the Verizon ONT.  Off that switch I'd see my 5 IPs (I believe I did test this back at install time).  I'd use 3 of those 5 for a CARP failover set-up.  If that's all true (I think it is), then my question is how I can use the other 2 IPs.  Without the switch I use CARP/VIPs to associate those other addresses to my primary IP.  It's not clear to me how that looks with a switch in between the two now.  Seems like I could either still do the CARP/VIP trick (switch has no effect other than splitting off the two IPs I need to separate for failover), or it seems like I might have to have pfSense see the split extra IPs as multiple WAN IPs (which I'd use without failover).  The problem with that second scenario would seem to be that I can no longer pool the extras for a set of NAT rules to fan them out to the DMZ behind pfSense – that if I split them either one pfSense box would get the extra two, or each pfSense box (primary, secondary) would get one of the two extras.

Finally, they manage to route the whole block of IPs to one interface of their equipment (the one connected to pfsense), so I created IP Aliases for routing services to other IPs.

Best regards

Kostas

@cmb:

You can't. Your interconnect with your ISP must be a /29. They should be willing to switch you over to that, it's not an uncommon request since basically every router/firewall redundancy protocol requires it.

Ok, thats what I thought.
I'll see about getting that changed.

link light blinking rapidly wouldn't be indicating traffic (if it actually is the link light, should have a different light for activity if it has one at all), that's more likely to indicate link cycling on the NIC. I have seen NICs not play nicely when directly connected, does it behave better if you throw a small switch between? That'd at least confirm or deny that suspicion.

To use carp you will need one real ip for each pfsense plus all others using carp.

I suggest you to use one of your 4 ethernet ports to sync between boxes.
A new feature on 2.x that will help on vip assigns is in this post from jimp
http://forum.pfsense.org/index.php/topic,45209.msg240909.html#msg240909

After sync and carp, just create your 1:1 nat on firewall -> nat and then change your outbound nat to manual to create your specific outgouing nat translation rules.

By default, all interfaces but lan has no access to anywhere. You will need to change this default rule to deny access from lan to dmz.
All other rules you can create on interface that traffic starts. If you want to allow internet access from a host on dmz, the rule will be on dmz. If you want to allow that everyone can reach your web server, then rule will be assigned on wan.

I gave this a second thought, and realized this just can't be possible!
So down to the cellar again, testing a third cable and another port on the switch - now it works!

Nothing wrong with config, most likely the switch "remembering" where that host is. (Sometimes I miss those good ol' hubs!  ;) )

In the docs are a Description of the diferent VIP types:  http://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses%3F

I know this thread is old,  but I just wanted to comment that I ran into this issue also and noticed that on the the 2.0 version there is a setting under  on primary node goto Firewall –> Virtual IP's --> Carp --> scroll down to "remote system password" here you can enter the new system password.

Thanks Perry

Have switched better all mine PC and VoIP to LAN Subnet 192.168.2.0/2 and use LAN Gateway 192.168.2.101 from HP pfSense.
Works excellently. Now must I more learn about pfSense.

No PPPoE, no DHCP, all statically.

Have in

/etc/rc.conf

ifconfig_msk0="inet 192.168.2.3 netmask 255.255.255.0"

defaultrouter="192.168.2.101"

#defaultrouter="192.168.1.110"

Cool thanks I can confirm this working.

When enabled, everything That pass through firewall will be nated using interface address. Just like the rule created to wan when you selecet outbound.

It's done on pf level, not in gui.

I got a chance to power cycle the Master today.  That did not help.  Since this problem started occurring after upgrading to 2.0.1, I'm tempted to open a bug report.  The issue seem to relate to the number of states we are running.  We had been setup (by default I think) for 388K states.  As we were running as much as 350K states I changed the systems to support 800K states - that seems to have made the problem a little worse.  I cannot see a way to configure my way out of this issue, I believe the hardware and physical layer are working properly (can't find any problems there).  Any other thoughts from the community are appreciated.

@jimp:

Ah, the 'carp' bit was probably left over from 1.2.3 and not updated. If you just use "vip" it may work also.

That file isn't written from the GUI, it's just there on the install. It would be overwritten during an upgrade, but it's left alone otherwise.

That explains a lot.

I use explecit vip1 because we also have a vip2 and that may not trigger the bridge port to UP or DOWN.

That should work exactly as you describe, though you will need to make sure your firewall rules on that interface will pass traffic from the new subnet.

I've done that several times

@network1:

What i've done is gone to Virtual IP's > Created a new IP Alias and Assigned it to WAN CARP interface. This has replicated over onto my other box so presume this is the way to do it?

assign a valid ip on each wan interface
configure sync between pfsense boxes(use a dedicated interface for sync or a vlan)
go on firewall-> virtual ip and add a carp ip(not an ip alias) with the same subnet you configured wan interface

Performance isn't relevant to VIPs. It's best to have the bigger subnet routed to an IP in your smaller subnet, but VIPs generally fine too, though that gives you less flexibility on using the second subnet.