Wait some seconds until your switch flush mac address table. Also connect to your switch and see if there is any problem with mac table(full) or cpu usage.

I've just noticed that if i change the VIP in PfSense from CARP to IP Alias, then the problem disappears. Any thoughts?

Nevermind, I'm pretty sure this falls under the "well, duh" heading. I probably had the wrong password set on the primary as well as the "sync usernames/passwords" enabled. So when I set the admin password on the backup machine to the correct one, the XML got synced across, which reset its admin password to the wrong value which caused future syncs to fail and locked me out of the web guis. I reset the password on both, reset them to the proper values I want and then made sure syncing of usernames and passwords was not enabled in the virtual IP XML sync settings. So far, so good.

Did you use port forward or 1:1 NAT? If you are using port forward, then you will need to use advanced outbound NAT (manual mode) to transform the outgoing ip to 201.73.17.178. Remember that it is first matching rule in AON so if your LAN rule is above your custom outbound, then the custom outbound will never happen.

@zeratoun: Exactly, i want that, from the localhost of the pfsense firewall itself it uses the VIP LAN or WAN …. it's possible ? Best regards, It is possible but highly NOT recommended. I got that running in my test environment and CARP was not happy as ping stopped to the gateway on the secondary firewall. I think this will have an adverse effect on the clusters ability to fail over correctly. I didn't have a chance to test fail over, but i did notice that I could not download packages or ping the gateway. There is not reason I can think of to do this. Would you mind telling us why you would like to do that?

The CARP bad gateway has nothing to do with that. I would upgrade to the latest 2.0 release regardless, though I don't think that will fix your problem, 1.2.2 is a very dated release. Not enough info there to have much idea what's happening, exactly what pings work and don't, and where they're initiated and destined isn't clear.

Thanks to everyone I now have what "appears" to be a working config. Here are the steps I took. For the basic /30 link to ISP I setup the /30 IP on the WAN interface. I then added my /27 public ips as VIPs using IP Alias - as pointed out above, use /32 and add them 1 at a time. Since I have a Layer 3 switch attached to my LAN interface, with 3 routed subnets + the pf LAN interface subnet, I had to create 3 routes in pfSense with the gateway pointing to the switch IP on the pf LAN Subnet. THEN - TO MAKE MY LIFE EASIER (and this may differ for you), I created a Network-Type "ALIAS" in pfSense and added my 4 LAN subnets to that alias. Then I turned on MANUAL OUTBOUND NAT (AON = MANUAL). I edited the default 2 LAN subnet rules and changed the LAN Subnet to my "Network-type ALIAS". Then finally, I edited the default LAN Firewall Rule and where it originally said "LAN SUBNET" I simply changed that to my "Network Type Alias". EVERYTHING SEEMS TO BE WORKING – AT LEAST AS FAR AS OUTBOUND TRAFFIC USING VIPs. My next project is to get VPN working using one of those VIPs as the destination.  Whether this will continue to work remains to be seen. MANY MANY THANKS TO ALL WHO HAVE HELPED IN MY NUMEROUS THREADS ON THIS ISSUE. ALSO -- A Special thanks to "Metu69Salemi" who was relentless in his efforts helping me through PM.

Bingo. I'm not a network admin by any means (sysadmin is my primary role), but that gave me enough information to figure out the problem. I didn't explain the details of how all of the switches were connected (which appeared to be more important than I thought), but here's the diagram for future people.   Internet       |       | ------------- | L L L L L | Switch 1 -------------     |  |          |  --------- (to router WAN)     |          | WAN |        -------------     |        | W L L L L | Router 1     |        ------------- firewall1        |  |   |    |        |  |   |    |        |  | LAN  OPT2 -------  -------- (to secondary LAN switch) The CARP multicast was going out of the firewall WAN, then reaching the router WAN. The router WAN (D-Link DI-524) was setup to pass multicast traffic through to the LAN, which then made it show up on the same network as OPT2. I disabled the multicast forwarding on the DI-524 and the packets stopped. For those that wonder why I'm doing it this way, the two firewalls on the secondary LAN also use load balancing on their WAN interfaces and need three IP addresses. I don't have enough public IP addresses to assign to the two pfSense firewalls plus the other two firewalls, so I'm doing a 1-1 NAT of one public IP through the DI-524 onto the virtual IP of the two firewalls on the secondary LAN. The OPT2 interface is for creating an IPsec tunnel between the two networks that doesn't traverse any of the public Internet. Thanks for your help.

Well if you don't want the CARP VIP settings to sync then don't sync the VIPs. Just uncheck that option under the CARP settings. Then you can adjust the VIPs however you like. I will note, however, that splitting the MASTER role of VIPs between the boxes is not a config that is currently supported, so do not be surprised if some bits don't work as you expect.

Is there a reason you want to disable it? It's not usually very useful to have that disabled. You could add a system tunable to set net.inet.carp.preempt=0 (but it may not stick since it gets reset by the card code). The GUI option was removed back in 2006, so it hasn't been in any recent version.

@anagh: use isp wan series on wan side and isp lan series i.e first public ip on lan side open firewall nat click Manual Outbound NAT rule generation and SAVE delete all  auto generated  nat rule This doesn't provide the private IP network interfaces he requires.

I had the same type of issue with switching from pfsync setup from multicast to peer mode, i finally gave up on trying to switch them and just used a crossover cable and a dedicated port for syncing. i didnt want additional traffic (multicast sync packets) on my lan interface.

If the wireless doesn’t need direct access to your network resources, then by just having it on a separate VLAN and making sure there is an ACL to block any packets that are part of the WIFI network to your network. However, if you’re WIFI needs access to your network that should also work. What we have done for our installation at our office and data center is to have two switches,  and two firewalls. A trunk between the two switches but then VLANed them into different groups, for example VLAN1 is our LAN VLAN2 is WAN1 VLAN3 is WAN2 Your ISP hand-off would go into the right VLAN group and then so would your firewall connections. If you’re not using trunks then you will have a direct cable for each of the VLANs back to your firewalls with correct addresses. For the DMZ you can add extra ports to the VLANs you need and have them outside the firewall. And then the placement for the WAP again depends if it’s part of your internal or external network access requirements. What this design allows you to do is have redundancy, what if your WAN switch went out? You would loss all WAN connections, where if the wans were both on two different switches the risk of a failed switch is migrated. To answer your question about if it would be secure, if VLANs are done correctly, its taking a physical switch and logically breaking it down to multiple logical switches.

hey, no problem, the fact your helping is amazing so again thank you. i will try  this, we are only allowed to make changes of Tuesday nights, so i will try to  slide this in for this weeks testing. you know that makes sense, you have to have an identifier at each of the end of the tunnel. if it was the direct interface and not the CARP then when fail over occurred the tunnel would have been built with the wrong address (since it was built by the master with one IP which is no longer valid as its down).

I had some trouble at first, what I had to do to fix it is first Verify that ONLY the master sync server has the various sync buttons checked. And just to be safe remove any IP address in the Sync form on the slave servers. Found in the PFSense Documents at: http://doc.pfsense.org/index.php/CARP_Configuration_Sync_Troubleshooting next make sure that snyc is set up correclt by checking: Enable pfSync in Firewall -> Virtual IPs -> CARP settings -> Synchronize Enabled (check it) on all cluster members. -> Synchronize Virtual IPs [ X ] -> Synchronize to IP [ insert Slave IP ONLY on Master! ] -> Remote System Password [ do not forget! ] Select the dedicated Sync interface with the Synchronize Interface dropdown on all cluster members, if it’s on a dedicated port select that port if not then select the port on switch your syncing across.. Afterward visit Firewall -> Rules and add an allow all from any to any rule on each cluster member for the newly created pfsync interface. Found in the PFSense Documents at: http://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_%28CARP%29 I know that this might be a common mistake, but I am new to PFSense and I did the above and it fixed my syncing issues.

Just add new dns entrys for your dns server. I have windows AD with something.local as interal domain and outside domain is actually something_else.com, if i want to connect webserver internally i've created another domain into my AD-server(DNS), now it has internal and outside domains side-by-side. Only different thing to outside domain entrys is that i use only internal ip-addresses for those A & PTR Records at my own server. And ofcourse this AD-server is the one which i share via dhcp to clients

Thanks for the reply.  The switch involved doesn't seem to matter.  The test jig uses a netgear 1000M capable 5 port switch on the lan side, connecting the two routers  (192…2 and 192...3) carp sharing 192..1.  Whichever of them is the carp master can't ping .1, while all other systems can.  The system hosting as the carp master .1 can ping its own native address, but not .1 It seems the actual mac address on the outbound frames routed via the carp interface is not the carp lladdr, but the interface native address.  So, the mediacom cable modem, which must bind to a specific mac address, won't bind to the carp llaaddr (which is VHID specific).  This makes PFSense useless in a failover router setup with that ISP. I've 'worked around' the problem by setting up a third little pfsense box with just two ports, acting as an extension of the cable modem, can't really use 1-1 nat, so I just port forward what little I need.  It's a single point of failure, but then so is the cable modem and it still only risks that ISP's connection, the others are still protected by the failover router pair. Still, if CARP could be improved to use the CARP logical link address when transmitting packets sent out the carp interface, and not the interface's native MAC address, then I could avoid maintaining an extra router and dealing with NAT issues on what should have been a native connection to the net.