Now that I had time to work on this after hours when traffic is to a minimal, here is my solution.  It only took 5 minutes to config and test.  All seems fine.

1. Setup WAN Interface with the new public IP
2. Created a ProxyARP VIP for Old Wan IP
3. FW Rules don't need changing … nor do I need 1:1 mapping
4. Downloaded current XML backup file
5. Copy/pasted all my NAT rules in backup file for quick duplication
6. Added this to each duplicate rule: <external-address>Old IP</external-address>
7. Restored FW with updated XML file
8. Tested a few of our services and sites which still have DNS with Old IP... all OK
9. After the Old IP expires I just delete the VIP and duplicate NAT rules.

I hope this helps someone with a similiar issue.

Thank you very much for the info.

The state table won't become a problem as far as I can judge now.

Now it realy is a GREAT firewall! ;-)

I think the problem was mostly a bad network connection.

However I am having the problem now where the secondary box keeps taking over master role and the primary releases and runs as backup.

I keep going to the Firewall >>VIP >> Carp settings,  on the master machine and hitting save.

I think that should reassert its role as master correct…? If not, how do you?

I have gotten this to work by just enabling "Synchronize Enabled" in the CARP Settings and selecting the interface desired, the firewalls will find each other via multicast and tell each other what states they have. I am load balancing across multiple firewalls and need to handle as many states as possible. I have also gotten syncing of rules working by following all the instructions for CARP but leaving out the virtual IP parts.

dotdash, you saved the day!  :)

So it has been working all along, I just did not properly perform the ping test, when I had the VIPs set up with type Proxy ARP.

Thank you very much, your 4 steps showed the way to success.

BTW I had conversations with a lot of ppl today in various ways, and they all just told me, that this setup can not work properly, and that my best chances are to return to the former setup with the DSL modem handling the PPPoE connection. Now I can proof them wrong! :)

only the primary machine has anything to sync, aside from the config. If you sync the config from primary to secondary and secondary back to primary you're going to break stuff. It's possible to set it up that way, as I've heard of people doing it before and breaking stuff.

@dbuckle:

I was having a similar problem on a wrap board where whenever the CRAP interface was changed

Hey now, don't call it names.  ;D

Yeah, the config file can be restored. You'll have to reflash to get to 1.2b1, but once you're on that version you can use the firmware update page to upgrade going forward.

Hi,

Thanks for the kind link. I was searching the forum, but in some strange way I got 0 results for some time.

I have read about the spanning tree option, this migt be a good idea, but this solution is also what I really like, thanks a lot !!

I got it.

My ISP had not removed the filter on their side, even though they had assigned the ip to me.  Once they made the change, everything started working.

Cheers!
Ken

There's really not much to the VHID numbering other than making them unique.  This is not something that you'll ever need to change or inspect once it's set up.

-Martin

turn on nat reflection

Thanks a lot for your help!  The passwords were correct.  What I ended up doing is deleting that CARP interface and recreating it.  Then I rebooted the backup machine.  When it came back up, it became backup for all the interfaces, including the one I had a problem with.  I made the master fail so that the IPs were handed over to the backup, and when the master came back up, the backup gave all the IPs back to the master without issue.  I hope this just stays this way and wasn't a one time thing…

Please test a recent snapshot.  I have added a patch that may resolve this issue.

Not to make life more complicated, but how would I add BGP into the mix to provide failover to another site?

Eric

@Perry:

You setup vlans like any other nic
http://pfsense.hotserv.dk/hmm.htm

VERY, VERY helpful … thanks bunches!!  I have it up and running now with little difficulty thanks to this great presentation.

@rexsrexs:

I can't make the CARP type VIP with subnet xxx.xxx.xxx.44/32 the pfbox will also complaint, it said

Sorry, we could not locate an interface with a matching subnet for 202.133.1.44/32. Please add an ip in this subnet on a real interface.

If you are using a CARP VIP, the subnet mask of the VIP should match the subnet mask of the Interface (/29 in your case). The 1-1 NAT should still be a /32 to match one internal and one external address.