Carp setup issues

[subarunut]

Hello,

I am new at pfsense and have been given a task to setup 2 pfsnese boxes to replace our current pfsense firewall with a redundant setup (CARP).  While I wait for the new hardware I am testing the config.  I am running into some road blocks that I can't seem to find a solution to.  I have searched, but haven't found where I am messing up.  I am following the configuration doc: https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP).

My setup is:

box 1:
WAN: EM2: 192.168.1.100/24
LAN: EM1: 10.1.9.251/24
SYNC: EM0: 10.0.0.1/24

box 2:
WAN: EM2: 192.168.1.101/24
LAN: EM1: 10.1.9.252/24
SYNC: EM0: 10.0.0.2/24

Virtual IP's:
WAN: 192.168.1.102/24
LAN: 10.1.9.250/24

Client:
Win7
IP: 10.1.9.100/24

I have been starting from a factory default and setting it up as per the document.  When I get the base config complete, I am able to set the clients gateway to both pfsense boxes and ping the internet just fine.  I then run through the setup with the client pointing to the master box and I am still able to get to the internet.  If I point the client to the virtual IP, I am unable to ping out the firewall.  If I fail the master, and point the client to the backup I am unable to ping out either.  I must be missing something or not setting it up right.  Any advise would be great before I move to the live systems.

Attached are some screenshots of my current config.






[viragomann]

Hello,

you have to set all the NAT Addresses in Outbound NAT to your wan CARP IP 192.168.1.102. Then it should work.

Regards

[subarunut]

Thanks, that worked, at least till I tried to do a failover.

Is the client supposed to be pointing to the vip as the gateway?  I would assume so because the address would change during fail over then.  If I use the VIP as the gateway, at least so far, I have not been able to get out the firewall.

[viragomann]

Yes, of course you have to set your clients to use the vip as gateway. That is the only one available on both pfSense boxes and is still present after a failover.

Same thing an wan side. You can use vips for your purposes exclusively. Therefore at least 3 wan ips are necessary for CARP, one for the wan interface of each pfSense and the other for your use.

[subarunut]

Thanks, that is what I thought.  But I can't seem to get the VIP on at least on the LAN side to be pingable and work as the gateway.

On the wan side, is there anything I missed?  I think I have it set.  but still doesn't work.

[subarunut]

I have created a gateway pointing to the VIP for the wan side and that breaks all internet communication.  So, I am currently thinking that there might be something wrong with the VIP's.  Any thoughts?  I believe that they are setup right (as seen in the attachments above), or am I off my rocker.

[viragomann]

Hi!

You must not use your WAN VIP as standardgateway!
If your WAN VIP is 192.168.1.102 the gateway can be anything else in the same subnet. Normaly it given to you by your internet provider.

[subarunut]

thanks, I have that set then, my gateway is 192.168.1.1.  And in my virtual environment that I am setting p, it is not working.  I use the LAN VIP as the gateway on the clients.  I have created a WAN VIP, but have the WAN NIC setup with the gateway from the internet side (my home router in this case, so I have full control for now, but will have that in the live environment when I move to it).

What else could it be?  should I remove and re-create the VIP's? 
Thanks,
Ben

[viragomann]

Okay, let me repeat.
You have set up your pfSense gateway in System > routing to 192.168.1.1 on both machines, which is you real LAN gateway.
You have set the Outbound NAT Address to your WAN VIP.
Your test clients gateways points at your LAN VIP.
All other settings are as you described in your initial post.
Both boxes show the correct CARP state, master on first pfSense, backup on the second.

And what is not working yet?
Do you get internet on the clients?

[subarunut]

You are correct on how I have everything setup.

The CARP state is correct on both systems.

I am unable to ping though the firewall or reach any internet sites from the client.  I can ping from the pfsense boxes them selves, but the client cannot, and I have verified the client settings numerous times.

[viragomann]

Have you set rules to allow outbound traffic from your LAN?
For testing set up this rule on LAN interface:
ID Proto Source Port Destination Port Gateway Queue Schedule
IPv4 * * * * * * none   Outbound

Also for pinging through the firwall you have to setup additional NAT rule depending on the direction.

pfSense itself responses ping per default on LAN reail and VIP only. If you want to ping WAN VIP you have to add an according rule on WAN if like
IPv4 ICMP * * * * * none   allow ping

Configure all your rules to log for debugging. So the "Default deny rule" (if the packet fits to no other rule) will be logged also.

[subarunut]

Thanks, I verified that the rule is set already on the LAN interface.

I was trying to ping the LAN VIP from the client (10.1.9.160 (client) -> 10.1.9.250 (lan vip)) and did not respond.

I am unable to ping though the firewall from the client also when the gateway is set to the lan vip.  pinging either yahoo.com or 8.8.8.8.  They both ping fine from the CLI of the pfsense boxes.

[subarunut]

attaching my current running config to see if it can help shed light in my mess up.

Thanks,

Ben

config-firewall-backup.local-20140327163947.txt

[viragomann]

Oh my god!

You have blocked private ip on WAN interface! And your WAN is in a private subnet.
Got to interfaces > WAN and remove the checks at the private network area. That is only for use in internet.

Furthermore you have configured a "CARP_WANGW". That is not needed. Delete it please.

I hope it's done by that.

Regards

[viragomann]

One thing else:
That was the config of your backup, don't know why. Anyway here is the LAN VIP 10.1.9.253. You have written to set it to 250.
The clients have to point at this.

[subarunut]

well, drat, that didn't work.  I had created that gateway long ago in TS'ing the issue and just didn't delete it, yet.  hmmm, I am out of thoughts myself.  I had made a brief change on the LAN VIP to that address just to try a theory at that time and have pointed it back to .250.

I didn't have much hope that it would work as before I pointed everything to the firewall directly (.251) it all works fine.  but if I point everything to the lan vip (.250) noting gets though.

[viragomann]

I cant believe.
:-[

Do you have removed the 2 checks at WAN interface config?

I have taken your config and have imported it into a virtual pfSense in a similar environment like yours. I have made the described changes and reconfigured interfaces and IPs to fit for my subnets.
I have made it to the master, then I had to delete and redefined the CARP IPs and the Outbound NAT and now it works well! Both, browsing Websites on LAN client or pinging public IP addresses work here.

I cant see why it shouldn't in your environment.

[subarunut]

at least I am not the only one here banging my head then :)

Maybe when I get the hardware in it will magically work then.

any other thoughts would be most welcome.

[subarunut]

This is so weird.  So I got one of the hardware boxes in and set it up with a basic setup (missing the 3rd nic so I just created a vlan on the LAN network) and setting up the vip 10.1.9.250 and setting the NAT rules destinations to point to the WAN vip and the client gateway to 10.1.9.250.  And the bloody thing works.  now why it doesn't in my virtual environment, I have no idea.

[viragomann]

You didn't tell us what kind of hypervisor you are using for your virual pfSense.

Maybe there are some configuration changes to be made. For instance the virtual switch the interfaces are connected to have to allow the interface to change the MAC adresse.

If you use ESX take a look here: https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP)