Whoa, I knew it's not problem of pfSense. My co-worker had done mistake in ESX advanced configuration - the 'Net.ReversePathFwdCheckPromisc' parameter must have the value of '1'.

This is mostly a routing way of doing.
You can do through gateway failover or through ospf routing protocol.

So each of you should see the others firewall as a provider/gateway

Note: I believe this should be moved to Routing/MultiWan

I wasn't able to make this work. I made a better diagram:

Note the "Desired Configuration" on the image.

I can see connections coming in from the internet on the 30.10.0.2 IP just fine. The Cisco still routes this connection because the IP is on its subnet and it is directly accessible.

But replies will go out through the pfSense because that's the default gateway of the client.

How can I make the pfSense route connections from 30.10.0.x/26 back to the Cisco?

I tried using policy routing with a rule of Source 30.10.0.x/26 -> gateway IP: 192.168.100.1 (dedicated VLAN interface to Cisco) - but, initially, pfSense just dropped the packet and didn't even let it exit the firewall even with a 'pass all' rule. I had my head scratching for a few hours until I tried changing 'keep state' to 'none' on the rule, and I could now see it leave the DMZ interface, but it now gets stuck trying to exit the VLAN interface. The 'none' trick didn't work here, no matter what I tried (pass all, etc), the firewall didn't let the packet go out.

Here's the packet getting stuck on its way out:

My understanding is that the following needs to happen:

I looked through the pfSense book several times but couldn't find a similar scenario.

Any ideas? Is this even possible or is there a better way?

(I'm still not sure whether the Cisco will allow these packets to go out, they probably wouldn't have any state associated and they would come in on a different interface - the VLAN.)

I have the exact same problem as the thread starter. As a workaround it works like gullio said, but only for a few hours.
The WAN interface is the via-rhine nic of my Via Epia-M. This nic is connected to a Cisco 1700 router from my ISP.
Is it ARP related? Would it help to use another nic for WAN?

CARP systems must have an identical set of interfaces in the exact same order.

That has always been the case.

Thank you jimp it work :)

Ah, I think I have a better understanding of what is really happening here.

The only IP addresses that are showing up are ones that are for a Microsoft Load Balanced IP with two members.  I guessing what is getting blocked are the packets that are viewed as out of order by the non-active based on the fact that the primary firewall has already gotten past the part of the connection setup that a given packet type would be expected.

So sorry for the false-alarm.  I just noticed when I went back through the logs that it was only happening on the LB IPs.

The more I'm exposed to this implementation of load balancing the less I like it–unfortunately, we are committed to this at least for the near future.

Thx Jimp for your reply. Your method solve my problem. Cheers!  :D

In that case you'd use the whole /27 on a "DMZ" segment (And you can still do CARP there if you want if you need redundant routers) which doesn't get NAT, and then have a "LAN" segment with private IPs that does get NAT. You can filter between the interfaces that way.

If you want to split the /27 on the inside into multiple interfaces you'd have to setup one interface with the /27 on it and then bridge the second internal interface to that one. I try to avoid bridging if at all possible, though.

Hi Jim, Thanks for the clarification. Makes perfect sense.

Melvin

If anyone cares, we setup OSPF and removed the GW's

Ordered.. thanks

To add more to the description currently:

1. There isa VIP (Carp/LAN) setup 10.0.0.3
2. Load balancer is configured on LAN 10.0.0.3
3. Everything IS WORKING from my workstation that's vpn'ed in. IE. I can connect to the LB system at 10.0.0.3 from my vpn'ed machine

Now I just have to be able to connect to the LB IP from a machine located at say 10.0.0.4.

Interesting. I was thinking of the same thing. But i was just gonna let The Master PFsense connect to one switch with one cable, and the Slave Pfsense connect to the other switch.

Isn't carp set up such that if the connectivity of the LAN interface fails(i.ex switch failure), it will make the pfsense-slave take over? The you would just fix the switch, add it back to stack, and promote the original master again? That's how I thought i worked. But I'm asking more than telling you. I haven't bought my equipment for this project yet and ProCurve switches AND pfsense is totally new to me. hehe

Please let me know how this turns out for you. I'm interesting in hearing your experiences since this sounds similar to what i'm trying to do.

You can't sync proxy ARP VIPs between CARP nodes, it would cause an IP conflict.

If the IPs are in the same subnet as WAN, use CARP VIPs. If they are in a different subnet, have your ISP route the other subnet to your WAN CARP VIP and use the 'other' VIP type.

On 2.0 you could also add an IP alias inside the second subnet on both boxes and then setup CARP VIPs for the remaining IPs.

Yep, that's correct.

I have resolved this issue. It appears the comcast modem just needed to be rebooted.