Two FW using CARP and two ISPs, traffic routed to wrong ISP of when failover

[eskild]

Hi,
using 1.2.3-RC1, built on Wed Apr 15 21:23:36 EDT 2009.

I have two firewalls set up with CARP for redundancy. Each FW is configured with two WAN links, also using CARP.

I have tested the following:

Primary FW + ISP1 -> Traffic OK
Primary FW + ISP2 -> Traffic OK
Secondary FW + ISP1 -> Traffic OK
Secondary FW + ISP2 -> Traffic FAIL

When it fails, i see in tcpdump on secondary FW that the traffic goes out on the interface towards ISP1, and not towards ISP2, even though ISP1 is marked as Offline.

In the load balancer set up, the ISPs is set up in failover mode, with ISP1 as primary(top of the list).

Is this a bug or is it something i might have done wrong (wouldn't be the first time)

Any comments or inputs will be highly appreciated.

Thank you.

[eskild]

I tried to change the order of the ISPs in load balancer setup, and the result is as follows:
Primary FW + ISP2 -> Traffic OK
Primary FW + ISP1 -> Traffic OK
Secondary FW + ISP2 -> Traffic OK
Secondary FW + ISP1 -> Traffic FAIL

This clearly shows that the FW and ISP setup seems to be ok, and pfSense is not using the correct ISP according to the current online/offline status.

I also noticed that the LB pool status color on the primary fw is green, but yellow on the secondary fw. Is that normal? Last change is the same on both FWs.

Comments?

[Eugene]

what is your methodology of testing?
what does "Primary FW + ISP2" mean please?

[eskild]

Hi,
i'm testing redundancy between the two firewalls by enabling/disabling CARP, and testing WAN failover by blocking traffic to ISP1 gateway(blocking access in a firewall further out in the network).

Primary FW + ISP1 : Means, primary CARP member carrying traffic towards ISP1.
Primary FW + ISP2 : Means, primary CARP member carrying traffic towards ISP2, connection towards ISP1 is down.
Secondary FW + ISP1 : Means, secondary CARP member carrying traffic towards ISP1.
Secondary FW + ISP2 : Means, secondary CARP member carrying traffic towards ISP2, connection towards ISP1 is down.

I have adressed this towards premium support, and Chris Buechler has found a problem and is looking for a solution.