Policybased routing not working properly with failover

[Veni]

Running 2.0-RC1 (i386) built on Fri Apr 15 14:16:25 EDT 2011 from CD-ROM.
Noticed the same problem in 2.0-RC1-i386-20110408-1805 but have not had the time until the weekends to narrow down the problem.

The problem:
One LAN computer bypasses the firewall rules which implements a specific gateway(GW_WAN) for that computer.
The reason for this rule is that one computer on the LAN generates allot of traffic and i don't wan't it to access
the internet if running on any of the failover links. That computer should only use the WAN link, and if the WAN link
dies then it should not have any access to the internet until WAN is restored.

Testing method:

• Method 1: Pulling the WAN cable:

• After the apinger notices that the WAN link is dead, all computers access the WAN through ADSL(OPT3), except for GODZILLA, it starts to use 3G(OPT2)(not right :P).

• Method 2: Blocking traffic to pfSense WAN interface without pulling the WAN cable(changed VLAN config on the port in the switch to a VLAN with no access to anything, it's basically alone).

• 1. After the apinger notices that the WAN link is dead, GODZILLA gets no access to internet**(It works :))**. All other computers use the ADSL(OPT3) link.
• 2. Pulling the telephone cable to the ADSL modem(only one WAN left, 3G), all computers access the WAN through 3G(OPT2), even GODZILLA(not right :P).

The setup:

• Primary WAN(WAN), Ethernet running through a switch, 100/100 Mbps(all computers allowed this way).
• Secondary WAN(OPT3), ADSL modem, 6/1 Mbps(GODZILLA computer excluded due to the amount of traffic it generates).
• Tertiary WAN(OPT2), 3G, Sony Ericsson W902 through USB, 1.5/0.2 Mbps(GODZILLA computer excluded due to the amount of traffic it generates).

The minimal firewall rules(LAN) description:
A have 3 LAN rules that allow/block traffic to the internet:

• One(top) that allows GODZILLA(the machine that generates allot of traffic) to only use GW_WAN(default WAN 100/100 Mbps).
• One(middle) that blocks GODZILLA.
• One that allows all other LAN traffic to the internet through the Failover gateway group.

OPT3 and OPT2 rules:

• Destination address 192.168.11.248(GODZILLA) is blocked on any protocol from any source address.
  
  
  
  

[gerdesj]

I think I've worked my way around your config.

As a test, could you try changing your last rule to "source not 192.168.11.248" instead of "source LAN net".  Godzilla should then hit the default deny rule.

Also, do you have any floating rules?

I don't think you need any rules on your OPT interfaces that refer to Godzilla unless you are allowing all other inbound traffic, in which case you don't have a firewall!

Cheers
Jon

[Veni]

@gerdesj:

I think I've worked my way around your config.

As a test, could you try changing your last rule to "source not 192.168.11.248" instead of "source LAN net".  Godzilla should then hit the default deny rule.

Will do. Will try tomorrow or durring the weekend. Thanks for the tip.

@gerdesj:

Also, do you have any floating rules?

None.

@gerdesj:

I don't think you need any rules on your OPT interfaces that refer to Godzilla unless you are allowing all other inbound traffic, in which case you don't have a firewall!

I agree with you.

[Veni]

Well, i have some good news and some bad news.

The good news:
With testing method number 2 everything works as it should. GODZILLA is blocked from OPT2 and OPT3 while all other computers can use the failover group. Perfect :).
A little bit strange, was not expecting it to work because a firewall rule higher up should block GODZILLA from the failover group. Strange, but still good.

The bad news:
With testing method number 1 i have the same problem. GODZILLA gets access to the Internet through OPT2(3G, the last backup) after WAN fails. All other computers start to use OPT3(ADSL).
When i kill the ADSL modem every other computer starts to use 3G, and sadly, even GODZILLA is getting access to the Internet through that connection ???. It should not get any access after WAN stops working.