Multi-WAN & Gateway Group Issues

Rhongomiant

pfSense Build Info:

2.1-BETA1 (amd64)
built on Sat Feb 16 10:55:42 EST 2013
FreeBSD 8.3-RELEASE-p5

I now have two WANs, I have setup gateway groups and applied them to rules and I am having some issues.

I have a rule on the LAN interface to allow all "LAN net" traffic through. After setting up a gateway group I applied it to this rule. Now I cannot access any internal IP on a different subnet behind the firewall. I can access the IPs on the firewall for the interface. When running an mtr/traceroute to yahoo.com I see a difference in the first hop. With default set for the gateway group the first hop is the local network gateway and the second hop is the WAN interface gateway. With the gateway group set the first hop is the WAN interface. I have mtr output below showing the difference.

With Default Set for Gateway Group:
My traceroute [v0.75]
andLinux (0.0.0.0) Mon Feb 18 04:40:46 2013
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.1.1 0.0% 4 0.0 5.0 0.0 10.0 5.8
2. 96.211.11.1 0.0% 4 10.0 10.0 10.0 10.0 0.0
3. 130.81.133.128 0.0% 4 10.0 10.0 10.0 10.0 0.0
4. 130.81.199.28 0.0% 4 10.0 30.0 10.0 90.0 40.0
5. 130.81.17.38 0.0% 4 100.0 95.0 90.0 100.0 5.8

With Gateway Group Applied:
My traceroute [v0.75]
andLinux (0.0.0.0) Mon Feb 18 04:38:29 2013
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 96.211.11.1 0.0% 6 0.0 5.0 0.0 10.0 5.5
2. 130.81.133.128 0.0% 6 10.0 16.7 10.0 30.0 10.3
3. 130.81.199.28 0.0% 6 10.0 8.3 0.0 10.0 4.1
4. 130.81.17.38 0.0% 5 90.0 92.0 90.0 100.0 4.5

I am not sure of this is the way it is supposed to work. If it is, why would the gateway part only apply when it is needed. This creates a fair amount of time consuming work with the current options available in the source and destination lists in the web interface as users will need to create unique a rule to allow traffic from one LAN interface to every other LAN interface. So it looks like using gateway groups adds much more work than simply assigning a gateway group to an existing rule.

Update: Based on the messages from athurdent and phil.davis and my own research, it does look like the result that I am seeing is expected. However, looking at rules via pfctl, I see that you are doing something cool in that pfSense have a table called negate_networks and when a rule is set to use a different gateway, pfSense creates two rules. The first rule routes traffic to IPs in the negate_networks table using the default route table and the following rule routes everything else through the selected gateway or gateway group. This brings up some questions.

1a) The negate_networks table does not have my PPTP server subnet. Is there a reason for this?

1b) Are you able to create a table of internal networks and create a rule using the default route table for those networks when a gateway or gateway group is selected for a rule?

It seemed like my IPv6, provided through a gif tunnel to Hurricane Electric, was broken. However, I found that to get to internet sites passed the HE gateway, like ipv6.google.com, I had to set the gateway in my IPv6 rules to the IPv6 gateway rather than default. I do not know if this breaks access to internal IPv6 IPs on other internal LANs as I do not have active IPv6 devices on those LANs, but since the first hop in an MTR is the HE gateway, I am guessing it does. I am not sure if having multiple IPv4 gateways cause this or if having gateway groups applied to my IPv4 rules created this issue. My IPv6 rules do come after my IPv4 rules and yes, I have the protocol set to IPv4 or IPv6 in all my rules.
I am going to be setting up a secondary firewall for redundancy and have some questions on how gif can be setup for this.

3a) How do I configure the gif tunnel to HE in a redundant pfSense firewall environment?

3b) Is there anyway for me to configure the gif HE tunnel to failover to the secondary WAN interface if the primary WAN interface fails? I assume that there is no way for me to do this since the gif settings require that the outbound interface be assigned. However maybe there is a more complicated way to accomplish this result?

athurdent

Regarding 1) I think, you will have to write a normal pass rule before those rules with gateway group set, allowing traffic to your other internal nets connected to the firewall. The rules with gateway option set always have a route-to statement in them, so pf will route them out on your internet gateway(s) no matter what.

phil.davis

Regarding 1) - I leave the default "pass all" rule on LAN as it is at the bottom. I make an alias for all my internal subnets - called "InternalSubnets". Then put a policy-routing rule above the "pass all" rule, that has source any, destination !InternalSubnets, gateway = desired gateway group.
This directs all traffic that is not internal, into the gateway group, which then works out how to get it out a WAN that is up…
Internal traffic falls through to the "pass all" rule, and gets routed by the ordinary routing table.