NAT and gateway group policy route and clients all on 1 interface

phil.davis

This is a rather unusual setup. The issue is the same on 2.1-RELEASE and 2.1.1-prerelease (Feb 2).
LAN IP 10.49.160.250/24 - LAN works fine out WAN and OPT1 with gateway groups and failover…
WAN IP 10.49.175.1/24 gateway 10.49.175.250 (ADSL router to internet, plus has wireless AP built-in)
OPT1 IP 10.49.174.1/24 gateway 10.49.174.250 (TP-Link device with 3G dongle that reaches internet via "Ncell" 3G mobile network)
Gateway group (called it NcellP) Tier1 WAN, Tier2 OPT1.
Rule on LAN to pass traffic to destination all to GWG Failover
Manual Outbound NAT - existing default rules on LAN (works fine), extra rules to NAT from the WAN subnet to WANaddress and OPT1address when going out those interfaces.

Scenario 1 - working
Rule on WAN to pass all traffic, no gateway group specified (i.e. take the default route)

Clients connect to the ADSL device wireless and get a DHCP from pfSense WAN, pfSense WAN is their gateway and DNS.

/tmp/rules.debug has these things related to getting the clients on WAN back out WAN to the ADSL router and internet:

nat on $WAN  from 10.49.175.0/24 to any -> 10.49.175.1/32 port 1024:65535  
nat on $OPT1  from 10.49.175.0/24 to any -> 10.49.174.1/32 port 1024:65535  

pass  in  quick  on $WAN reply-to ( vr1 10.49.175.250 ) inet from 10.49.175.0/24 to any keep state  label "USER_RULE: Pass all incoming on WAN"

When doing a ping from a client on WAN, it looks like this:

12:08:34.433769 IP 10.49.175.32 > 64.22.81.44: ICMP echo request, id 3, seq 23457, length 40
12:08:34.433851 IP 10.49.175.1 > 64.22.81.44: ICMP echo request, id 62249, seq 23457, length 40
12:08:34.790019 IP 64.22.81.44 > 10.49.175.1: ICMP echo reply, id 62249, seq 23457, length 40
12:08:34.790094 IP 64.22.81.44 > 10.49.175.32: ICMP echo reply, id 3, seq 23457, length 40

The packet arrives from .32, NAT changes .32 to .1 and it is sent out. The reply comes back to .1, NAT undoes it and delivers it back to .32

This is all really good and allows clients to use the wireless AP already on the ADSL router (no need for a separate wireless device on LAN behind pfSense).

Scenario 2 - not working
Now make use of the gateway group to give failover, change WAN pass rule to:
Rule on WAN to pass traffic to destination all to GWG NcellP

/tmp/rules.debug now has these things related to getting the clients on WAN back out WAN to the ADSL router and internet:

nat on $WAN  from 10.49.175.0/24 to any -> 10.49.175.1/32 port 1024:65535  
nat on $OPT1  from 10.49.175.0/24 to any -> 10.49.174.1/32 port 1024:65535  

table <negate_networks>{ 10.49.80.0/20 10.49.224.0/20 10.50.80.0/20 10.51.80.0/20 10.49.235.0/24 10.49.0.0/16 10.50.0.0/20 10.50.32.0/20 10.51.0.0/16 10.49.251.0/24 10.50.160.0/24 }

GWNcellP = "  route-to { ( vr1 10.49.175.250 )  }  "

pass  in  quick  on $WAN inet from 10.49.175.0/24  to <negate_networks>keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $WAN  $GWNcellP inet from 10.49.175.0/24 to any keep state  label "USER_RULE: Push all traffic from private WAN out through NcellP"</negate_networks></negate_networks>

Now when doing a ping from a client on WAN, it looks like this:

11:37:44.901637 IP 10.49.175.32 > 64.22.81.44: ICMP echo request, id 3, seq 22710, length 40
11:37:44.901746 IP 10.49.175.32 > 64.22.81.44: ICMP echo request, id 3, seq 22710, length 40

There is no NAT applied, and the packet is sent "unharmed" straight out to the ADSL device and internet. On return, the ADSL device delivers it directly to the client on its wireless AP, because the ADSL device knows how to deliver directly to 10.49.175.32 - there is an asymmetric route. This works for "ping". But when downloading a file, the state gets created in pfSense but the reply traffic is not seen, so the state soon dies and so does the client download.

This seems to be the problem - NAT is not getting applied in this scenario.

Scenario 3 - working
Next I pull the ADSL phone cable, to make the ADSL line go down, but still have the wireless client working through the AP on WAN. pfSense fails over the traffic to OPT1.
/tmp/rules.debug now has these things related to getting the clients on WAN out to the internet via OPT1:

table <negate_networks>{ 10.49.80.0/20 10.49.224.0/20 10.50.80.0/20 10.51.80.0/20 10.49.235.0/24 10.49.0.0/16 10.50.0.0/20 10.50.32.0/20 10.51.0.0/16 10.49.251.0/24 10.50.160.0/24 }

GWNcellP = "  route-to { ( vr2 10.49.174.250 )  }  "

nat on $WAN  from 10.49.175.0/24 to any -> 10.49.175.1/32 port 1024:65535  
nat on $OPT1  from 10.49.175.0/24 to any -> 10.49.174.1/32 port 1024:65535  

pass  in  quick  on $WAN inet from 10.49.175.0/24  to <negate_networks>keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $WAN  $GWNcellP inet from 10.49.175.0/24 to any keep state  label "USER_RULE: Push all traffic from private WAN out through NcellP"</negate_networks></negate_networks>

Now when doing a ping from a client on WAN, it looks like this:

WAN packets:
11:53:40.687187 IP 10.49.175.32 > 64.22.81.44: ICMP echo request, id 3, seq 22854, length 40
11:53:41.003139 IP 64.22.81.44 > 10.49.175.32: ICMP echo reply, id 3, seq 22854, length 40
11:53:41.708829 IP 10.49.175.32 > 64.22.81.44: ICMP echo request, id 3, seq 22855, length 40
11:53:42.023387 IP 64.22.81.44 > 10.49.175.32: ICMP echo reply, id 3, seq 22855, length 40

OPT1 packets:
11:52:54.025358 IP 10.49.174.1 > 64.22.81.44: ICMP echo request, id 3736, seq 22808, length 40
11:52:54.360641 IP 64.22.81.44 > 10.49.174.1: ICMP echo reply, id 3736, seq 22808, length 40
11:52:55.040753 IP 10.49.174.1 > 64.22.81.44: ICMP echo request, id 3736, seq 22809, length 40
11:52:55.350019 IP 64.22.81.44 > 10.49.174.1: ICMP echo reply, id 3736, seq 22809, length 40

All is good - packets come and go from the WAN client at 10.49.175.32 and come and go from the OPT1 interface translated as 10.49.174.1
No asymmetric routing. The client on WAN can happily download, traffic goes client<->pfSenseWAN<->pfSenseOPT1<->3G-router<->internet and all is well.

Summary
It seems that if the pf rule set has a NAT rule to apply on traffic leaving WAN and also a policy-routing pass and 'route-to' rule that forces the traffic back out WAN, then the NAT is not being done to the packet/s.

I looked at the pf man page, but I can't see where this is prohibited or what other things might be able to be added to the NAT and/or pass rules to make both NAT and route-to happen???
zzz-extra-NAT-rules.png_thumb

Sgala

did you found a solution?