Traffic from LAN to OPT2 (OPTLAN) goes out wrong interface, not NATed

overand

Not sure if this is something I want in here as Multi-WAN or under NAT, but:

I have multiple LAN networks, (and multiple WAN networks for that matter), configured on a pfSense-installed 1.2.3-RELEASE box. I can't reach the OPTLAN2(or 3) network from behind LAN - though I can reach another OPT-LAN networks just fine.

Everything's using VLANs, and that part's working. This is weird enough that I'm almost wondering if I've found a bug re: "too many interfaces" behavior.

For completeness, I'll describe the interfaces and subnets, with WAN IPs munged to 1.2.3.0/24 and 5.6.7.0/24

"LAN" - 172.16.0.0/24
"WAN" - 1.2.3.0/24 (with gateway)
"SYNC"(opt1) - 192.168.254.0/24 (pfSync)
"OPTWAN"(opt2) - 5.6.7.0/24 (with gateway)
"OPTLAN1" - 172.31.0.0/24 (no gateway) - this works
"OPTLAN2" - 192.168.8.0/24 (no gateway) - this "doesn't work"
"OPTLAN3" - etc - doesn't work

Pertinent details: I have two pfSense boxes, doing CARP, and I'm using the outbound loadbalancer successfully. I have multiple virtual IPs on the WAN / OPTWAN interfaces

What works correctly:

From my workstation - 172.16.0.172, via 172.16.0.1 (CARP IP of the two pfSense boxes) - I'm able to get to the various WAN subnets, as well as route out that way as appropriate (using the CARPed VIP). I can also reach machines on the OPTLAN1 network, going out via another CARPed IP on that network.

However, I don't get NATed to the CARP IP on OPTLAN2 or OPTLAN3.

Example, via states, of what I get to OPTLAN1, working:

icmp 172.31.0.86:22882 <- 172.16.0.172 0:0
icmp 172.16.0.172:22882 -> 172.31.0.11:10061 -> 172.31.0.86 0:0

172.16.0.172 - workstation. 172.31.0.11 (CARP IP on pfSense) - 172.31.0.86 - machine i'm pinging as a test.

Here's an example of it not working, via OPTLAN2, with WAN IP munged:

icmp 192.168.8.14:17762 <- 172.16.0.172 0:0
icmp 172.16.0.172:17762 -> 5.6.7.8:8123 -> 192.168.8.14 0:0

or Workstation, CARP IP on OPTWAN, testing machine.

I have various advanced outbound NAT rules, configured functionally identically for both the working and not-working subnets.

Rule: OPTLAN1
Interface Optlan1, source 172.16.0.0/24 *, Destination 172.31.0.0/24 *, NAT address 172.31.0.11

Rule: OPTLAN2
Interface Optlan2, source 172.16.0.0/24 *, Destination 192.168.8.0/24 *, NAT address 192.168.8.49

… various rules ...

Rule: Out-WAN2
Interface Wan2, Source 172.16.0.0/24 *, Destination * *, NAT address 5.6.7.8

(That seems to be the one I'm hitting instead of the OPTLAN2 rule)

--

Here's one thing I do see: The output of pfctl -s nat includes - among a billion other things - the following lines, which seem perfectly correct (I've verified that the interfaces are as expected, that the carp and vlan interfaces match the right subnets):

nat on vlan2 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535
nat on carp6 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535

...

nat on vlan4 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535
nat on carp9 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535

...

nat on vlan0 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
nat on carp5 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
nat on carp7 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535

(carp7 is another IP - not 5.6.7.8 - on the same network - not sure why it's showing 5.6.7.8 there)

Moving around the order of my advanced outbound NAT rules doesn't seem to change anything.

So - I have outbound NAT / multi-wan working as expected for some interfaces, but not for others, with no noticable configuration differences.

There are no IPSec or OpenVPN tunnels overlapping the subnets listed here, nor static routes, nor anything else that I can figure out that would cause this.

In fairness, there are 7 interfaces here, 5 of which are on VLANs, across 2 interface cards, and 12 AON rules (mostly getting a single machine /32 to a sopecific VIP) - so it's complex enough of a setup that I may have missed something, etc.

jimp

It might help to see actual screencaps of the outbound NAT rules with the public IP blanked out.

There shouldn't be a problem with any number of interfaces, plenty of people use configurations crazier than what you have just fine.

overand

Here's the outbound NAT stuff.

WAN is 1.2.3.0/24, OPTWAN is 5.6.7.0/24

Apologies for the middle-school level GIMP usage here. =]

On the OPTLAN interfaces (actually I believe in all cases), the IPs specified are the CARP IPs.

So - goal is, get anything going from LAN (172.16.0.0/24) to OPTLANx to go out via OPTLANx's interface… yeah.

outnat.png_thumb

Eugene

It seems your nat is working well but your routing is not.
Can you give us```
netstat -rn

overand

I've removed the UHLW entries here, and am only showing Internet, not Internet6 for obvious reasons.

# netstat -rn | grep -v UHLW
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            1.2.3.1         UGS         0 13118874   fxp0
8.8.4.4            5.6.7.1       UGHS        0   428318  vlan0 =>
8.8.4.4/32         5.6.7.1       UGS         0        0  vlan0
8.8.8.8            1.2.3.1         UGHS        0   428316   fxp0
10.149.0.0/24      10.149.1.1         UGS         0        0  vlan3
10.149.1.0/24      link#11            UC          0        0  vlan3
10.149.1.252       10.149.1.252       UH          0        0  carp8
1.2.3.0/24      link#2             UC          0        0   fxp0
1.2.3.70        1.2.3.70        UH          0        0  carp3
1.2.3.83        1.2.3.83        UH          0        0  carp0
1.2.3.224       1.2.3.224       UH          0        0  carp4
1.2.3.254       1.2.3.254       UH          0        0  carp2
127.0.0.1          127.0.0.1          UH          0        0    lo0
172.16.0.0/24      link#9             UC          0        0  vlan1
172.16.0.1         172.16.0.1         UH          0        0  carp1
172.16.1.0/24      172.16.251.2       UGS         0        0   tun0
172.16.251.0/24    172.16.251.2       UGS         0        0   tun0
172.16.251.2       172.16.251.1       UH          3        0   tun0
172.31.0.0/24      link#10            UC          0        0  vlan2
172.31.0.11        172.31.0.11        UH          0        0  carp6
5.6.7.0/25    link#8             UC          0        0  vlan0
5.6.7.37      5.6.7.37      UH          0        7  carp5
5.6.7.38      5.6.7.38      UH          0        7  carp7
192.168.4.0/24     172.16.251.2       UGS         0        0   tun0
192.168.8.0/24     link#12            UC          0        0  vlan4
192.168.8.49       192.168.8.49       UH          0        0  carp9
192.168.170.0/24   link#1             UC          0        0    re0

(I munged IPs again here, and mac addersses)

You can see 192.168.8.0/24 (link #12) - UC 0 0 vlan4

Here's that interface:

# ifconfig vlan4
vlan4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	ether 00:02:a5:XX:XX:XX
	inet6 fe80::XXX:XXXX:XXXX:XXXX%vlan4 prefixlen 64 scopeid 0xc 
	inet 192.168.8.47 netmask 0xffffff00 broadcast 192.168.8.255
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	vlan: 8 parent interface: fxp0</full-duplex></up,broadcast,running,promisc,simplex,multicast>

incidentally, 10.149.0.0/24 is OPTLAN3 which works the same as OPTLAN2 (i.e. it 'doesn't) - and the 192.168.169.x network is for CARP SYNC.

Eugene

You have fxp0 configured as WAN and vlans on top of that?

overand

yes, fxp0 is "WAN" - though that interface (in native mode as fxp0) isn't being used for much.

Inbound NAT and all of the VLANs work fine, and outbound NAT to IPs on both "fxp0" and the OPTWAN interface living as tagged traffic are fine.