Traffic from LAN to OPT2 (OPTLAN) goes out wrong interface, not NATed
-
Not sure if this is something I want in here as Multi-WAN or under NAT, but:
I have multiple LAN networks, (and multiple WAN networks for that matter), configured on a pfSense-installed 1.2.3-RELEASE box. I can't reach the OPTLAN2(or 3) network from behind LAN - though I can reach another OPT-LAN networks just fine.
Everything's using VLANs, and that part's working. This is weird enough that I'm almost wondering if I've found a bug re: "too many interfaces" behavior.
For completeness, I'll describe the interfaces and subnets, with WAN IPs munged to 1.2.3.0/24 and 5.6.7.0/24
"LAN" - 172.16.0.0/24
"WAN" - 1.2.3.0/24 (with gateway)
"SYNC"(opt1) - 192.168.254.0/24 (pfSync)
"OPTWAN"(opt2) - 5.6.7.0/24 (with gateway)
"OPTLAN1" - 172.31.0.0/24 (no gateway) - this works
"OPTLAN2" - 192.168.8.0/24 (no gateway) - this "doesn't work"
"OPTLAN3" - etc - doesn't workPertinent details: I have two pfSense boxes, doing CARP, and I'm using the outbound loadbalancer successfully. I have multiple virtual IPs on the WAN / OPTWAN interfaces
What works correctly:
From my workstation - 172.16.0.172, via 172.16.0.1 (CARP IP of the two pfSense boxes) - I'm able to get to the various WAN subnets, as well as route out that way as appropriate (using the CARPed VIP). I can also reach machines on the OPTLAN1 network, going out via another CARPed IP on that network.
However, I don't get NATed to the CARP IP on OPTLAN2 or OPTLAN3.
Example, via states, of what I get to OPTLAN1, working:
icmp 172.31.0.86:22882 <- 172.16.0.172 0:0
icmp 172.16.0.172:22882 -> 172.31.0.11:10061 -> 172.31.0.86 0:0172.16.0.172 - workstation. 172.31.0.11 (CARP IP on pfSense) - 172.31.0.86 - machine i'm pinging as a test.
Here's an example of it not working, via OPTLAN2, with WAN IP munged:
icmp 192.168.8.14:17762 <- 172.16.0.172 0:0
icmp 172.16.0.172:17762 -> 5.6.7.8:8123 -> 192.168.8.14 0:0or Workstation, CARP IP on OPTWAN, testing machine.
I have various advanced outbound NAT rules, configured functionally identically for both the working and not-working subnets.
Rule: OPTLAN1
Interface Optlan1, source 172.16.0.0/24 *, Destination 172.31.0.0/24 *, NAT address 172.31.0.11Rule: OPTLAN2
Interface Optlan2, source 172.16.0.0/24 *, Destination 192.168.8.0/24 *, NAT address 192.168.8.49… various rules ...
Rule: Out-WAN2
Interface Wan2, Source 172.16.0.0/24 *, Destination * *, NAT address 5.6.7.8(That seems to be the one I'm hitting instead of the OPTLAN2 rule)
--
Here's one thing I do see: The output of pfctl -s nat includes - among a billion other things - the following lines, which seem perfectly correct (I've verified that the interfaces are as expected, that the carp and vlan interfaces match the right subnets):
nat on vlan2 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535
nat on carp6 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535...
nat on vlan4 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535
nat on carp9 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535...
nat on vlan0 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
nat on carp5 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
nat on carp7 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535(carp7 is another IP - not 5.6.7.8 - on the same network - not sure why it's showing 5.6.7.8 there)
Moving around the order of my advanced outbound NAT rules doesn't seem to change anything.
So - I have outbound NAT / multi-wan working as expected for some interfaces, but not for others, with no noticable configuration differences.
There are no IPSec or OpenVPN tunnels overlapping the subnets listed here, nor static routes, nor anything else that I can figure out that would cause this.
In fairness, there are 7 interfaces here, 5 of which are on VLANs, across 2 interface cards, and 12 AON rules (mostly getting a single machine /32 to a sopecific VIP) - so it's complex enough of a setup that I may have missed something, etc.
-
It might help to see actual screencaps of the outbound NAT rules with the public IP blanked out.
There shouldn't be a problem with any number of interfaces, plenty of people use configurations crazier than what you have just fine.
-
Here's the outbound NAT stuff.
WAN is 1.2.3.0/24, OPTWAN is 5.6.7.0/24
Apologies for the middle-school level GIMP usage here. =]
On the OPTLAN interfaces (actually I believe in all cases), the IPs specified are the CARP IPs.
So - goal is, get anything going from LAN (172.16.0.0/24) to OPTLANx to go out via OPTLANx's interface… yeah.
-
It seems your nat is working well but your routing is not.
Can you give us```
netstat -rn -
I've removed the UHLW entries here, and am only showing Internet, not Internet6 for obvious reasons.
# netstat -rn | grep -v UHLW Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 1.2.3.1 UGS 0 13118874 fxp0 8.8.4.4 5.6.7.1 UGHS 0 428318 vlan0 => 8.8.4.4/32 5.6.7.1 UGS 0 0 vlan0 8.8.8.8 1.2.3.1 UGHS 0 428316 fxp0 10.149.0.0/24 10.149.1.1 UGS 0 0 vlan3 10.149.1.0/24 link#11 UC 0 0 vlan3 10.149.1.252 10.149.1.252 UH 0 0 carp8 1.2.3.0/24 link#2 UC 0 0 fxp0 1.2.3.70 1.2.3.70 UH 0 0 carp3 1.2.3.83 1.2.3.83 UH 0 0 carp0 1.2.3.224 1.2.3.224 UH 0 0 carp4 1.2.3.254 1.2.3.254 UH 0 0 carp2 127.0.0.1 127.0.0.1 UH 0 0 lo0 172.16.0.0/24 link#9 UC 0 0 vlan1 172.16.0.1 172.16.0.1 UH 0 0 carp1 172.16.1.0/24 172.16.251.2 UGS 0 0 tun0 172.16.251.0/24 172.16.251.2 UGS 0 0 tun0 172.16.251.2 172.16.251.1 UH 3 0 tun0 172.31.0.0/24 link#10 UC 0 0 vlan2 172.31.0.11 172.31.0.11 UH 0 0 carp6 5.6.7.0/25 link#8 UC 0 0 vlan0 5.6.7.37 5.6.7.37 UH 0 7 carp5 5.6.7.38 5.6.7.38 UH 0 7 carp7 192.168.4.0/24 172.16.251.2 UGS 0 0 tun0 192.168.8.0/24 link#12 UC 0 0 vlan4 192.168.8.49 192.168.8.49 UH 0 0 carp9 192.168.170.0/24 link#1 UC 0 0 re0
(I munged IPs again here, and mac addersses)
You can see 192.168.8.0/24 (link #12) - UC 0 0 vlan4
Here's that interface:
# ifconfig vlan4 vlan4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 ether 00:02:a5:XX:XX:XX inet6 fe80::XXX:XXXX:XXXX:XXXX%vlan4 prefixlen 64 scopeid 0xc inet 192.168.8.47 netmask 0xffffff00 broadcast 192.168.8.255 media: Ethernet autoselect (100baseTX <full-duplex>) status: active vlan: 8 parent interface: fxp0</full-duplex></up,broadcast,running,promisc,simplex,multicast>
incidentally, 10.149.0.0/24 is OPTLAN3 which works the same as OPTLAN2 (i.e. it 'doesn't) - and the 192.168.169.x network is for CARP SYNC.
-
You have fxp0 configured as WAN and vlans on top of that?
-
yes, fxp0 is "WAN" - though that interface (in native mode as fxp0) isn't being used for much.
Inbound NAT and all of the VLANs work fine, and outbound NAT to IPs on both "fxp0" and the OPTWAN interface living as tagged traffic are fine.