Traffic from LAN to OPT2 (OPTLAN) goes out wrong interface, not NATed
Not sure if this is something I want in here as Multi-WAN or under NAT, but:
I have multiple LAN networks, (and multiple WAN networks for that matter), configured on a pfSense-installed 1.2.3-RELEASE box. I can't reach the OPTLAN2(or 3) network from behind LAN - though I can reach another OPT-LAN networks just fine.
Everything's using VLANs, and that part's working. This is weird enough that I'm almost wondering if I've found a bug re: "too many interfaces" behavior.
For completeness, I'll describe the interfaces and subnets, with WAN IPs munged to 22.214.171.124/24 and 126.96.36.199/24
"LAN" - 172.16.0.0/24
"WAN" - 188.8.131.52/24 (with gateway)
"SYNC"(opt1) - 192.168.254.0/24 (pfSync)
"OPTWAN"(opt2) - 184.108.40.206/24 (with gateway)
"OPTLAN1" - 172.31.0.0/24 (no gateway) - this works
"OPTLAN2" - 192.168.8.0/24 (no gateway) - this "doesn't work"
"OPTLAN3" - etc - doesn't work
Pertinent details: I have two pfSense boxes, doing CARP, and I'm using the outbound loadbalancer successfully. I have multiple virtual IPs on the WAN / OPTWAN interfaces
What works correctly:
From my workstation - 172.16.0.172, via 172.16.0.1 (CARP IP of the two pfSense boxes) - I'm able to get to the various WAN subnets, as well as route out that way as appropriate (using the CARPed VIP). I can also reach machines on the OPTLAN1 network, going out via another CARPed IP on that network.
However, I don't get NATed to the CARP IP on OPTLAN2 or OPTLAN3.
Example, via states, of what I get to OPTLAN1, working:
icmp 172.31.0.86:22882 <- 172.16.0.172 0:0
icmp 172.16.0.172:22882 -> 172.31.0.11:10061 -> 172.31.0.86 0:0
172.16.0.172 - workstation. 172.31.0.11 (CARP IP on pfSense) - 172.31.0.86 - machine i'm pinging as a test.
Here's an example of it not working, via OPTLAN2, with WAN IP munged:
icmp 192.168.8.14:17762 <- 172.16.0.172 0:0
icmp 172.16.0.172:17762 -> 220.127.116.11:8123 -> 192.168.8.14 0:0
or Workstation, CARP IP on OPTWAN, testing machine.
I have various advanced outbound NAT rules, configured functionally identically for both the working and not-working subnets.
Interface Optlan1, source 172.16.0.0/24 *, Destination 172.31.0.0/24 *, NAT address 172.31.0.11
Interface Optlan2, source 172.16.0.0/24 *, Destination 192.168.8.0/24 *, NAT address 192.168.8.49
… various rules ...
Interface Wan2, Source 172.16.0.0/24 *, Destination * *, NAT address 18.104.22.168
(That seems to be the one I'm hitting instead of the OPTLAN2 rule)
Here's one thing I do see: The output of pfctl -s nat includes - among a billion other things - the following lines, which seem perfectly correct (I've verified that the interfaces are as expected, that the carp and vlan interfaces match the right subnets):
nat on vlan2 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535
nat on carp6 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535
nat on vlan4 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535
nat on carp9 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535
nat on vlan0 inet from 172.16.0.0/24 to any -> 22.214.171.124 port 1024:65535
nat on carp5 inet from 172.16.0.0/24 to any -> 126.96.36.199 port 1024:65535
nat on carp7 inet from 172.16.0.0/24 to any -> 188.8.131.52 port 1024:65535
(carp7 is another IP - not 184.108.40.206 - on the same network - not sure why it's showing 220.127.116.11 there)
Moving around the order of my advanced outbound NAT rules doesn't seem to change anything.
So - I have outbound NAT / multi-wan working as expected for some interfaces, but not for others, with no noticable configuration differences.
There are no IPSec or OpenVPN tunnels overlapping the subnets listed here, nor static routes, nor anything else that I can figure out that would cause this.
In fairness, there are 7 interfaces here, 5 of which are on VLANs, across 2 interface cards, and 12 AON rules (mostly getting a single machine /32 to a sopecific VIP) - so it's complex enough of a setup that I may have missed something, etc.
It might help to see actual screencaps of the outbound NAT rules with the public IP blanked out.
There shouldn't be a problem with any number of interfaces, plenty of people use configurations crazier than what you have just fine.
Here's the outbound NAT stuff.
WAN is 18.104.22.168/24, OPTWAN is 22.214.171.124/24
Apologies for the middle-school level GIMP usage here. =]
On the OPTLAN interfaces (actually I believe in all cases), the IPs specified are the CARP IPs.
So - goal is, get anything going from LAN (172.16.0.0/24) to OPTLANx to go out via OPTLANx's interface… yeah.
It seems your nat is working well but your routing is not.
Can you give us```
I've removed the UHLW entries here, and am only showing Internet, not Internet6 for obvious reasons.
# netstat -rn | grep -v UHLW Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 126.96.36.199 UGS 0 13118874 fxp0 188.8.131.52 184.108.40.206 UGHS 0 428318 vlan0 => 220.127.116.11/32 18.104.22.168 UGS 0 0 vlan0 22.214.171.124 126.96.36.199 UGHS 0 428316 fxp0 10.149.0.0/24 10.149.1.1 UGS 0 0 vlan3 10.149.1.0/24 link#11 UC 0 0 vlan3 10.149.1.252 10.149.1.252 UH 0 0 carp8 188.8.131.52/24 link#2 UC 0 0 fxp0 184.108.40.206 220.127.116.11 UH 0 0 carp3 18.104.22.168 22.214.171.124 UH 0 0 carp0 126.96.36.199 188.8.131.52 UH 0 0 carp4 184.108.40.206 220.127.116.11 UH 0 0 carp2 127.0.0.1 127.0.0.1 UH 0 0 lo0 172.16.0.0/24 link#9 UC 0 0 vlan1 172.16.0.1 172.16.0.1 UH 0 0 carp1 172.16.1.0/24 172.16.251.2 UGS 0 0 tun0 172.16.251.0/24 172.16.251.2 UGS 0 0 tun0 172.16.251.2 172.16.251.1 UH 3 0 tun0 172.31.0.0/24 link#10 UC 0 0 vlan2 172.31.0.11 172.31.0.11 UH 0 0 carp6 18.104.22.168/25 link#8 UC 0 0 vlan0 22.214.171.124 126.96.36.199 UH 0 7 carp5 188.8.131.52 184.108.40.206 UH 0 7 carp7 192.168.4.0/24 172.16.251.2 UGS 0 0 tun0 192.168.8.0/24 link#12 UC 0 0 vlan4 192.168.8.49 192.168.8.49 UH 0 0 carp9 192.168.170.0/24 link#1 UC 0 0 re0
(I munged IPs again here, and mac addersses)
You can see 192.168.8.0/24 (link #12) - UC 0 0 vlan4
Here's that interface:
# ifconfig vlan4 vlan4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 ether 00:02:a5:XX:XX:XX inet6 fe80::XXX:XXXX:XXXX:XXXX%vlan4 prefixlen 64 scopeid 0xc inet 192.168.8.47 netmask 0xffffff00 broadcast 192.168.8.255 media: Ethernet autoselect (100baseTX <full-duplex>) status: active vlan: 8 parent interface: fxp0</full-duplex></up,broadcast,running,promisc,simplex,multicast>
incidentally, 10.149.0.0/24 is OPTLAN3 which works the same as OPTLAN2 (i.e. it 'doesn't) - and the 192.168.169.x network is for CARP SYNC.
You have fxp0 configured as WAN and vlans on top of that?
yes, fxp0 is "WAN" - though that interface (in native mode as fxp0) isn't being used for much.
Inbound NAT and all of the VLANs work fine, and outbound NAT to IPs on both "fxp0" and the OPTWAN interface living as tagged traffic are fine.