Traffic from LAN to OPT2 (OPTLAN) goes out wrong interface, not NATed



  • Not sure if this is something I want in here as Multi-WAN or under NAT, but:

    I have multiple LAN networks, (and multiple WAN networks for that matter), configured on a pfSense-installed 1.2.3-RELEASE box.  I can't reach the OPTLAN2(or 3) network from behind LAN - though I can reach another OPT-LAN networks just fine.

    Everything's using VLANs, and that part's working.  This is weird enough that I'm almost wondering if I've found a bug re: "too many interfaces" behavior.

    For completeness, I'll describe the interfaces and subnets, with WAN IPs munged to 1.2.3.0/24 and 5.6.7.0/24

    "LAN" - 172.16.0.0/24
    "WAN" - 1.2.3.0/24 (with gateway)
    "SYNC"(opt1) - 192.168.254.0/24 (pfSync)
    "OPTWAN"(opt2) - 5.6.7.0/24 (with gateway)
    "OPTLAN1" - 172.31.0.0/24 (no gateway) - this works
    "OPTLAN2" - 192.168.8.0/24 (no gateway) - this "doesn't work"
    "OPTLAN3" - etc - doesn't work

    Pertinent details: I have two pfSense boxes, doing CARP, and I'm using the outbound loadbalancer successfully.  I have multiple virtual IPs on the WAN / OPTWAN interfaces

    What works correctly:

    From my workstation - 172.16.0.172, via 172.16.0.1 (CARP IP of the two pfSense boxes) - I'm able to get to the various WAN subnets, as well as route out that way as appropriate (using the CARPed VIP).  I can also reach machines on the OPTLAN1 network, going out via another CARPed IP on that network.

    However, I don't get NATed to the CARP IP on OPTLAN2 or OPTLAN3.

    Example, via states, of what I get to OPTLAN1, working:

    icmp  172.31.0.86:22882 <- 172.16.0.172  0:0 
    icmp 172.16.0.172:22882 -> 172.31.0.11:10061 -> 172.31.0.86 0:0

    172.16.0.172 - workstation.  172.31.0.11 (CARP IP on pfSense) - 172.31.0.86 - machine i'm pinging as a test.

    Here's an example of it not working, via OPTLAN2, with WAN IP munged:

    icmp  192.168.8.14:17762 <- 172.16.0.172  0:0
    icmp  172.16.0.172:17762 -> 5.6.7.8:8123 -> 192.168.8.14  0:0

    or Workstation, CARP IP on OPTWAN, testing machine.

    I have various advanced outbound NAT rules, configured functionally identically for both the working and not-working subnets.

    Rule: OPTLAN1
    Interface Optlan1, source 172.16.0.0/24 *, Destination 172.31.0.0/24 *, NAT address 172.31.0.11

    Rule: OPTLAN2
    Interface Optlan2, source 172.16.0.0/24 *, Destination 192.168.8.0/24 *, NAT address 192.168.8.49

    … various rules ...

    Rule: Out-WAN2
    Interface Wan2, Source 172.16.0.0/24 *, Destination * *, NAT address 5.6.7.8

    (That seems to be the one I'm hitting instead of the OPTLAN2 rule)

    --

    Here's one thing I do see:  The output of pfctl -s nat includes - among a billion other things - the following lines, which seem perfectly correct (I've verified that the interfaces are as expected, that the carp and vlan interfaces match the right subnets):

    nat on vlan2 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535
    nat on carp6 inet from 172.16.0.0/24 to 172.31.0.0/24 -> 172.31.0.11 port 1024:65535

    ...

    nat on vlan4 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535
    nat on carp9 inet from 172.16.0.0/24 to 192.168.8.0/24 -> 192.168.8.49 port 1024:65535

    ...

    nat on vlan0 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
    nat on carp5 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535
    nat on carp7 inet from 172.16.0.0/24 to any -> 5.6.7.8 port 1024:65535

    (carp7 is another IP - not 5.6.7.8 - on the same network - not sure why it's showing 5.6.7.8 there)

    Moving around the order of my advanced outbound NAT rules doesn't seem to change anything.

    So - I have outbound NAT / multi-wan working as expected for some interfaces, but not for others, with no noticable configuration differences.

    There are no IPSec or OpenVPN tunnels overlapping the subnets listed here, nor static routes, nor anything else that I can figure out that would cause this.

    In fairness, there are 7 interfaces here, 5 of which are on VLANs, across 2 interface cards, and 12 AON rules (mostly getting a single machine /32 to a sopecific VIP) - so it's complex enough of a setup that I may have missed something, etc.


  • Rebel Alliance Developer Netgate

    It might help to see actual screencaps of the outbound NAT rules with the public IP blanked out.

    There shouldn't be a problem with any number of interfaces, plenty of people use configurations crazier than what you have just fine.



  • Here's the outbound NAT stuff.

    WAN is 1.2.3.0/24, OPTWAN is 5.6.7.0/24

    Apologies for the middle-school level GIMP usage here. =]

    On the OPTLAN interfaces (actually I believe in all cases), the IPs specified are the CARP IPs.

    So - goal is, get anything going from LAN (172.16.0.0/24) to OPTLANx to go out via OPTLANx's interface…  yeah.




  • It seems your nat is working well but your routing is not.
    Can you give us```
    netstat -rn



  • I've removed the UHLW entries here, and am only showing Internet, not Internet6 for obvious reasons.

    # netstat -rn | grep -v UHLW
    Routing tables
    
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            1.2.3.1         UGS         0 13118874   fxp0
    8.8.4.4            5.6.7.1       UGHS        0   428318  vlan0 =>
    8.8.4.4/32         5.6.7.1       UGS         0        0  vlan0
    8.8.8.8            1.2.3.1         UGHS        0   428316   fxp0
    10.149.0.0/24      10.149.1.1         UGS         0        0  vlan3
    10.149.1.0/24      link#11            UC          0        0  vlan3
    10.149.1.252       10.149.1.252       UH          0        0  carp8
    1.2.3.0/24      link#2             UC          0        0   fxp0
    1.2.3.70        1.2.3.70        UH          0        0  carp3
    1.2.3.83        1.2.3.83        UH          0        0  carp0
    1.2.3.224       1.2.3.224       UH          0        0  carp4
    1.2.3.254       1.2.3.254       UH          0        0  carp2
    127.0.0.1          127.0.0.1          UH          0        0    lo0
    172.16.0.0/24      link#9             UC          0        0  vlan1
    172.16.0.1         172.16.0.1         UH          0        0  carp1
    172.16.1.0/24      172.16.251.2       UGS         0        0   tun0
    172.16.251.0/24    172.16.251.2       UGS         0        0   tun0
    172.16.251.2       172.16.251.1       UH          3        0   tun0
    172.31.0.0/24      link#10            UC          0        0  vlan2
    172.31.0.11        172.31.0.11        UH          0        0  carp6
    5.6.7.0/25    link#8             UC          0        0  vlan0
    5.6.7.37      5.6.7.37      UH          0        7  carp5
    5.6.7.38      5.6.7.38      UH          0        7  carp7
    192.168.4.0/24     172.16.251.2       UGS         0        0   tun0
    192.168.8.0/24     link#12            UC          0        0  vlan4
    192.168.8.49       192.168.8.49       UH          0        0  carp9
    192.168.170.0/24   link#1             UC          0        0    re0
    
    

    (I munged IPs again here, and mac addersses)

    You can see 192.168.8.0/24 (link #12) - UC 0 0 vlan4

    Here's that interface:

    # ifconfig vlan4
    vlan4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	ether 00:02:a5:XX:XX:XX
    	inet6 fe80::XXX:XXXX:XXXX:XXXX%vlan4 prefixlen 64 scopeid 0xc 
    	inet 192.168.8.47 netmask 0xffffff00 broadcast 192.168.8.255
    	media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    	vlan: 8 parent interface: fxp0</full-duplex></up,broadcast,running,promisc,simplex,multicast> 
    

    incidentally, 10.149.0.0/24 is OPTLAN3 which works the same as OPTLAN2 (i.e. it 'doesn't) - and the 192.168.169.x network is for CARP SYNC.



  • You have fxp0 configured as WAN and vlans on top of that?



  • yes, fxp0 is "WAN" - though that interface (in native mode as fxp0) isn't being used for much.

    Inbound NAT and all of the VLANs work fine, and outbound NAT to IPs on both "fxp0" and the OPTWAN interface living as tagged traffic are fine.


Locked