Traffic from LAN to OPT2 (OPTLAN) goes out wrong interface, not NATed

  • Not sure if this is something I want in here as Multi-WAN or under NAT, but:

    I have multiple LAN networks, (and multiple WAN networks for that matter), configured on a pfSense-installed 1.2.3-RELEASE box.  I can't reach the OPTLAN2(or 3) network from behind LAN - though I can reach another OPT-LAN networks just fine.

    Everything's using VLANs, and that part's working.  This is weird enough that I'm almost wondering if I've found a bug re: "too many interfaces" behavior.

    For completeness, I'll describe the interfaces and subnets, with WAN IPs munged to and

    "LAN" -
    "WAN" - (with gateway)
    "SYNC"(opt1) - (pfSync)
    "OPTWAN"(opt2) - (with gateway)
    "OPTLAN1" - (no gateway) - this works
    "OPTLAN2" - (no gateway) - this "doesn't work"
    "OPTLAN3" - etc - doesn't work

    Pertinent details: I have two pfSense boxes, doing CARP, and I'm using the outbound loadbalancer successfully.  I have multiple virtual IPs on the WAN / OPTWAN interfaces

    What works correctly:

    From my workstation -, via (CARP IP of the two pfSense boxes) - I'm able to get to the various WAN subnets, as well as route out that way as appropriate (using the CARPed VIP).  I can also reach machines on the OPTLAN1 network, going out via another CARPed IP on that network.

    However, I don't get NATed to the CARP IP on OPTLAN2 or OPTLAN3.

    Example, via states, of what I get to OPTLAN1, working:

    icmp <-  0:0 
    icmp -> -> 0:0 - workstation. (CARP IP on pfSense) - - machine i'm pinging as a test.

    Here's an example of it not working, via OPTLAN2, with WAN IP munged:

    icmp <-  0:0
    icmp -> ->  0:0

    or Workstation, CARP IP on OPTWAN, testing machine.

    I have various advanced outbound NAT rules, configured functionally identically for both the working and not-working subnets.

    Rule: OPTLAN1
    Interface Optlan1, source *, Destination *, NAT address

    Rule: OPTLAN2
    Interface Optlan2, source *, Destination *, NAT address

    … various rules ...

    Rule: Out-WAN2
    Interface Wan2, Source *, Destination * *, NAT address

    (That seems to be the one I'm hitting instead of the OPTLAN2 rule)


    Here's one thing I do see:  The output of pfctl -s nat includes - among a billion other things - the following lines, which seem perfectly correct (I've verified that the interfaces are as expected, that the carp and vlan interfaces match the right subnets):

    nat on vlan2 inet from to -> port 1024:65535
    nat on carp6 inet from to -> port 1024:65535


    nat on vlan4 inet from to -> port 1024:65535
    nat on carp9 inet from to -> port 1024:65535


    nat on vlan0 inet from to any -> port 1024:65535
    nat on carp5 inet from to any -> port 1024:65535
    nat on carp7 inet from to any -> port 1024:65535

    (carp7 is another IP - not - on the same network - not sure why it's showing there)

    Moving around the order of my advanced outbound NAT rules doesn't seem to change anything.

    So - I have outbound NAT / multi-wan working as expected for some interfaces, but not for others, with no noticable configuration differences.

    There are no IPSec or OpenVPN tunnels overlapping the subnets listed here, nor static routes, nor anything else that I can figure out that would cause this.

    In fairness, there are 7 interfaces here, 5 of which are on VLANs, across 2 interface cards, and 12 AON rules (mostly getting a single machine /32 to a sopecific VIP) - so it's complex enough of a setup that I may have missed something, etc.

  • Rebel Alliance Developer Netgate

    It might help to see actual screencaps of the outbound NAT rules with the public IP blanked out.

    There shouldn't be a problem with any number of interfaces, plenty of people use configurations crazier than what you have just fine.

  • Here's the outbound NAT stuff.

    WAN is, OPTWAN is

    Apologies for the middle-school level GIMP usage here. =]

    On the OPTLAN interfaces (actually I believe in all cases), the IPs specified are the CARP IPs.

    So - goal is, get anything going from LAN ( to OPTLANx to go out via OPTLANx's interface…  yeah.

  • It seems your nat is working well but your routing is not.
    Can you give us```
    netstat -rn

  • I've removed the UHLW entries here, and am only showing Internet, not Internet6 for obvious reasons.

    # netstat -rn | grep -v UHLW
    Routing tables
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default           UGS         0 13118874   fxp0         UGHS        0   428318  vlan0 =>       UGS         0        0  vlan0           UGHS        0   428316   fxp0         UGS         0        0  vlan3      link#11            UC          0        0  vlan3       UH          0        0  carp8      link#2             UC          0        0   fxp0        UH          0        0  carp3        UH          0        0  carp0       UH          0        0  carp4       UH          0        0  carp2          UH          0        0    lo0      link#9             UC          0        0  vlan1         UH          0        0  carp1       UGS         0        0   tun0       UGS         0        0   tun0       UH          3        0   tun0      link#10            UC          0        0  vlan2        UH          0        0  carp6    link#8             UC          0        0  vlan0      UH          0        7  carp5      UH          0        7  carp7       UGS         0        0   tun0     link#12            UC          0        0  vlan4       UH          0        0  carp9   link#1             UC          0        0    re0

    (I munged IPs again here, and mac addersses)

    You can see (link #12) - UC 0 0 vlan4

    Here's that interface:

    # ifconfig vlan4
    vlan4: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	ether 00:02:a5:XX:XX:XX
    	inet6 fe80::XXX:XXXX:XXXX:XXXX%vlan4 prefixlen 64 scopeid 0xc 
    	inet netmask 0xffffff00 broadcast
    	media: Ethernet autoselect (100baseTX <full-duplex>)
    	status: active
    	vlan: 8 parent interface: fxp0</full-duplex></up,broadcast,running,promisc,simplex,multicast> 

    incidentally, is OPTLAN3 which works the same as OPTLAN2 (i.e. it 'doesn't) - and the 192.168.169.x network is for CARP SYNC.

  • You have fxp0 configured as WAN and vlans on top of that?

  • yes, fxp0 is "WAN" - though that interface (in native mode as fxp0) isn't being used for much.

    Inbound NAT and all of the VLANs work fine, and outbound NAT to IPs on both "fxp0" and the OPTWAN interface living as tagged traffic are fine.

Log in to reply