Mutiple VLAN -> NAT -> Multi VIP static IP WAN.. questions…



  • I'm trying to configure pfSense with 8 VLAN'ed LAN interfaces (10.0.xxx.0/24 subnets, trunked through a single interface, sis2), each pointing to a separate public WAN ip (from a /28 block). I've configured all 8 interfaces identically wrt each VLAN, and each interface will successfully dish out a DHCP address and will allow traffic on the LAN side, but 4 (specifically the last 4) interfaces will not route/NAT traffic out to the WAN side.

    I'm not seeing anything out of the ordinary when I compare one of the working interfaces (e.g. VLAN210) with one that doesnt (e.g. VLAN215), with the exception of only one thing - I noticed on the serial console that the 4 I'm having trouble with do not have asterisks next to their interface names:

    *** Welcome to pfSense 1.2.2-embedded on redfw01 ***

    WAN*****                    ->  sis6    ->      wan.xxx.xxx.210
      LAN*****                    ->  vlan0  ->      10.0.210.254
      OPT1(VLAN211)*****          ->  vlan1  ->      10.0.211.254
      OPT2(VLAN212)*****          ->  vlan2  ->      10.0.212.254
      OPT3(VLAN213)*****          ->  vlan3  ->      10.0.213.254
      OPT4(VLAN214)            ->  vlan4  ->      10.0.214.254
      OPT5(VLAN215)            ->  vlan5  ->      10.0.215.254
      OPT6(VLAN216)            ->  vlan6  ->      10.0.216.254
      OPT7(VLAN220_GUESTNET)  ->  vlan7  ->      10.0.220.254

    I looked around for documentation on what these asterisks indicate, but alas, I cannot find any; I'm not entirely sure if this has anything remotely to do with the problem that I'm having.. The "vlan0 - vlan7" noted above I think are just internal markers, and are not actually parsed as the actual VLAN tag numbers; here's the VLAN info from the GUI interfaces tab:

    Interface  VLAN tag  Description
    sis2  210  VLAN210_MGMT     
    sis2 211 VLAN211 
    sis2 212 VLAN212 
    sis2 213 VLAN213 
    sis2 214 VLAN214 
    sis2 215 VLAN215 
    sis2 216 VLAN216 
    sis2 220 VLAN220_GUESTNET

    Since I can connect and communicate from a host client IP on each of the 8 VLAN's to the router, I know that my internal trunking and VLAN's are working correctly, and that I suspect the issue lies either in NAT or Firewall rules.

    Here is the list of the outbound NAT table:

    Interface  Source  Source Port  Destination  Destination Port  NAT Address  NAT Port  Static Port  Description
    WAN    10.0.0.0/24  *  *  *  *  *  NO Auto created rule for LAN 
    WAN  10.0.210.0/24 * * * wan.xxx.xxx.210 * NO VLAN210–>WAN210 
    WAN  10.0.211.0/24 * * * wan.xxx.xxx.211 * NO VLAN211-->WAN211
    WAN  10.0.212.0/24 * * * wan.xxx.xxx.212 * NO VLAN212-->WAN212 
    WAN  10.0.213.0/24 * * * wan.xxx.xxx.213 * NO VLAN213-->WAN213 
    WAN  10.0.214.0/24 * * * wan.xxx.xxx.214 * NO VLAN214-->WAN214 
    WAN  10.0.215.0/24 * * * wan.xxx.xxx.215 * NO VLAN215-->WAN215 
    WAN  10.0.216.0/24 * * * wan.xxx.xxx.216 * NO VLAN216-->WAN216 
    WAN  10.0.220.0/24 * * * wan.xxx.xxx.220 * NO VLAN220_GUESTNET-->WAN220

    Here is the VIP list:

    Virtual IP address  Type  Description
    wan.xxx.xxx.210/32  [Proxy ARP]  210WANIP 
    wan.xxx.xxx.211/32 [Proxy ARP] 211WANIP 
    wan.xxx.xxx.212/32 [Proxy ARP] 212WANIP 
    wan.xxx.xxx.213/32 [Proxy ARP] 213WANIP 
    wan.xxx.xxx.214/32 [Proxy ARP] 214WANIP 
    wan.xxx.xxx.215/32 [Proxy ARP] 215WANIP 
    wan.xxx.xxx.216/32 [Proxy ARP] 216WANIP 
    wan.xxx.xxx.220/32 [Proxy ARP] 220WANIP_DMZ

    Here are examples of the Firewall rules:

    LAN Interface: (working)
    Proto  Source  Port  Destination  Port  Gateway  Schedule  Description

    • LAN net * * * *   Default LAN -> any

    VLAN213 Interface: (working)
    Proto  Source  Port  Destination  Port  Gateway  Schedule  Description

    • VLAN213 net * * * *   VLAN213 -> any

    VLAN214 Interface: (NOT working)
    Proto  Source  Port  Destination  Port  Gateway  Schedule  Description

    • VLAN214 net * * * *   VLAN214 -> any

    As mentioned earlier, I can ping the router on each interface address from each of the 8 VLANs (e.g. 10.0.215.10 -> 10.0.215.254 works, as well as 10.0.215.10 -> 10.0.211.254) and on the first 4 interfaces, I can ping the GW address (wan.xxx.xxx.209) from a host on VLAN210-VLAN213, but I cannot from VLAN214-VLAN216,VLAN220.

    I'm pretty stuck on why this isnt working; if someone's got any ideas, I'm all ears :-)

    If there's any other data needed, let me know…



  • Bump, And…

    So I did some further testing, trying to narrow down where the issue existed, and in the process I think I may have found a couple defects...

    The first issue is with the original problem in that I have not been able to get any traffic to route from VLAN214 - VLAN220 interface/networks to the WAN connection, given the configuration provided in my OP.

    During my testing, I tried the following scenarios:

    Scenario #1: Reduce # of VLAN interfaces from 8 to 3; (theory: pfSense cannot route traffic for >4 LAN interfaces)

    Configuration: Same as what was documented in the OP, however I removed VLAN's #211, 212, 213. (reconfigured from factory default to maintain consistency in configuration comparison; VLAN214 = opt1, VLAN215 = opt2, etc..)

    Result: Negative; issue still exists. I cannot ping the WAN GW from VLAN 214, 215, 216, etc. (though can still ping LAN GW, i.e. 10.0.215.10 -> 10.0.215.254).

    Conclusion: The issue is NOT related to the number of interfaces which pfSense can route traffic for.

    Scenario #2: Change the VLAN Tag of a working interface; (theory: pfSense has an issue with vlan tag ID's => 214; secondary theory: pfSense has an issue routing traffic from a /24 subnet that is => 214)

    Configuration: Using configuration from OP as a starting point, I changed the VLAN tag ID of opt1 (VLAN211) to 214 keeping the original 211 wan IP address in the NAT/VIP config (wan.xxx.xxx.211). Removed original VLAN214 interface/configuration,VIP/NAT/FW rules to avoid conflicts.

    Result: Positive; traffic can route OK.

    Conclusion: pfSense has no issue with VLAN Tags => 214 or subnets equal or greater to the same number.

    Scenario #3: Change the WAN VIP address of a working NAT config (VLAN211->wan.xxx.xxx.211) to one having an issue (VLAN211->wan.xxx.xxx.214); (theory: pfSense has an issue either with VIP's => 4 instances AND/OR VIP's => 214 (as in /32 address number))

    Configuration: Using configuration from OP as a starting point, I changed the VIP/NAT config of opt1 (VLAN211) to use wan.xxx.xxx.214, and changed the VIP/NAT config of opt4 (VLAN214) to use wan.xxx.xxx.211.

    Result: VLAN211 (using WAN IP 214) failed to route; VLAN215 is able to successfully route.

    Conclusion: pfSense is having an issue routing NAT traffic to VIP WAN IP's =>214.

    So.. Not sure where to go from here; unfortunately, I cannot get another /28 block of IP's from my ISP at this time that is below the hypothetical threshold/limitation I'm running into. In the interim this is figured out, I've had to scratch my VIP/NAT config entirely and have all 8 VLAN interfaces NAT to a single WAN IP (wan.xxx.xxx.210). Not ideal for the configuration I was hoping, but it works.

    Is there anyone out there that would be willing to try and duplicate this issue? I'd like to confirm my sanity…

    The second issue I found while I was testing is a rather nasty bug; I'll post it in another thread to avoid hijacking the purpose of identifying a solution to the original problem. (update: here's the other thread: http://forum.pfsense.org/index.php/topic,14940.0.html)


Log in to reply