Mutiple VLAN -> NAT -> Multi VIP static IP WAN.. questions…

jaubine

I'm trying to configure pfSense with 8 VLAN'ed LAN interfaces (10.0.xxx.0/24 subnets, trunked through a single interface, sis2), each pointing to a separate public WAN ip (from a /28 block). I've configured all 8 interfaces identically wrt each VLAN, and each interface will successfully dish out a DHCP address and will allow traffic on the LAN side, but 4 (specifically the last 4) interfaces will not route/NAT traffic out to the WAN side.

I'm not seeing anything out of the ordinary when I compare one of the working interfaces (e.g. VLAN210) with one that doesnt (e.g. VLAN215), with the exception of only one thing - I noticed on the serial console that the 4 I'm having trouble with do not have asterisks next to their interface names:

*** Welcome to pfSense 1.2.2-embedded on redfw01 ***

WAN***** -> sis6 -> wan.xxx.xxx.210
LAN***** -> vlan0 -> 10.0.210.254
OPT1(VLAN211)***** -> vlan1 -> 10.0.211.254
OPT2(VLAN212)***** -> vlan2 -> 10.0.212.254
OPT3(VLAN213)***** -> vlan3 -> 10.0.213.254
OPT4(VLAN214) -> vlan4 -> 10.0.214.254
OPT5(VLAN215) -> vlan5 -> 10.0.215.254
OPT6(VLAN216) -> vlan6 -> 10.0.216.254
OPT7(VLAN220_GUESTNET) -> vlan7 -> 10.0.220.254

I looked around for documentation on what these asterisks indicate, but alas, I cannot find any; I'm not entirely sure if this has anything remotely to do with the problem that I'm having.. The "vlan0 - vlan7" noted above I think are just internal markers, and are not actually parsed as the actual VLAN tag numbers; here's the VLAN info from the GUI interfaces tab:

Interface VLAN tag Description
sis2 210 VLAN210_MGMT
sis2 211 VLAN211
sis2 212 VLAN212
sis2 213 VLAN213
sis2 214 VLAN214
sis2 215 VLAN215
sis2 216 VLAN216
sis2 220 VLAN220_GUESTNET

Since I can connect and communicate from a host client IP on each of the 8 VLAN's to the router, I know that my internal trunking and VLAN's are working correctly, and that I suspect the issue lies either in NAT or Firewall rules.

Here is the list of the outbound NAT table:

Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
WAN 10.0.0.0/24 * * * * * NO Auto created rule for LAN
WAN 10.0.210.0/24 * * * wan.xxx.xxx.210 * NO VLAN210–>WAN210
WAN 10.0.211.0/24 * * * wan.xxx.xxx.211 * NO VLAN211-->WAN211
WAN 10.0.212.0/24 * * * wan.xxx.xxx.212 * NO VLAN212-->WAN212
WAN 10.0.213.0/24 * * * wan.xxx.xxx.213 * NO VLAN213-->WAN213
WAN 10.0.214.0/24 * * * wan.xxx.xxx.214 * NO VLAN214-->WAN214
WAN 10.0.215.0/24 * * * wan.xxx.xxx.215 * NO VLAN215-->WAN215
WAN 10.0.216.0/24 * * * wan.xxx.xxx.216 * NO VLAN216-->WAN216
WAN 10.0.220.0/24 * * * wan.xxx.xxx.220 * NO VLAN220_GUESTNET-->WAN220

Here is the VIP list:

Virtual IP address Type Description
wan.xxx.xxx.210/32 [Proxy ARP] 210WANIP
wan.xxx.xxx.211/32 [Proxy ARP] 211WANIP
wan.xxx.xxx.212/32 [Proxy ARP] 212WANIP
wan.xxx.xxx.213/32 [Proxy ARP] 213WANIP
wan.xxx.xxx.214/32 [Proxy ARP] 214WANIP
wan.xxx.xxx.215/32 [Proxy ARP] 215WANIP
wan.xxx.xxx.216/32 [Proxy ARP] 216WANIP
wan.xxx.xxx.220/32 [Proxy ARP] 220WANIP_DMZ

Here are examples of the Firewall rules:

LAN Interface: (working)
Proto Source Port Destination Port Gateway Schedule Description

LAN net * * * * Default LAN -> any

VLAN213 Interface: (working)
Proto Source Port Destination Port Gateway Schedule Description

VLAN213 net * * * * VLAN213 -> any

VLAN214 Interface: (NOT working)
Proto Source Port Destination Port Gateway Schedule Description

VLAN214 net * * * * VLAN214 -> any

As mentioned earlier, I can ping the router on each interface address from each of the 8 VLANs (e.g. 10.0.215.10 -> 10.0.215.254 works, as well as 10.0.215.10 -> 10.0.211.254) and on the first 4 interfaces, I can ping the GW address (wan.xxx.xxx.209) from a host on VLAN210-VLAN213, but I cannot from VLAN214-VLAN216,VLAN220.

I'm pretty stuck on why this isnt working; if someone's got any ideas, I'm all ears :-)

If there's any other data needed, let me know…

jaubine

Bump, And…

So I did some further testing, trying to narrow down where the issue existed, and in the process I think I may have found a couple defects...

The first issue is with the original problem in that I have not been able to get any traffic to route from VLAN214 - VLAN220 interface/networks to the WAN connection, given the configuration provided in my OP.

During my testing, I tried the following scenarios:

Scenario #1: Reduce # of VLAN interfaces from 8 to 3; (theory: pfSense cannot route traffic for >4 LAN interfaces)

Configuration: Same as what was documented in the OP, however I removed VLAN's #211, 212, 213. (reconfigured from factory default to maintain consistency in configuration comparison; VLAN214 = opt1, VLAN215 = opt2, etc..)

Result: Negative; issue still exists. I cannot ping the WAN GW from VLAN 214, 215, 216, etc. (though can still ping LAN GW, i.e. 10.0.215.10 -> 10.0.215.254).

Conclusion: The issue is NOT related to the number of interfaces which pfSense can route traffic for.

Scenario #2: Change the VLAN Tag of a working interface; (theory: pfSense has an issue with vlan tag ID's => 214; secondary theory: pfSense has an issue routing traffic from a /24 subnet that is => 214)

Configuration: Using configuration from OP as a starting point, I changed the VLAN tag ID of opt1 (VLAN211) to 214 keeping the original 211 wan IP address in the NAT/VIP config (wan.xxx.xxx.211). Removed original VLAN214 interface/configuration,VIP/NAT/FW rules to avoid conflicts.

Result: Positive; traffic can route OK.

Conclusion: pfSense has no issue with VLAN Tags => 214 or subnets equal or greater to the same number.

Scenario #3: Change the WAN VIP address of a working NAT config (VLAN211->wan.xxx.xxx.211) to one having an issue (VLAN211->wan.xxx.xxx.214); (theory: pfSense has an issue either with VIP's => 4 instances AND/OR VIP's => 214 (as in /32 address number))

Configuration: Using configuration from OP as a starting point, I changed the VIP/NAT config of opt1 (VLAN211) to use wan.xxx.xxx.214, and changed the VIP/NAT config of opt4 (VLAN214) to use wan.xxx.xxx.211.

Result: VLAN211 (using WAN IP 214) failed to route; VLAN215 is able to successfully route.

Conclusion: pfSense is having an issue routing NAT traffic to VIP WAN IP's =>214.

So.. Not sure where to go from here; unfortunately, I cannot get another /28 block of IP's from my ISP at this time that is below the hypothetical threshold/limitation I'm running into. In the interim this is figured out, I've had to scratch my VIP/NAT config entirely and have all 8 VLAN interfaces NAT to a single WAN IP (wan.xxx.xxx.210). Not ideal for the configuration I was hoping, but it works.

Is there anyone out there that would be willing to try and duplicate this issue? I'd like to confirm my sanity…

The second issue I found while I was testing is a rather nasty bug; I'll post it in another thread to avoid hijacking the purpose of identifying a solution to the original problem. (update: here's the other thread: http://forum.pfsense.org/index.php/topic,14940.0.html)