nested VLANs can't connect through PFSense

tahunua

hello all,
I am in the process of attempting to implement a PFsense appliance for a small lan. I have 3 data vlans nested inside an exit vlan which connects my core switch to the appliance. if I place a device in the exit vlan I can get a very solid internet connection, but none of the data vlans can pass traffic. I have not enabled any firewall rules yet so it shouldn't be filtering anything as of yet. I have tried adding all of the vlan numbers to the vlan list and have nat rules in place for all of the data vlans. has anyone else run into this problem when implementing a new pfsense device?

JKnott

@tahunua
I assume by nested VLANs you mean QinQ. VLANs are a layer 2 function and pfSense operates at layer 3. I don't think what you're trying to do is possible. Please provide more info on what you're doing.

tahunua

I suppose I was rather imprecise, wasn't I? sorry about that.

I have 3 VLANs separating my network, and each has a /24 private IP network built on top of it. what I call the exit VLAN has a .29 private IP network built to allow for possible unforeseen expansions.

I have added all vlans to the vlan list on the PFsense device, and have created custom NAT rules for each of the IP networks that reside on those vlans (except for the exit vlan).

if I connect a test laptop to another port on my core switch attached to the exit vlan, and assign IP addresses in that /29 range, I can surf the web do my connection to to the network is not to blame. however I cannot access the internet from any of the 3 ip networks on the data vlans.

as previously stated, I have not enabled any filter rules so I shouldn't be having issues with the appliance itself dropping the packets, but rather is likely something I just haven't thought to set up. I probably should also mention that this is not a new network coming up for the first time, but rather the appliance is replacing my existing gateway.

since it was brought up, what is QinQ? I am unfamiliar with that term.

JKnott

@tahunua said in nested VLANs can't connect through PFSense:

since it was brought up, what is QinQ? I am unfamiliar with that term.

I'm getting the impression you're unfamiliar with VLANs in general. A VLAN is a way to create virtual networks over 1 physical network. Logically, each VLAN appears as those it's completely separate from the others. The only way to pass traffic between them is with a router. This isolation is created by using something called VLAN tags, in which 4 bytes are inserted at the front of the Ethernet payload, to identify the packet as being a VLAN and the VLAN number. The tag is then followed by the Ethernet bytes, which had originally been at the beginning. Because of the tags, a router cannot see the frame as IP, as the wrong Ethertype is at the beginning. QinQ means VLAN over VLAN, with 2 tags, instead of 1. This is often used when a carrier provides fibre to a customer and uses 1 VLAN layer to keep the customers separate and the 2nd is free for the customer to use.

Are you sure you're using VLANs? Or just separate networks connected to pfSense?

tahunua

I happen to teach intro to networking so yes I am quite aware of what a VLAN is. with that said, my core switch is a layer 3 device, so it acts as the router that communicates between the 3 data vlans and the gateway. I have already experimented with disabling the vlan connecting to the pfsense device and creating a routed port connection (no more layer 2 separation) and it did not correct the issue.

JKnott

@tahunua said in nested VLANs can't connect through PFSense:

my core switch is a layer 3 device, so it acts as the router that communicates between the 3 data vlans and the gateway

Where does pfSense fit in?

Derelict

So you only need the transit network to the layer 3 switch, not the VLANs.

Creating the VLANs on pfSense will create additional, unnecessary, layer 3 interfaces there. You already have them on the switch.

tahunua

ok I'll get rid of the vlans in the pfsense and advise.

tahunua

@JKnott said in nested VLANs can't connect through PFSense:

Where does pfSense fit in?

PFsense is replacing my gateway, which connects my core switch to the ISP.

an update as of today's attempt to fail over.
I have deleted the vlan interfaces I created on PFsense and reattached to the network (removing the existing gateway device.
I have tried both hybrid outbound NAT and manual outbound.
I have created a new gateway and created static routes to each of the local IP networks, excluding the directly attached (I assume it already knows how to route to directly connected networks.

none of these has created any change in network operation. if I connect a client into the local interface LAN, then it can communicate just fine over the internet, however none of the networks on the other side of my core switch are able to communicate outside. they can ping the LAN interface, but that is about as far as they can get.

my setup is very similar to the diagram posted by derelict, only the network with host C is also connected through the layer 3 switch, rather than a direct connection to an interface on the PFsense. I can get to the internet from the pink network, but I cannot get out from any of the three black lans.

Derelict

Switching to hybrid outbound NAT is not good enough.

You need to be sure that any source addresses that require NAT are actually included there.

If not they need to be manually added.

This is usually picked up by pfSense from the static route that routes the traffic for those networks to the Layer 3 Switch/Downstream Router.

tahunua

explicit NAT rules for each subnet were part of my original configuration before TS steps began. adding the static routes was the most recent change to config.

update: I have compared my NAT rules with the pool from my old gateway, and I did discover an error in the public IP mappings but after correcting this in the NAT rules I still have no change in operation.

Derelict

Well something wasn't done correctly or it would have worked.

Your best bet is to get it to where you think it should be working but isn't and post screenshots of all of the elements outlined in the dialog boxes in that diagram. Pretty sure I covered everything that needs to be in place there.

tahunua

ok I have finally figured out what has happened.

the PF sense appliance was not smart enough to find my core switch as an internal gateway unless I reconfigured the switch port as a routed port and assigned the IP address directly to the port, rather than an access port and using the SVI as the gateway address.
with all rules disabled, the PFsense still maintained the deny all rule at the end, so all traffic was being denied. after I just created an outbound rule that allowed everything, my local vlans can now pass traffic so it is just me being an idiot with my rules.

thank you very much for your help. I feel like such a newb right now.no to start securing it.

Derelict

@tahunua said in nested VLANs can't connect through PFSense:

ok I have finally figured out what has happened.

the PF sense appliance was not smart enough to find my core switch as an internal gateway unless I reconfigured the switch port as a routed port and assigned the IP address directly to the port, rather than an access port and using the SVI as the gateway address.

If your switch properly responded to ARP in that configuration it would have worked. It is all standards-based. There are no "smarts" necessary. If the gateway is defined on an interface and routed to, and it responds to ARP, that's where the traffic will be sent. Not familiar enough with your switch so say whether it would one way or another in that configuration.

This is my transit interface on my brocade switch. Nothing special needed on the switch ports. Just tagged to the LACP LAGG and an "SVI":

vlan 1000 name TRANSIT by port
 tagged ethe 1/1/35 to 1/1/36 
 router-interface ve 1000
!
interface ve 1000
 ip address 192.168.230.2 255.255.255.252
 ip ospf area 0
!

with all rules disabled, the PFsense still maintained the deny all rule at the end

Wholly expected.

so all traffic was being denied. after I just created an outbound rule that allowed everything, my local vlans can now pass traffic so it is just me being an idiot with my rules.

You mean a rule allowing those subnet source addresses into the pfSense side of the transit interface right?

JKnott

@tahunua said in nested VLANs can't connect through PFSense:

the PF sense appliance was not smart enough to find my core switch as an internal gateway unless I reconfigured the switch port as a routed port and assigned the IP address directly to the port, rather than an access port and using the SVI as the gateway address.

????

Switches are normally transparent. You have routers as gateways, not switches. BTW, where is this core switch. I only see 1 switch.

JKnott

@Derelict said in nested VLANs can't connect through PFSense:

If your switch properly responded to ARP in that configuration it would have worked. It is all standards-based.

????

Why would a switch respond to an ARP, unless you're accessing the management interface? I suspect the confusion may be caused by the OPs descriptions. What switch is he referring to? Is it layer 2 or 3? If 3, then it would respond to an ARP request, as would any router.

tahunua

You mean a rule allowing those subnet source addresses into the pfSense side of the transit interface right?

actually I did what you should never do in a live environment and did a shotgun rule to allow any source to any destination on any port using any protocol. my current set up is more targeted, with rules applying to specific ports and protocols with specific source IP networks. so far it is stable, and just has a few hiccups which I should be able to TS on my own from here out.

as for ARP responses, I don't know what could have been the source of the interference. so far the routed port has remained stable.

Switches are normally transparent. You have routers as gateways, not switches. BTW, where is this core switch. I only see 1 switch.

I am going to assume you are self educated when it comes to networking. there is nothing wrong with this, in fact I admire people who have figured this all out for themselves. however you appear to be missing a fair bit of nomenclature/terminology when it comes to network design and network diagrams.

a core switch refers to the backbone of a LAN, CAN, or MAN. it is the framework that disseminates data throughout a building, campus, or organization. typically because of a need for added security, compartmentalization, and optimization, core switches rarely are straight layer 2 forwarding devices. instead they have a lot of the capabilities of routers, firewalls, monitors and other layer 3 and up devices. my core switch is very basic so it does not have more advanced layer 3 features such as NAT, but it can do static and dynamic layer 3 routing, vlan tagging and segregation, etherchannel (also called channel aggregation), and DHCP services. on a network diagram, multilayer switches such as this do not have the normal rectangle with 2 arrows pointing left and right, instead they are typically a square with arrows fanning out in all directions. the device on the left of Derelict's diagram is a multilayer (Layer 3) switch, while the device on the right hand side is a basic layer 2 (mostly transparent) switch.

Derelict

@JKnott said in nested VLANs can't connect through PFSense:

Why would a switch respond to an ARP, unless you're accessing the management interface? I suspect the confusion may be caused by the OPs descriptions. What switch is he referring to? Is it layer 2 or 3? If 3, then it would respond to an ARP request, as would any router.

Layer 3 interface, man. Layer 3.

tahunua

I should also use the disclaimer that I tend to use a lot of Cisco Specific terminology so I apologize if there are vendor neutral terms that would make it easier to understand what I am trying to communicate. with the exception of the PFsense appliance running on an old Dell server blade, my entire network is Cisco.

JKnott

@Derelict said in nested VLANs can't connect through PFSense:

Layer 3 interface, man. Layer 3.

Yes, however, my point was he was not clear in what he was referring to. If a layer 3 switch, then that should have been explicit. The diagram refers to both L2 and L3/