Need help understanding VLANs and pfSense

[benderbot]

Hi all,

First, I'm brand new to VLANing and pfSense, and I don’t have a lot of networking experience. Second, sorry for the long post.

I am having trouble wrapping my head around VLANs when it comes to using them in chorus with my physical router, switches and hosts and with the ESXi host and pfSense vm.

I’m new to my company and am in the process of setting up a managed network for them (Their mistake but, hey, they hired me). The goal is to have three VLANs: One for management, one for restricted personnel and devices and one for mixed use, non-restricted access. We work with the federal government, so we need to keep their data on a separate subnet and VLAN.

I have a test network set up and that will eventually be migrated to production, if and when I figure out VLANing. For now, I’m using the very few physical devices we have on hand. This is what I have:

• Netgear FVS336Gv3 (to be replaced by a Cisco ASA 5506 in the very near future)
• Netgear GS308E managed switch (to be replaced by a newer Netgear managed switch)
• A server running ESXi 6.7.0u3 (to be upgraded to 7.0 once I have NICs that will work with it.)
• pfSense software 2.5.1

On the FVS336Gv router:

Inter VLAN routing is disabled

• WAN port = our outside connection
• Port 1 = VLAN 10 for management (Gateway 10.10.10.1, DHCP Enabled)
• Port 2 = VLAN 20 for restricted access (Gateway 10.10.20.1, DHCP Enabled)
• Port 3 = VLAN 30 for unrestricted access (Gateway 10.10.30.1, DHCP Enabled)
• Port 4 = VLAN 2 connected directly to the ESXi host and used as the pfSense WAN (172.16.0.1, DHCP Enabled)

Eventually, DHCP will be the job of a domain controller, unless using pfSense is better. Not quite sure about that yet either.

On the GS308E switch:

VLAN 1

• Ports 1, 7 and 8; all untagged

VLAN 10

• Ports 1 and 2; 1 is tagged, 2 is untagged

VLAN 20

• Ports 1, 3 and 4; 1 is tagged,3 and 4 are untagged

VLAN 30

• Ports 1, 5 and 6; 1 is tagged,5 and 6 are untagged

Other

• VLAN 2 not included since it is connected directly to the ESXi host.
• Rather than connect the router directly to the ESXi host, I’ve run everything through this switch. This is how it will be in production.
• I was able to connect the ports used for 10, 20 and 30 to my laptop with some USB NICs and it picked up an IP address from each VLAN.

On the physical ESXi host:

2 physical NICs:

• 1 onboard (vmnic0)
• 1 PCIe (vmnic1)

ESXi host:

I’m using an evaluation copy until we purchase a license. This means, I can’t set up a distributed switch, only a standard one. I do not have vCenter, so everything is done through the host’s console.

VMKernel NICs

• vmk0

Virtual Switches:

• The default vSwtich0 with vmnic0 as the uplink
• “WAN-Switch” with vmnic1 as the uplink

Port Groups:

VM Network

• VLAN 0
• vSwitch0

Management Network

• VLAN 0
• vSwitch0

LAN

• VLAN 4095
• vSwtich0

WAN

• VLAN 0
• WAN-Switch

Management

• VLAN 10
• vSwtich0

Restricted

• VLAN 20
• vSwitch0

Unrestricted

• VLAN 30
• vSwitch0

pfSense Interfaces:

• WAN = em1
• LAN = em0
• VLAN 10 on em0
• VLAN 20 on em0
• VLAN 30 on em0

I believe the setup described above is using what VMWare would consider External Switch Tagging (EST). Ideally, I would like to have one place to manage the VLANs and I don’t think configuring them on the switches accomplishes that. I would like to avoid guest tagging since, like EST, it’s not easily scalable. That leaves me with Virtual Switch Tagging but then how does that work with physical devices, like desktops and laptops? I was hoping pfSense would be a central point of management but I’m not so sure.

My questions regarding pfSense, aside from those above, are:

• When I try to give my pfSense VLANs a static IP, I get an error that the address and the address range are already in use. I assume it is getting that info from the router. Does that mean I can’t have both setups running at the same time?

• I don’t quite understand what happens when the IPv4 Configuration Type is left on DHCP. Does that mean it will then use the IPs the router is serving? If that’s the case, what’s the point of setting up VLANs in pfSense in the first place? Is it so firewall rules can be set?

• What happens if the IPv4 Configuration Type is set to none?

• If I do use pfSense as the central place of management, how would that work with physical devices? Would I have to assign a VLAN on each host i.e. guest tagging?

I know that’s a lot, so thanks in advance for your help. In the meantime, I’ll keep reading and watching videos.

[benderbot]

I disabled DHCP on the router and enabled it on the pfSense VM. I'm now able to connect to the proper VLAN in ESXi by assigning the desired ESXi port group to my guest VM. For example, I have Win10 VM with the restricted port group (VLAN 20) attached. That VM is now showing up on the VLAN 20 network. However, now I am not getting VLAN traffic through my physical switch to my physical computer. So, it seems like I get one or the other. Either it works on the physical network when the router is set up as a DHCP server or it works on the virtual network when pfSense is the DHCP server. Naturally, I need both. Since I am currently working primarily in the virtual world, I can move on to getting the rest of my virtual nodes and appliances set up. But, very soon I will have to incorporate physical devices. Any idea what I am missing? Thanks!

[striker-pl]

@troybdex have you setup any firewall rules to actually allow the traffic to be passed anywhere?

I'm pretty new to VLAN's as well, just starting to figure them out myself, but this was one thing that I stumbled on in the early phases too.

If you don't have any rules setup, the traffic will be limited to only what can be reached inside it's own VLAN.

Depending on how you are doing NAT, you may have to setup some rules in NAT as well so the traffic can get out to the internet.

[benderbot]

@striker-pl Thanks for the reply!

Yes, I have rules set and they work with my VMs.

I know it seems counter intuitive, but the “Mixed” VLAN pictured below is the restricted one. It is not allowed to see any traffic at all on the “USPO” (US Persons Only) VLAN but the USPO VLAN is allowed to see traffic on the Mixed. It’s not the best naming convention, but it’s also not one I created.

I have one Win 10 VM guest on each VLAN. At first, the guests couldn’t ping each other. They could only ping each others gateways. I disabled the Win 10 firewalls on both Win 10 vm guests and then I could cross ping clients. Since that is not what I need, I made a firewall rule to block all traffic from the MIXED VLAN to the USPO VLAN. After that, I could no longer ping either the client or the gateway of the USPO Win 10 guest from the Mixed guest. Success!

Unfortunately, I still am not getting traffic across my Netgear GS308E switch to my Dell laptop. The router is plugged into port 1 on the GS300E switch and the to physical LAN interface on the ESXi host through a wifi router I have set up as a dumb switch. I have a cable running from port 3 of the GS308E to a USB NIC on my laptop and another running from port 5 to a different USB NIC on my laptop.

The physical config is:

• Main router (FVS336Gv3) to LAN port 1 (nothing plugged into WAN connection) on the dumb WiFi router acting as a temp switch. (I can take this out of the config by editing the PVID on the GS308E. I will do that but not yet.)
• Port 2 on the dumb WiFi to port 1 on the GS308E.
• Port 3 on WiFi to physical LAN interface on the ESXi host.
• Port 3 on GS308E to USB NIC on my Dell Laptop
• Port 5 on GS308E to second USB NIC on my Dell Laptop

Port 1 is tagged on all VLANs, except VLAN 1 on which is it untagged. Maybe my PVID is configured wrong?

Thanks again for replying. I hope I haven't muddied the waters with this post. I appreciate any help you can provide.

Troy

[striker-pl]

@troybdex i'll say that i'm having a tough time wrapping my head around the physical connections. A simple diagram may help myself (or others) that read this thread.

Typically your PVID should be set to the same as your untagged vlan on the port as this the default vlan that you want the traffic to be on unless otherwise tagged by the device connected to the port.

[benderbot]

@striker-pl Thanks for the reply. Here is a drawing of my current setup. Hope this makes sense.

The laptop is also connected to the WiFi router via WiFi. Everything else is a physical or virtual connection.

I reconfigured the GS308E a bit.

[striker-pl]

@troybdex is port 2 and port 4 on your Linksys unit tagged for all vlans? If not, that would be your issue. You need to have it tagged on both sides for it to work and communicate properly.

[benderbot]

@striker-pl I removed the WiFi router from the mix but it didn't resolve the issue.

Here's the new config.

One thing I think I forgot to mention is that I still have DHCP enabled for VLAN 10 on the FVS336G. pfSense is handling DHCP for VLANs 20 and 30.

I have VLAN 10 enabled in pfSense, but have the IP config set to none and have not set up a DHCP server for this VLAN. Is that my problem? Should I disable DHCP on the router and just let pfSense handle it? I was afraid to do that, since I'm not getting VLAN info over the physical lines and that's what my laptop is connected to.

[striker-pl]

@troybdex you should be able to use whatever DHCP server you want on each VLAN.

Do you have an IPv4 address setup for VLAN20 or VLAN30 in your interface management screen?

I see that you don't have one setup for VLAN10.

[benderbot]

@striker-pl I do.

This is the config for VLAN 20 and 30

[striker-pl]

@troybdex and the IPv4 Address is set to /24 right?

Just to confirm, your DHCP servers are Enabled on VLAN20 and VLAN30? You can configure them, but there is a separate Enable checkbox that needs to be set as well.

[benderbot]

@striker-pl yep. Both have a check in the "Enable DHCP server on MIXED interface" box. I am getting addresses from both on my VMs. I have these VLAN's set up on my router, but they are disabled.

[benderbot]

@striker-pl

[johnpoz]

So you understand when you have tags going into esxi, if you don't set 4095 on the vswitch - the tags will be removed.

If you don't want esxi messing with tags you need to set 4095 on any vswitch the vlan tags will cross.

You have that set but what about those other vlans you have on esxi? How do you have that connected into pfsense to route? Your esxi breakout doesn't show any detail on how you have your pfsense vm setup tied to what vswitches in esxi, which is connected how exactly to your physical network.

And looks like you have the same IPs setup on you router, that you have as pfsense IP.. 10.10.20.1, 10.10.30.1 for example? You can not do that on the same L2.. Or your going to have dupe IP issues..

To honest I have no idea what your wanting to accomplish here.. If your going to have 2 routers. and pfsense is going to be downstream of your other router. Then they would be connected with a transit, and that upstream router wouldn't have settings for any of those vlans - only routes via the transit to get to them via pfsense.

[benderbot]

@johnpoz Thanks for the reply. Yes, I understand that.

Here is my ESXi config. This has not been modified since my original post. You can ignore VLAN 40 as it is not in use at the moment. VLAN 10 is also not in use, but when I had a vm tied to it, it got the proper IP address. All other VMs are receiving an IP address in their respective VLAN range.

Could this issue with the configuration on my physical laptop? I am running Ubuntu Mate 20.04. Is there anything I need to configure on there before this will work. I don't have a physical Win 10 Eval box, but can build one for testing if need be.

[johnpoz]

And that makes no sense at all.. Pfsense is not going to be able to route any of those..

[benderbot]

@johnpoz As I stated in my previous post, VLANs 20 and 30 are configured on the router but are disabled. The pictures verify this.

In the end I will have a physical and virtual network that pass the same VLAN tags. I need VLANing to work with my routers, switches, physical desktops and servers and virtual desktops and servers.

[johnpoz]

Yeah - well what you have there isn't going to do that.

What do you want to route between your vlans? Doesn't matter if they are on virtual, physical or both..

You still need something to route between them - is that going to be your upstream router, or pfsense.. If its pfsense it has to have an interface in that L2.. Are you trying to send traffic back out to physical just to come back in another physical via your 4095 vswitch? So that pfsense can see the tags?

What you have there is via the technical term - a MESS ;)

[benderbot]

@johnpoz said in Need help understanding VLANs and pfSense:

What you have there is via the technical term - a MESS ;)

Yeah, it's a hot mess. The problem is, I have no clue what I'm doing and I have 4 more months to do it. Thankfully, management is patient with me, as they know even less than I do.

Are you trying to send traffic back out to physical just to come back in another physical via your 4095 vswitch? So that pfsense can see the tags?

To be honest, because my networking skills are non-existent, I don't know.

We're a small office, with three local employees, myself included, and six remote. Currently, there is no centralized network management. Everything is done on the local computer or server. It's my job to transition the network to one that is centrally managed and that is secure.

The company recently - before I was hired - set up two ESXi hosts and migrated some of their physical servers to them. One host is for USPO and one for Mixed. The USPO network is for our government work that can only be accessed by US citizens. The Mixed network is for everyone else. Mixed cannot see USPO computers, applications, etc. but USPO is allowed to see everything. Right now, all of the network segmentation is handled on the routers, using separate subnets but not using VLANs. The boss would like USPO and Mixed each on their own isolated VLAN. The company also has four or five additional servers that will eventually be virtualized. However, a few physical servers will remain indefinitely and those must also be on the VLANs. Then, of course, there are the physical desktops and laptops we use to access all of these systems, which also need to be on their respective VLANs.

In the end, I need to have that segmentation and to have all of it work with Active Directory. Eventually, I need to throw a VPN and a DMZ into the mix. I figured I would start with the perimeter firewall first and work my way back from there. Since pfSense can also be a router, I figured I would run DHCP from pfSense rather than having to configure each and every physical router, switch and host. I realize that some router/switch configuration may be necessary.

So, basically, all traffic, regardless of location or whether it is physical or virtual, needs to transit a VLAN. As to how that is accomplished...that's why I'm here. I'm open to any ideas, better solutions and/or advice.

Please note that the setup I have shared is on a test network that is isolated from our production network, so I have a lot of room for play.

If its pfsense it has to have an interface in that L2

I'm not entirely sure what you mean but all of my VLANs are assigned to the LAN port in pfSense.

[benderbot]

@johnpoz is this something that is possible?