PfSense & VMWare ESXi VLAN integration

reloded

Just in case: I'd posted this on stackexchange earlier today. I honestly wish pfSense forum and stackexchange were linked like the ubuntu forums:
https://networkengineering.stackexchange.com/questions/41910/pfsense-vmware-esxi-vlan-integration

Were setting up a pfsense box as a virtual machine inside a VMWare ESXi 6.0 environment (In a VxRail Box: https://www.emc.com/en-us/converged-infrastructure/vxrail/index.htm).

The target configuration is that to access any machine within this box, you need to go through the Firewall. Traffic between machines within the same box also need to go through the firewall. Since they are different servers e.g. Web Server, Database Server, the VMs are setup to be in different VLANs.

As such, we've setup pfSense with two interfaces. One is the WAN that will be used by the "Outside World" to communicate to servers within the ESXI environment and the other is a Trunk that should then connect to all the VLANs protected by the pfSense box.

ESXi:

We've followed the guide https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1004252 to setup a Distributed vSwitch in VxRail. we've setup a distributed port group? of VLAN Type VLAN Trunking and with VLAN IDs 0-200:

To our understanding, this Port Group is what we will attach to pfsense Trunk so that it is able to "read" all the VLAN tagged traffic?

Trunk on PfSense:

To create a trunk on pfsense is basically adding the NIC to the pfsense VM. The NIC should be the Port Group we've created above.

LAN on pfSense:

After that we create a VLAN on pfSense and add a VLAN ID. This VLAN is sitting on the Trunk we've created above. Example below:

We then add an interface based on this VLAN and give it an IP of 192.168.152.1

Protected Machine:

We then create a machine that will be protected by the firewall. So first, we add an NIC to it. The NIC is based on a host network that has a VLAN tag e.g. 152 as seen below.

We then assign the protected machine an IP of 192.168.152.10 with a default gateway of 192.168.152.1.

Problem Statement:

Issue is, after doing all this, the protected machine cannot ping its default gateway. The default gateway cannot ping that machine. It's like there is no communication between them at all. We've added a firewall rule to allow all traffic on Interface152 and logged everything but we cannot see any traffic being accepted or rejected.

What could we have missed? The biggest confusion we have is on the VXRail ESXI setup but any correction on pfSense setup is also welcome.

PS: I don't know why images are not appearing in this post despite me uploading the images to imgur.com… But if you right click on the image, and select show on new tab, it will display. Else you can refer to the question on StackExchange which shows the images

image_1.png_thumb

image_2.png_thumb

image_3.png_thumb

image_4_0.png_thumb

image_4_1.png_thumb

image_5.png_thumb

image_6.png_thumb

m3xiz

I am also running pfSense in an ESXi. However I do not have the license to play with a Distriubted Switch.
I can however attested that playing with VLAN + TRUNK + pFsense works perfectly. This makes me wonder if you have not make a mistake that I am doing quite often when playing with network configuration: forgetting that it is a firewall!.
In short, did you put a rules authorizing the protected pc to ping the firewall?

dreamslacker

Why are you setting the VLAN tag on the VM network adapter setting in ESXI if you are going to tag it from pfSense?

reloded

Yes. In fact for testing we've allowed all on that interface. See attached image..

@m3xiz:

I am also running pfSense in an ESXi. However I do not have the license to play with a Distriubted Switch.
I can however attested that playing with VLAN + TRUNK + pFsense works perfectly. This makes me wonder if you have not make a mistake that I am doing quite often when playing with network configuration: forgetting that it is a firewall!.
In short, did you put a rules authorizing the protected pc to ping the firewall?

image_7.png_thumb

reloded

@dreamslacker:

Why are you setting the VLAN tag on the VM network adapter setting in ESXI if you are going to tag it from pfSense?

You mean why we gave VLAN ID 152 to the Interface for the machine we want to protect?

We'll the assumption is that all traffic coming from that Interface will come with the VLAN Tag 152. We then also create a VLAN with ID/tag 152 on pfSense so that pfSense knows how to interpret traffic from the protected machine. Again, that was an assumption. Correct me if am wrong. How else would we force traffic from the protected machine to get to pfSense on VLANID 152?

EDIT

We've untagged the interface on ESXi but still there is no communication between the VMs. i.e no communication from the protected machine to pfSense and from pfSense to the protected machine

m3xiz

That 's a good point…You should not tagged the VLAN in both systems.

Either, you create a trunk in ESXI (put the vlan TAG to 4095. Than you can add a vlan interface in the firewall. You will have to attach your protected PC on a switch pushing the VLAN 152 to you PC. If you dno have it you can try doing it with openvswitch but this is not obvious.

Or you tag the VLAN in the ESXI and you do not tag it in the pFsense.

reloded

@m3xiz:

That 's a good point…You should not tagged the VLAN in both systems.

Either, you create a trunk in ESXI (put the vlan TAG to 4095. Than you can add a vlan interface in the firewall. You will have to attach your protected PC on a switch pushing the VLAN 152 to you PC. If you dno have it you can try doing it with openvswitch but this is not obvious.

Or you tag the VLAN in the ESXI and you do not tag it in the pFsense.

Please correct me if am wrong.

In ESXi, I need to create a distributed port switch that will accept several VLANs. And the way to do this is to select VLAN Type as VLAN Trunking and put a range in my case I've put 0-200. This ESXi Trunk is what I will add to pfSense as a network interface but I won't assign an IP to it making it a Trunk.
On pfSense, I then create a VLAN with tag 152 using the interface above which is a trunk.
On pfSense, I then add an interface using the VLAN above and I assign it an IP e.g. 192.168.152.1
From there, I create another Interface on ESXi and I DON'T add a VLAN ID. This new interface is what I will use to add a network interface to my protected PC. The interface will have an IP address e.g. 192.168.152.10?

Am I getting you right or is my logic wrong.

PS: For a distributed switch, one cannot set VLAN Trunking to be 4095. See this article: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1004252 where they say "
Note: To improve security, virtual Distributed Switches allow you to specify a range or selection of VLANs to trunk rather than allowing all VLANS via VLAN 4095. "

m3xiz

Regarding the esxi trunking, as said I always played with the basic free license of esxi, as such there is no distributed switch. This is where my knowledge ends :-(

planetinse

Just a note i got this type of setup working fine within a EXSi host - but between two hosts it fails.

mikeisfly

I'm not seeing any images, a diagram would be helpful in helping. With that being said I'm thinking that your LAN traffic from PFsense needs to be tagged going to the vswitch in VMware will know where to direct the traffic (I'm thinking you have that set up correctly). The traffic going to your VMs should be on the associated VLAN but it should not be tagged as hosts by default don't except tagged traffic. If you are sending the host tagged traffic depending on the OS and driver there will need to be additional configuration that is needed. I know in the case of Windows and Intel NICs you may need to download the pro Driver.

planetinse

My setup is; (I tried with traditional separate switchgroups first - same result)

Basically a Distributed-Switch over the two hosts with 2 Port-groups,
one Trunk VLAN (all) and one VLAN tagged with 100)

pfsense connected to Trunk VLAN - and created VLAN inside pfsense with 100 tag

then VM uses the VLAN portgroup (that is tagged to 100)

works like a charm, DHCP, internet etc. - when Iam on the same Host.

but when VM is on the other host, nothing works, no DHCP, even if i set static ip to what i have selected - i can not even ping the gateway.

I have moved both pfsense and VM's back and forth to exclude there is a specific issue with one of the hosts.

I guess there is something in the underlying network that is the problem, according to the vendor (Iam colocated) this network (that my distributed switch is using for uplinks thru one card per host) is a PRIVATE VLAN allowing 0-4095, so I assumed it would work... this is really out of my competence zone :)

However I don't see how this can happen within the DSwitch in ESXi (that should be distributed over the hosts)