Using PFSense as a router between two ESXI hosts - VLANS, routing help
-
Hey Folks.
Hoping for some help here as I am working to use PFSense as a router between networks on my two ESXi hosts.
Basically, here is what I have setup:I created one Virtual Distributed switch that contained one physical NIC from each ESXi host.
I created two Distributed Port Groups: One for "LAN" and a second for "INTERVLAN". The idea here is that the PFSENSE VM will have both the LAN and INTERVLAN networks attached to it and then route traffic between different VLAN's and out to the internet.My initial setting up, i was able to get PFsense to to talk to one test VM on my INTERVLAN network. PFsense was routing traffic for that VM out to the internet and also providing DHCP.
Where I am stumbling is when i try and setup VLAN's.
Ideally, I would like to have at least two VLAN's:DEV - VLAND ID 30
WORK - VLAND ID 60The idea being that I can create Distributed Port groups (DEV, WORK) and assign them VLAN ID's in Vsphere appropriately, them attach those networks to PFSense as interfaces.
First, is that correct?
I was testing around in Pfsense, i added a VLAN (VLAN ID 30, what i setup in vsphere) interface in pfsense and attached it to my INTERVLAN interface.
I am slightly confused on how i should configure the INTERVLAN interface and the new VLAN (configuring the IP address basically)But in my testing, when I try to use the VLAN setup, my test VM that has the INTERVLAN network connected to it, is unable to get an IP address. Little stuck
Also, on my 24 port switch, I left the two physical ports that this distributed switch is using as 'trunked' basically. Nothing specified.
Hope I made sense.
i appreciate all the help.
Cheers,TCG
-
If you created a port group that would be the vlan. Pfsense would not have a clue to the vlan in such a setup, if you want pfsense to handle the vlan tags then you would set your vlan ID on your port group to 4095 so that esxi does not strip the tags.
If your going to connect pfsense interface to a port group that is doing the vlan handling, ie some ID on it 30 or 60 say then you would just connect a vnic to this port group on pfsense VM..
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
If you created a port group that would be the vlan. Pfsense would not have a clue to the vlan in such a setup, if you want pfsense to handle the vlan tags then you would set your vlan ID on your port group to 4095 so that esxi does not strip the tags.
If your going to connect pfsense interface to a port group that is doing the vlan handling, ie some ID on it 30 or 60 say then you would just connect a vnic to this port group on pfsense VM..
Ok. That helps me out.
yes, i would like to have PFSense route the traffic between my different networks in my ESXi hosts.
For now, I just created the two port groups: WAN (192.168.1.x network) which would be the outbound for pfsense interface to send traffic upstresam and DEV (10.0.4.x network) which would be the network i would put my work VM's in.So basically, I should make that port group (DEV in this case) a VLAN trunk setup (specifically set the VLAN ID on the DPG to 4095)?
That DEV Port group network, i would attach it to both PFsense and my test VM?EDIT: Actually, I am wondering if i am confusing myself and where I am making the mistake is in the VMWare networking piece. I kinda played around with some items, based on what you said. I went into the DPG and tested.
The interesting piece is that when i set that DPG (DEV in this case) to basically a trunked VLAN, it does not grab an IP address from Pfsense, but instead grabs an ip address from my external router, plugged into my ISP.
Now, if i modify that DPG and set the VLAN ID to 30, it grabs a DHCP from pfsense from the DEV interface, even though i have no VLAN's configured on PFsense.
Hoping that makse sense.I think my problem is in VMware networking?
Just wrapping my head around this.
ThxTCG
EDIT: One other question: Could my layer 2 switch be part of the problem? Ubiqutity Unifi 24 port. Just thought i'd throw that out there.
-
So I was up late last night working on this for several hours, trying to figure this out. If I could just ask for clarification to make sure I am on the right path, or wrong.
In trying to 'visualize' this setup, here is what I think is the way to do it:
- Create one or more Virtual Distributed switches (add the NIC's to this switch)
- Create distributed port groups for the VLAN's I want to setup (LAN, dev, work)
a. In the distributed port groups, assign them a VLAN ID? (Dev VLAN ID 30 for example). - Attach the port groups to the VDS.
- Attach the DPG/networks to the VM's (PFSense would get two or three, LAN and DEV.)
That should be it for the VMware side of the house?
In the pfsense side, i would create a VLAN and attach it to the DEV DPG interface? (DEV DPG interface would be em1 in Pfense).
Does that look correct? If not, where did i mess up?
Also, on the physical switch side, the two nics that are part of the VDS, are currently set to TRUNK (ubiquity 24 port unif) where the profile is 'ALL'.
I appreciate all the help.
Cheres,
TCG
-
If your creating port groups then no you would not have vlans in pfaense - only interfaces... If your going to do vlan tagging in pfsense then you would need to have your vswitch/port group with ID 4095 so esxi does not strip tags...
Drawing is worth 10K words...
Some understanding of what exactly your trying to route between? I would assume both your hosts are connected to the same Layer 2s and you just want to run pfsense on 1 or both of these esxi hosts.. You could setup a carp for HA if you wanted..
Or you could run 2 different pfsense routers 1 on each host with a transit between them if the networks are different on each esxi host, etc..
Why don't you draw up what your wanting to do and we can discuss the best way to do it.
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
If your creating port groups then no you would not have vlans in pfaense - only interfaces... If your going to do vlan tagging in pfsense then you would need to have your vswitch/port group with ID 4095 so esxi does not strip tags...
Drawing is worth 10K words...
Some understanding of what exactly your trying to route between? I would assume both your hosts are connected to the same Layer 2s and you just want to run pfsense on 1 or both of these esxi hosts.. You could setup a carp for HA if you wanted..
Or you could run 2 different pfsense routers 1 on each host with a transit between them if the networks are different on each esxi host, etc..
Why don't you draw up what your wanting to do and we can discuss the best way to do it.
Completely agree.
The bolded part above to your reponse, is what i was thinking about today when messing around. starting to sink in. I think the challenge though in that model is you need lots of NIC's if the network grows?here is a quick and dirty image (i am terrible with visio) but I hope this gets the idea across.
The idea is to use pfsense to basically route traffic between my networks (internal ESXi networks) as well as to allow my PC and laptop, the ability to get into those networks via https, ssh, RDP etc. But, I don't want those networks (in this case DEV), running a muck all over my home network. They should not be going to my wifi, just outbound access to the internet, HTTPS most likely. Also, i can keep my work VM's separate from my HOME vm's (which would be another network like HOME or something, not on the drawing).
The way I have been doing it is attaching the DPG (WAN and DEV) to the pfsense VM and they come up as interfaces em0 and em1. I have been able to get some network connectivity for my VM's on the DEV network when i do not use VLANS and ssetup the EM1 interface (DEV) as a DHCP server. When I try and create a VLAN in pfsense, that is where i get stuck.
For my layer 2 switch, i plugged the two ESXi nics into ports 14 and 15 and left them to ALL profile (which i believe is trunk in ubiquti terms).
Hoping my on the fly visio works ok.
Also, I am open to recommendations and suggestions. I am not locked into this setup by any means. I just do not want a flat network in my home. :)
Really appreciate the help!
TCG
-
@thecoffeeguy said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
as well as to allow my PC and laptop, the ability to get into those networks via https
Where is pc and laptop in this drawing? Why do you say trunk down to switch - do you have more network hanging off this edge isp router?
Why would that be a trunk? Transit network down to your pfsense sure.. So you have other networks hanging off that switch, and your ER is routing between them? If so pfsense becomes a downstream router and would need to be connected to your ER via a transit or you going to run into asymmetrical routing trying to get some other devices talking to your networks on your esxi hosts.
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
@thecoffeeguy said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
as well as to allow my PC and laptop, the ability to get into those networks via https
Where is pc and laptop in this drawing? Why do you say trunk down to switch - do you have more network hanging off this edge isp router?
Oooo...yea. I forgot that.
My PC is plugged into that same switch as well.
For the router to switch trunk piece, I did setup a WiFi VLAN on my router, that goes to a different network. I have two AP's that are plugged into my switch (AP's and switch are ubiquiti. In the unfi switch configuration, under wireless networks, I specified the VLAN ID (40) which is what i setup on my edgerouter (vlan 40 off eth1 with a DHCP server to devices on the wifi network)Hoping that fills in the gaps i missed.
Thanks
-
Here is the thing there is ZERO need for any vlans in pfsense if all your going to do is have VM only networks..
If your going to NAT pfsense into your lan network where PC is, then for PC to get to vm networks would all be port forwards. If your not going to nat and just make pfsense downstream routers/firewall from your ER, then you need to create a transit network from your ER, and setup the routing there for downstream into networks behind pfsense.
In both of these scenarios I see no need for any vlans in pfsense vms. Where is your vmkern network - assume this is the lan network your PC is in as well and the interface connected from your esxi host.. Take it only has 1 physical interface?
If your esxi host only has the one network connected to it this 192.168.5/24? Your vlan IDs make no matter in our discussion since its all Layer 3.. You have zero reason to be putting vlan IDs on any vswitches or port groups in such a setup. Nor do you have any need of vlans in pfsense.. It would just be virtual nics assigned to your pfsense VM and connected to different port groups/vswitches your other vms are connected to that are not connected to the physical world.
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
Here is the thing there is ZERO need for any vlans in pfsense if all your going to do is have VM only networks..
If your going to NAT pfsense into your lan network where PC is, then for PC to get to vm networks would all be port forwards. If your not going to nat and just make pfsense downstream routers/firewall from your ER, then you need to create a transit network from your ER, and setup the routing there for downstream into networks behind pfsense.
In both of these scenarios I see no need for any vlans in pfsense vms. Where is your vmkern network - assume this is the lan network your PC is in as well and the interface connected from your esxi host.. Take it only has 1 physical interface?
If your esxi host only has the one network connected to it this 192.168.5/24? Your vlan IDs make no matter in our discussion since its all Layer 3.. You have zero reason to be putting vlan IDs on any vswitches or port groups in such a setup. Nor do you have any need of vlans in pfsense.. It would just be virtual nics assigned to your pfsense VM and connected to different port groups/vswitches your other vms are connected to that are not connected to the physical world.
Ok. That makes sense. I had this feeling in the back of my mind that what i was doing, was not really feasible.
So honestly, I could create just two distributed port groups (home and dev) and attach those interfaces to PFSense. THen use pfsense to route traffic between those VM networks and provide NAT to outside world?Any suggestions/recommendations? Specifically on how to get my PC and laptop to access those VM only networks? I see the two options, is one easier to get going than the other?
Anything you can point me to on how to set this up?
Really appreciate your help on this and helping me understand it.
Thank you very muchTCG
EDIT: What about using OSPF? I read a thread where someone is using a similar setup (pfsense downstrream from a edgerrouter x) and is using OSPF for the two talk? Never used OSPF before, but i am always up for learning something new. :)
-
You can run a routing protocol if you want - but its WAY more complicated than simple adding the routes.. There are 2 ways to solve your issue. One you nat into your current network and port forward, other you route into your downstream networks via a transit network off your Edge router.
While I am a HUGE fan of pfsense - why do you not just do the routing/firewalling with your edge router? Why do you think you need pfsense running as VM on your host to route? You already have a router - the edgerouter.
Just add your vm networks as networks off your ER..
Adding a downstream router is more complicated then just doing all your routing with your ER. Or juts replace your ER with pfsense running as VM ;) And do all your routing on your network and to the internet with pfsense.
But here is how you could do it with your transit network
You still didn't tell me how many physical nics your esxi host have, or what your using for the vmkern network.
But in the above setting the vmkern network would be the vlan C network (the transit network)
To make it simple
A 192.168.0/24
B 192.168.1/24
C 172.16.0/29
VM network 1 192.168.2/24
VM network 2 192.168.3/24So ER ip in the transit would be 172.16.0.1
Pfsense 1 wan 172.16.0.2
vmkern 1 172.16.0.3
Pfsense 2 wan 172.16.0.4
vmkern 2 172.16.0.5ER route 192.168.2/24 172.16.0.2
ER route 192.168.3/24 172.16.0.4Both Pfsense wan default route would point to 172.16.0.1
You would TURN off NAT on pfsense routers. You would have to enable NAT on your ER for your downstream VM networks to be able to use the internet.
The blue inside the esxi boxes are clearly all virtual switches and nics..
This allows access to from your PC and wifi network(s) to your esxi vmkern and your downstream networks without any internal natting. The only place your doing any vlan setup/tagging is your switch and ER.
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
You can run a routing protocol if you want - but its WAY more complicated than simple adding the routes.. There are 2 ways to solve your issue. One you nat into your current network and port forward, other you route into your downstream networks via a transit network off your Edge router.
While I am a HUGE fan of pfsense - why do you not just do the routing/firewalling with your edge router? Why do you think you need pfsense running as VM on your host to route? You already have a router - the edgerouter.
Just add your vm networks as networks off your ER..
Adding a downstream router is more complicated then just doing all your routing with your ER. Or juts replace your ER with pfsense running as VM ;) And do all your routing on your network and to the internet with pfsense.
But here is how you could do it with your transit network
You still didn't tell me how many physical nics your esxi host have, or what your using for the vmkern network.
But in the above setting the vmkern network would be the vlan C network (the transit network)
To make it simple
A 192.168.0/24
B 192.168.1/24
C 172.16.0/29
VM network 1 192.168.2/24
VM network 2 192.168.3/24So ER ip in the transit would be 172.16.0.1
Pfsense 1 wan 172.16.0.2
vmkern 1 172.16.0.3
Pfsense 2 wan 172.16.0.4
vmkern 2 172.16.0.5ER route 192.168.2/24 172.16.0.2
ER route 192.168.3/24 172.16.0.4Both Pfsense wan default route would point to 172.16.0.1
You would TURN off NAT on pfsense routers. You would have to enable NAT on your ER for your downstream VM networks to be able to use the internet.
The blue inside the esxi boxes are clearly all virtual switches and nics..
This allows access to from your PC and wifi network(s) to your esxi vmkern and your downstream networks without any internal natting. The only place your doing any vlan setup/tagging is your switch and ER.
Incredibly helpful (and your visio skills are awesome! :) )
Before I forget:
You still didn't tell me how many physical nics your esxi host have, or what your using for the vmkern network.
Total combined useable NIC's betwen my two ESXi hosts is about 9. Vmkern network is using the standard LAN (192.168.1.x) which is where my PC sits.
You know, I was laying in bed last night and I was thinking about exactly what you said:
While I am a HUGE fan of pfsense - why do you not just do the routing/firewalling with your edge router? Why do you think you need pfsense running as VM on your host to route? You already have a router - the edgerouter.
I did think about using just my edgerouter to do everything. I have 3 open ports on it and could use those remaining pieces. I think the reason why I was throwing pfsense in there is that I really like pfsense and want to use more of it. Also, at some point, i need to setup a VPN back into my home network to get into my lab as i travel for work (i travel quite a bit). The need to get in and test stuff out remotely will be critical for me.
In addition to just using my edgerouter, I have thought about replacing it with a pfsense entirely. I would prefer to go a hardware route instead of the VM route (and because I am a geek, would be cool to get two for HA.)
Definitely something I am strongly considering because of my needs and all the functionality I gain with pfsense (more buttons and switches to test/use).
That in mind, and I may move over to the hardware section, but would look for something that has 4+ NIC's in it. Options, I like options. (and look between rackmountable and a small box..i use a ikea lack rack for my switch and patch panel.) Ideally, this would be uber quiet as well.
Really appreciate your help Johnpoz...Super helpful!
Thanks!
TCG
-
So I ran a USG p3 for a short time while I transitioned between pfsense on my OLD esxi box, it could not handle my then new 500/50 internet.. While the USG handled it without issue as long as didn't turn on IPS which disables the hardware offload.
As soon as the sg4860 I wanted back in stock and got it, the USG went on my self.. While I love the price point and that it can handle the 500/50 without much issue... It is no where close to the maturity and ease of use of pfsense..
If you have more interfaces then you for sure could do more with the networking on your esxi than that basic setup I posted.
Maybe something like the sg3100 (pair) of them would be something you could do if you really want HA.. Or the 5100s that are now shipping. I think that is a bit over the top for home setup myself. But since you have 2 esxi hosts and plenty of nics in them - you could for sure do it with the hardware you already have. Or if money not a problem - get yourself a couple of the new XG-7100-1U for your CARP setup ;)
Have Fun!! I predict your ER is not long for this world once you start playing with the power of pfsense ;)
-
@johnpoz said in Using PFSense as a router between two ESXI hosts - VLANS, routing help:
So I ran a USG p3 for a short time while I transitioned between pfsense on my OLD esxi box, it could not handle my then new 500/50 internet.. While the USG handled it without issue as long as didn't turn on IPS which disables the hardware offload.
As soon as the sg4860 I wanted back in stock and got it, the USG went on my self.. While I love the price point and that it can handle the 500/50 without much issue... It is no where close to the maturity and ease of use of pfsense..
If you have more interfaces then you for sure could do more with the networking on your esxi than that basic setup I posted.
Maybe something like the sg3100 (pair) of them would be something you could do if you really want HA.. Or the 5100s that are now shipping. I think that is a bit over the top for home setup myself. But since you have 2 esxi hosts and plenty of nics in them - you could for sure do it with the hardware you already have. Or if money not a problem - get yourself a couple of the new XG-7100-1U for your CARP setup ;)
Have Fun!! I predict your ER is not long for this world once you start playing with the power of pfsense ;)
Yea, I think my edgerouter days will be coming to a end in the very near future.
For now, maybe get a sg3100 with plans to do HA in the future (although reading on CARP, that sounds fun....not sure I could convince the wife on the xg-7100-1U :) ).I forgot to mention, at some point i am going to be adding a storage piece for backups, nfs mounts for my ESXi. THinking that will add 2-4 more NIC's to my setup. SG3100 still be ok for that?
I didnt realze that the SG3100 had 5 NIC's outside the WAN interface.
I have the SG3100 (one for now) in my cart...ready to buy (just making sure right one for now and future stuff (storage)...ready to purchase!!!
