Need some help installing PfSense in ESXi5.5 VM using 3 nics(two networks).
-
I am currently running a Watchguard Firebox x5 Edge behind my Motorola sb6121 cable modem which is handing out two networks…"trusted" and "home" according to the firebox, which are my home and closet lab networks. Below is the topology diagram.
This setup has worked beautifully for me but with one problem, a rather big problem. This firebox is EOL and since it's "free license" I'm limited to only 12 users/connections at a time and I constantly have to reboot the firebox to release those leases so I can connect new stuff. At any given time I'm constantly at 12 leases. I have a Dell PowerEdge 860 with Xeon x3220 x4, 8gb of RAM, 250gb SATA HDD, and three gigabit NICs(2x integrated broadcom and 1x PCI intel) just sitting here not being used so I figured PfSense would be a far better option for me.
I followed these instructions - https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5 - for configuring my ESXi host and creating the VM for PfSense but haven't actually installed it yet because I need to know how to utilize all three NICs and how I need to configure it in such a way that I end up with two separate networks like I have in the above diagram….one will go to my wrt54gs(home network) and the other will go to my netgear gs108 gigabit switch(closet lab/game server hosting/hardware testing network).
Anyone willing to lend me a hand with this? 8)
ETA: I plan to install other VMs on this server so internet connectivity on those is a must. Having to buy a second PCI NIC isn't a problem if that's what is requires.
The other VMs will probably just be an e-mail server on the same HDD as PfSense, and various linux distros to play around with on a second HDD.
-
Repeat the process described for "Creating the WAN" but give it a name like LAN2 or HOME.
Then add that network your pfSense VM and reboot pfSense so that it sees the new network.
Remember that WAN and OPT interfaces block all traffic by default.
-
So doing this would result in…
LAN 1 - vmnic0(main connection, currently running to my firebox, using for vSphere Client)
WAN - vmnic2
LAN 2 - vmnic1
Is this correct?
-
Correct.
Each physical NIC on its own vSwitch with pfSense having a vNIC connected to each of those vSwitches.
-
Now if I throw another VM on this HDD for an e-mail server, and then throw another HDD in for linux distro VMs and gameservers or whatever…how will they access the network/internet? Do I just assign the LAN NIC I want them to use in their VM settings? Example would be Linux Mint VM = vmnic1/LAN 2(closet lab/opt network).
Or would I have to install a fourth NIC(vmnic3) for those other VMs and have it go vmnic1/LAN2 <--> netgear gs108 8port switch <--> vmnic3?
-
You could also add another vSwitch that has no physical NICs attached - see the part about creating a DMZ.
You can connect any of your VMs to any of your vSwitches. Just like the real world - without the cables ;)
-
Sweet, thanks for the help. I'm gonna go get this installed in the VM now and report back in a bit.
-
Just one other thought and it may be what you intended to do but I would take the WRT out of the picture as a router. Just use it as an AP. At the moment it looks like you're doing NAT through both the Firebox and the WRT. That should be avoided.
-
The wrt54gs is not doing any DHCP or NAT. It's basically just acting as a switch+AP for my home network. Notice that the firebox connects to lan1 of the wrt54gs rather than the WAN port. My main rig connects to lan2. Everything else on my home network is wirelessly connected to it.
I plan on having PfSense LAN go to the linksys, running the same config in the linksys I am now, and have PfSense OPT go to the netgear gs108 switch which is what will feed all my closet lab hardware. I'm assuming here that OPT is a second physical network like it was in the firebox.
PfSense is installed now. WAN = em1, LAN = em0, and OPT = em2. I think those are the right options. I pretty much followed that install guide exactly, except the part where they don't do the OPT option which I did do since I want two physical networks.
-
Ah, OK. The labels "LAN 1" and "LAN 2" on the WRT in the diagram threw me off.
Glad the HowTo was useful. Adding a second LAN isn't described in there because it's so similar to adding any other virtual network/interface. I might add something to highlight that though.
Yes, OPT is another discrete network - not sure "physical" is the right word :)
Don't forget that, like WAN, OPT networks have a "block all" rule by default.
-
Thanks for the info.
Would you happen to have any links to some good, easy to follow documentation for getting PfSense configured now that it's installed? I want to make sure I have everything on hand for tomorrow when I take my old networks down and get the new networks up.
The plan is for LAN/em0 to be my home network and OPT/em2 to be my closet lab network. If I can have them set up the same as I do now…OPT can't see or access anything on LAN, LAN can see and access anything on OPT, and OPT has internet connectivity...that would be ideal. This way I can administrate everything on OPT from LAN, but LAN is secure from anyone on the OPT network.
-
After about 4 hours of headaches and problems I've finally managed to get this thing to hand out two networks on the 192.168.1.x and 10.0.0.x subnets. The problem I'm having now is that I'm not getting internet connectivity on either one. This was tried with PfSense LAN1 > Linksys > PC and PfSense LAN1 > PC, as well as the LAN2 port just to see if I was having issues with one of them.
Here are some screenshots of the WebGUI…
Index-
WAN-
LAN-
LAN2-
Status>Interfaces-
Like I said before I'm trying to go for the same exact setup I had previously with the firebox as seen in my topology map in the first post. I want two networks…192.168.1.x and 10.0.0.x...the first being my home network and the second being my lab network. Both must have internet connectivity. It would be a plus to have my home network inaccessible from the lab network, but not vice versa because I need to administrate stuff on lab network from home network.
As I pointed out earlier the Linksys wrt54gs is configured as router, not gateway. DHCP is disabled. LAN IP is 192.168.1.1 and WAN IP is 192.168.1.2. It is acting like switch. PfSense LAN is plugged in to LAN port 1 on the linksys. Yes, I did try connecting my rig directly to both LAN and LAN2 on PfSense and still had no internet connectivity.
DHCP works on LAN and LAN2 and it is handing out addresses in the ranges I set. Just not getting internet. Powercycled my modem, PfSense, ESXi host, and my main rig multiple times.
-
Got the firebox back on for now since I needed internet to work.
I disabled DynamicDNS which appears to have removed that DNS loopback seen in Status > Interfaces but that still hasn't fixed my problem. Still no internet connectivity on LAN direct or through my linksys, still no internet connectivity on LAN2. Everything in my screenshots looks fine to me. I just can't seem to figure this out. >:(
ETA: I removed the gateway from my LAN interface settings because apparently that can cause issues. I also re-enabled the IPv6 DHCP thinking that may have caused problems since I disabled it. Still no internet connectivity, and it doesn't appear that I have any internet within the PfSense web admin either.
-
How come you made pfSense 192.168.1.3? Out of the box it takes 192.168.1.1 as its default LAN address.
Shouldn't specify a gateway on the LAN.Sorry missed that.Did you set up an allow rule on LAN2?
-
How come you made pfSense 192.168.1.3? Out of the box it takes 192.168.1.1 as its default LAN address.
Shouldn't specify a gateway on the LAN.Sorry missed that.Did you set up an allow rule on LAN2?
I actually made PfSense 192.168.1.4 after those screenshots were taken so I could access the webGUI on it from the second NIC in my RIG. My linksys is LAN-192.168.1.1/WAN-192.168.1.2 and my firebox is LAN-192.168.1.3.
I removed the gateway from the LAN interface/config page because I read that you don't do this in PfSense. No change.
I re-enabled DNSforwarder(originally disabled thinking the DNS 127.0.0.1 was the problem). No change, but I did learn in general settings you can disable the 127.0.0.1 loopback in your DNS list.
I enabled (Automatic) on the NAT > Outbound page thinking "oh wow, this has to be the problem!" No change. :\
I figured I'd have to set up firewall rules for LAN2 since it doesn't have the default ones but I was going to wait until I could figure out why I wasn't getting internet connectivity on either one(specifically LAN which does have the default rules) first. LAN2 won't even be used all that much, it's just for testing/playing with gear I bring home and the occasional server hosting when my gaming community servers are down for maintenance.
Could this be a weird issue with PfSense and the fact it's running inside a VM? I can't even seem to find anyone else that had this issue. Usually it was some painfully obvious setting for them that had to be changed. In my case all my settings look great, even Status > Interfaces looks normal. Everything looks like it's good to go but the net just won't work inside or outside of PfSense.
-
And did you reboot your SB6121 once you changed out your firebox for your pfsense VM? Or looks like you get a IP there on your pfsense wan.. But can you ping the gateway it gives you from pfsense even?
This is just a no brainer setup to be honest.. It should take no more than a few minutes to setup.. I run very sim setup using esxi on N40L box. I have never had issue one with it, and initial setup was a breeze.
You can get fancy with multiple lan segments after - be it they are connected to only other VMs or physical segment as well.
So in esxi - you should have 2 vswitches one connected to physical nic that is connected to your cable modem, second nic that is going to be your lan network. I would suggest you change your wrt54g to something other than .1 for starters on the default pfsense network.. You can always change it back to that if you want - but to get it up and running lets not have to deal with changing the pfsense default lan IPs, etc.
So lets say you changed your wrtg lan IP to 192.168.1.10/24
That is all working.. Then setup your pfsense VM with its WAN interface connected to your vswitch you that is connected to nic that will connect to your modem.. I change the mac so you for sure now which interface is which in your vm - see blow images. so 01 is WAN and 02 is LAN – so when you setup pfsense your sure which interfaces your connecting to which vswitches.
So once you have that up and running - and you can hit pfsense on its lan interface using the web gui from your physical network.. Then turn off your modem. Shutdown your pfsense vm.. Now connect your modem to the nic in your esxi host that is connected to your wan vswitch.. I have to assume your vmkern is connected to same nic as your lan vswitch will be.
Once your once your modem has rebooted and up and showing sync fire up your pfsense vm.. Bing bang zoom you should have internet and everything golden.
Now you can play with changing pfsense lan IP to whatever you want or whatever network you want. And or adding more network segments be them real or virtual. As you see from the screen shot I have 4 interface in pfsense - wan, lan, wlan and dmz - dmz is not tied to any physical network, etc.
Let me know happy to help - and have been running such a setup for quite some time, before esxi it was just vmware server, have used virtualbox, etc etc.. I would never go back to running pfsense or physical hardware to be honest - too many advantages to running your firewall/router in VM ;)
-
Sorry for the delayed response. Just got home from work.
I always unplug my modem before disconnecting/reconnecting it to anything in order to ensure a new lease, or the correct lease, is applied.
I can ping the gateway I receive from Mediacom. Results below.
PING 173.17.240.1 (173.17.240.1): 56 data bytes 64 bytes from 173.17.240.1: icmp_seq=0 ttl=255 time=8.030 ms 64 bytes from 173.17.240.1: icmp_seq=1 ttl=255 time=8.627 ms 64 bytes from 173.17.240.1: icmp_seq=2 ttl=255 time=9.708 ms --- 173.17.240.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 8.030/8.788/9.708/0.694 ms
Screenshot of ESXi Host > Configuration > Networking-
Screenshot of PfSense VM > Settings >
It is possible that I have the names of LAN1 and LAN2 mixed up in that last screenshot but I have them all figured out and working on PfSense so that shouldn't matter, I hope.
Should I go ahead and change the LAN/WAN IP of my wrt54gs and wipe/reinstall the PfSense VM or do you think it's possible to fix this issue without having to do all that?
-
Well your lan1 is disconnected from physical nic
And according to your vmkern port group that nic is connected to a 192.168.1 network (which you have labeled lan2? Thought lan2 was your 10 network?
-
Like I said I probably have the names reversed in ESXi. I lost track of which one was which when installing and doing the initial CLI configuration of PfSense(since it uses em0, em1, etc).
There are two vertically-stacked ports on the back of this server. 1 on the top and 2 on the bottom. According to PfSense Port 2 = LAN 1 which is 192.168.1.x and is also my vSphere management network, and Port 2 = LAN 2 which is 10.0.0.x. In ESXi I am pretty sure I have those names reversed but this shouldn't be a problem. Just confusing is all. The third port is a PCI NIC and I made that my WAN.
Using your MAC edit idea this would have been a lot less confusing, haha. Either way both LAN ports hand out the proper IP from the ranges I set in PfSense webAdmin, and according to webAdmin it's also successfully talking with my modem/mediacom, so I'm not sure why I'm not getting any internet connectivity.
-
Did you do anything other than set the IPs, lan2 or whatever your second lan is going to need a firewall rules to allow traffic.
But by default really it is click click on pfsense and you should be up and running on the internet. There really is nothing to do to have basic internet access - as long as your wan gets an IP and can talk, your normal lan by default has allow all rule and nat is automatic.
So unless you dicked with something it should be working.
If you say you can you get IPs from pfsense dhcp, and you can ping pfsense.. And pfsense can talk to its gateway and the internet then there should be no reason why it doesnt work.