Need some help installing PfSense in ESXi5.5 VM using 3 nics(two networks).



  • I am currently running a Watchguard Firebox x5 Edge behind my Motorola sb6121 cable modem which is handing out two networks…"trusted" and "home" according to the firebox, which are my home and closet lab networks.  Below is the topology diagram.

    This setup has worked beautifully for me but with one problem, a rather big problem.  This firebox is EOL and since it's "free license" I'm limited to only 12 users/connections at a time and I constantly have to reboot the firebox to release those leases so I can connect new stuff.  At any given time I'm constantly at 12 leases.  I have a Dell PowerEdge 860 with Xeon x3220 x4, 8gb of RAM, 250gb SATA HDD, and three gigabit NICs(2x integrated broadcom and 1x PCI intel) just sitting here not being used so I figured PfSense would be a far better option for me.

    I followed these instructions - https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5 - for configuring my ESXi host and creating the VM for PfSense but haven't actually installed it yet because I need to know how to utilize all three NICs and how I need to configure it in such a way that I end up with two separate networks like I have in the above diagram….one will go to my wrt54gs(home network) and the other will go to my netgear gs108 gigabit switch(closet lab/game server hosting/hardware testing network).

    Anyone willing to lend me a hand with this?  8)

    ETA:  I plan to install other VMs on this server so internet connectivity on those is a must.  Having to buy a second PCI NIC isn't a problem if that's what is requires.

    The other VMs will probably just be an e-mail server on the same HDD as PfSense, and various linux distros to play around with on a second HDD.



  • Repeat the process described for "Creating the WAN" but give it a name like LAN2 or HOME.

    Then add that network your pfSense VM and reboot pfSense so that it sees the new network.

    Remember that WAN and OPT interfaces block all traffic by default.



  • So doing this would result in…

    LAN 1 - vmnic0(main connection, currently running to my firebox, using for vSphere Client)

    WAN - vmnic2

    LAN 2 - vmnic1

    Is this correct?



  • Correct.

    Each physical NIC on its own vSwitch with pfSense having a vNIC connected to each of those vSwitches.



  • Now if I throw another VM on this HDD for an e-mail server, and then throw another HDD in for linux distro VMs and gameservers or whatever…how will they access the network/internet?  Do I just assign the LAN NIC I want them to use in their VM settings?  Example would be Linux Mint VM = vmnic1/LAN 2(closet lab/opt network).

    Or would I have to install a fourth NIC(vmnic3) for those other VMs and have it go vmnic1/LAN2 <--> netgear gs108 8port switch <--> vmnic3?



  • You could also add another vSwitch that has no physical NICs attached - see the part about creating a DMZ.

    You can connect any of your VMs to any of your vSwitches.  Just like the real world - without the cables  ;)



  • Sweet, thanks for the help.  I'm gonna go get this installed in the VM now and report back in a bit.



  • Just one other thought and it may be what you intended to do but I would take the WRT out of the picture as a router.  Just use it as an AP.  At the moment it looks like you're doing NAT through both the Firebox and the WRT.  That should be avoided.



  • The wrt54gs is not doing any DHCP or NAT.  It's basically just acting as a switch+AP for my home network.  Notice that the firebox connects to lan1 of the wrt54gs rather than the WAN port.  My main rig connects to lan2.  Everything else on my home network is wirelessly connected to it.

    I plan on having PfSense LAN go to the linksys, running the same config in the linksys I am now, and have PfSense OPT go to the netgear gs108 switch which is what will feed all my closet lab hardware.  I'm assuming here that OPT is a second physical network like it was in the firebox.

    PfSense is installed now.  WAN = em1, LAN = em0, and OPT = em2.  I think those are the right options.  I pretty much followed that install guide exactly, except the part where they don't do the OPT option which I did do since I want two physical networks.



  • Ah, OK.  The labels "LAN 1" and "LAN 2" on the WRT in the diagram threw me off.

    Glad the HowTo was useful.  Adding a second LAN isn't described in there because it's so similar to adding any other virtual network/interface.  I might add something to highlight that though.

    Yes, OPT is another discrete network - not sure "physical" is the right word  :)

    Don't forget that, like WAN, OPT networks have a "block all" rule by default.



  • Thanks for the info.

    Would you happen to have any links to some good, easy to follow documentation for getting PfSense configured now that it's installed?  I want to make sure I have everything on hand for tomorrow when I take my old networks down and get the new networks up.

    The plan is for LAN/em0 to be my home network and OPT/em2 to be my closet lab network.  If I can have them set up the same as I do now…OPT can't see or access anything on LAN, LAN can see and access anything on OPT, and OPT has internet connectivity...that would be ideal.  This way I can administrate everything on OPT from LAN, but LAN is secure from anyone on the OPT network.



  • After about 4 hours of headaches and problems I've finally managed to get this thing to hand out two networks on the 192.168.1.x and 10.0.0.x subnets.  The problem I'm having now is that I'm not getting internet connectivity on either one.  This was tried with PfSense LAN1 > Linksys > PC and PfSense LAN1 > PC, as well as the LAN2 port just to see if I was having issues with one of them.

    Here are some screenshots of the WebGUI…

    Index-

    WAN-

    LAN-

    LAN2-

    Status>Interfaces-

    Like I said before I'm trying to go for the same exact setup I had previously with the firebox as seen in my topology map in the first post.  I want two networks…192.168.1.x and 10.0.0.x...the first being my home network and the second being my lab network.  Both must have internet connectivity.  It would be a plus to have my home network inaccessible from the lab network, but not vice versa because I need to administrate stuff on lab network from home network.

    As I pointed out earlier the Linksys wrt54gs is configured as router, not gateway.  DHCP is disabled.  LAN IP is 192.168.1.1 and WAN IP is 192.168.1.2.  It is acting like switch.  PfSense LAN is plugged in to LAN port 1 on the linksys.  Yes, I did try connecting my rig directly to both LAN and LAN2 on PfSense and still had no internet connectivity.

    DHCP works on LAN and LAN2 and it is handing out addresses in the ranges I set.  Just not getting internet.  Powercycled my modem, PfSense, ESXi host, and my main rig multiple times.



  • Got the firebox back on for now since I needed internet to work.

    I disabled DynamicDNS which appears to have removed that DNS loopback seen in Status > Interfaces but that still hasn't fixed my problem.  Still no internet connectivity on LAN direct or through my linksys, still no internet connectivity on LAN2.  Everything in my screenshots looks fine to me.  I just can't seem to figure this out.  >:(

    ETA:  I removed the gateway from my LAN interface settings because apparently that can cause issues.  I also re-enabled the IPv6 DHCP thinking that may have caused problems since I disabled it.  Still no internet connectivity, and it doesn't appear that I have any internet within the PfSense web admin either.



  • How come you made pfSense 192.168.1.3?  Out of the box it takes 192.168.1.1 as its default LAN address.

    Shouldn't specify a gateway on the LAN.  Sorry missed that.

    Did you set up an allow rule on LAN2?



  • @biggsy:

    How come you made pfSense 192.168.1.3?  Out of the box it takes 192.168.1.1 as its default LAN address.

    Shouldn't specify a gateway on the LAN.  Sorry missed that.

    Did you set up an allow rule on LAN2?

    I actually made PfSense 192.168.1.4 after those screenshots were taken so I could access the webGUI on it from the second NIC in my RIG.  My linksys is LAN-192.168.1.1/WAN-192.168.1.2 and my firebox is LAN-192.168.1.3.

    I removed the gateway from the LAN interface/config page because I read that you don't do this in PfSense.  No change.

    I re-enabled DNSforwarder(originally disabled thinking the DNS 127.0.0.1 was the problem).  No change, but I did learn in general settings you can disable the 127.0.0.1 loopback in your DNS list.

    I enabled (Automatic) on the NAT > Outbound page thinking "oh wow, this has to be the problem!"  No change. :\

    I figured I'd have to set up firewall rules for LAN2 since it doesn't have the default ones but I was going to wait until I could figure out why I wasn't getting internet connectivity on either one(specifically LAN which does have the default rules) first.  LAN2 won't even be used all that much, it's just for testing/playing with gear I bring home and the occasional server hosting when my gaming community servers are down for maintenance.

    Could this be a weird issue with PfSense and the fact it's running inside a VM?  I can't even seem to find anyone else that had this issue.  Usually it was some painfully obvious setting for them that had to be changed.  In my case all my settings look great, even Status > Interfaces looks normal.  Everything looks like it's good to go but the net just won't work inside or outside of PfSense.


  • Rebel Alliance Global Moderator

    And did you reboot your SB6121 once you changed out your firebox for your pfsense VM?  Or looks like you get a IP there on your pfsense wan.. But can you ping the gateway it gives you from pfsense even?

    This is just a no brainer setup to be honest..  It should take no more than a few minutes to setup.. I run very sim setup using esxi on N40L box.  I have never had issue one with it, and initial setup was a breeze.

    You can get fancy with multiple lan segments after - be it they are connected to only other VMs or physical segment as well.

    So in esxi - you should have 2 vswitches one connected to physical nic that is connected to your cable modem, second nic that is going to be your lan network.  I would suggest you change your wrt54g to something other than .1 for starters on the default pfsense network.. You can always change it back to that if you want - but to get it up and running lets not have to deal with changing the pfsense default lan IPs, etc.

    So lets say you changed your wrtg lan IP to 192.168.1.10/24

    That is all working..  Then setup your pfsense VM with its WAN interface connected to your vswitch you that is connected to nic that will connect to your modem.. I change the mac so you for sure now which interface is which in your vm - see blow images. so 01 is WAN and 02 is LAN – so when you setup pfsense your sure which interfaces your connecting to which vswitches.

    So once you have that up and running - and you can hit pfsense on its lan interface using the web gui from your physical network..  Then turn off your modem.  Shutdown your pfsense vm..  Now connect your modem to the nic in your esxi host that is connected to your wan vswitch.. I have to assume your vmkern is connected to same nic as your lan vswitch will be.

    Once your once your modem has rebooted and up and showing sync fire up your pfsense vm..  Bing bang zoom you should have internet and everything golden.

    Now you can play with changing pfsense lan IP to whatever you want or whatever network you want.  And or adding more network segments be them real or virtual.  As you see from the screen shot I have 4 interface in pfsense - wan, lan, wlan and dmz - dmz is not tied to any physical network, etc.

    Let me know happy to help - and have been running such a setup for quite some time, before esxi it was just vmware server, have used virtualbox, etc etc..  I would never go back to running pfsense or physical hardware to be honest - too many advantages to running your firewall/router in VM ;)




  • Sorry for the delayed response.  Just got home from work.

    I always unplug my modem before disconnecting/reconnecting it to anything in order to ensure a new lease, or the correct lease, is applied.

    I can ping the gateway I receive from Mediacom.  Results below.

    PING 173.17.240.1 (173.17.240.1): 56 data bytes
    64 bytes from 173.17.240.1: icmp_seq=0 ttl=255 time=8.030 ms
    64 bytes from 173.17.240.1: icmp_seq=1 ttl=255 time=8.627 ms
    64 bytes from 173.17.240.1: icmp_seq=2 ttl=255 time=9.708 ms
    
    --- 173.17.240.1 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 8.030/8.788/9.708/0.694 ms
    

    Screenshot of ESXi Host > Configuration > Networking-

    Screenshot of PfSense VM > Settings >

    It is possible that I have the names of LAN1 and LAN2 mixed up in that last screenshot but I have them all figured out and working on PfSense so that shouldn't matter, I hope.

    Should I go ahead and change the LAN/WAN IP of my wrt54gs and wipe/reinstall the PfSense VM or do you think it's possible to fix this issue without having to do all that?


  • Rebel Alliance Global Moderator

    Well your lan1 is disconnected from physical nic

    And according to your vmkern port group that nic is connected to a 192.168.1 network (which you have labeled lan2?  Thought lan2 was your 10 network?



  • Like I said I probably have the names reversed in ESXi.  I lost track of which one was which when installing and doing the initial CLI configuration of PfSense(since it uses em0, em1, etc).

    There are two vertically-stacked ports on the back of this server.  1 on the top and 2 on the bottom.  According to PfSense Port 2 = LAN 1 which is 192.168.1.x and is also my vSphere management network, and Port 2 = LAN 2 which is 10.0.0.x.  In ESXi I am pretty sure I have those names reversed but this shouldn't be a problem.  Just confusing is all.  The third port is a PCI NIC and I made that my WAN.

    Using your MAC edit idea this would have been a lot less confusing, haha.  Either way both LAN ports hand out the proper IP from the ranges I set in PfSense webAdmin, and according to webAdmin it's also successfully talking with my modem/mediacom, so I'm not sure why I'm not getting any internet connectivity.


  • Rebel Alliance Global Moderator

    Did you do anything other than set the IPs, lan2 or whatever your second lan is going to need a firewall rules to allow traffic.

    But by default really it is click click on pfsense and you should be up and running on the internet.  There really is nothing to do to have basic internet access - as long as your wan gets an IP and can talk, your normal lan by default has allow all rule and nat is automatic.

    So unless you dicked with something it should be working.

    If you say you can you get IPs from pfsense dhcp, and you can ping pfsense..  And pfsense can talk to its gateway and the internet then there should be no reason why it doesnt work.



  • I followed the install guide for PfSense in an ESXi VM exactly.  I then got in to the webGUI and that's where I added the second LAN(10.0.0.x).  I didn't mess with any settings other than DNSforwarder which I disabled and then reenabled.

    I'm getting a WAN IP, WAN gateway, WAN DNS, I can ping and access PfSense from either LAN NIC directly or through my linksys on LAN 1, DHCP on both networks is working, I can still access vSphere Client on the 192.168.1.x LAN NIC, I can ping the WAN gateway within PfSense webGUI, etc.  There just isn't any internet connectivity in PfSense or on either network, I can't ping or traceroute any outside hosts, etc.

    I'm just as confused as you are, haha.  From what I can see on my end I should have internet and I can see no reason why it wouldn't be working.  I'm starting to wonder if it's my Dell PowerEdge 860 hardware, or the ESXi drivers for that hardware, that are the problem.

    If there are any diagnostics I need to run or screenshots I need to take I'll be awake for about 2-3 more hours.


  • Rebel Alliance Global Moderator

    So are you not resolving dns?  Do a traceroute to say 4.2.2.2  what do you get from a client

    example

    Microsoft Windows [Version 6.1.7601]
    Copyright © 2009 Microsoft Corporation.  All rights reserved.

    C:>tracert -d 4.2.2.2

    Tracing route to 4.2.2.2 over a maximum of 30 hops

    1    1 ms    <1 ms    <1 ms  192.168.1.253
      2    25 ms    28 ms    13 ms  24.13.176.1
      3    11 ms    11 ms    11 ms  68.85.131.149
      4    17 ms    11 ms    11 ms  68.86.197.149

    Juts need past your gateway - see that 24.13.178.1 that is my ISP gateway, I told it not to resolve hostnames with -d but works that way too.

    C:>tracert 4.2.2.2

    Tracing route to b.resolvers.Level3.net [4.2.2.2]
    over a maximum of 30 hops:

    1    <1 ms    <1 ms    <1 ms  pfsense.local.lan [192.168.1.253]
      2    30 ms    29 ms    39 ms  c-24-13-176-1.hsd1.il.comcast.net [24.13.176.1]
      3    12 ms    11 ms    11 ms  te-0-0-0-17-sur03.mtprospect.il.chicago.comcast.net [68.85.131.149]
      4    12 ms    11 ms    11 ms  68.87.230.45
      5    13 ms    15 ms    15 ms  he-2-3-0-0-cr01.chicago.il.ibone.comcast.net [68.86.94.105]

    Here is the thing on your 2nd lan that you put on OPT1, that would NEED to create a firewall rule.  What does your traceroute look like from a client on your 192.168.1.0/24 network

    I run mine on esxi 5.5 - there is nothing special you have to do..

    Did you reboot your vm after you have changes its lan IP, etc.  I have heard of people having issues when they change their lan network.. And you have nat on auto right?

    Question are you running 32bit or 64 bit?  I run

    2.1-RELEASE (i386)
    built on Wed Sep 11 18:16:50 EDT 2013
    FreeBSD 8.3-RELEASE-p11

    There is little point to running the BUGGY as shit if you watch the forums ;) 64bit unless your going to give it more than 4GB..



  • I'm running 2.1-RELEASE (i386), the latest one from the FTP.  I read that 64bit was buggy and not worth running since PfSense doesn't require enough RAM to justify running 64bit.

    I'll get those traceroute results from this rig on LAN1 of PfSense here in a few mins.  Got a big update finishing up on something and I need that before I can swap the modem around.  ;)


  • Rebel Alliance Global Moderator

    What adapter you using - I use just e1000 it works fine.  What settings do you have on your vswitches?  This is my wan vswitch and lan vswitch






  • I'm using whatever the defaults were.  I know for sure the adapter type is E1000.

    Here is a screencap of ipconfig /all and tracert -d 4.2.2.2 from this rig plugged directly in to LAN1-



  • I notice in the DNS field it's pulling the gateway/LAN IP of PfSense rather than the two DNS Mediacom usually gives me.

    If I run an ipconfig /all on my rig on the firebox network right now I get…

    DNS Servers . . . . . . . . . . . : 97.64.183.164
                                                97.64.209.37

    This seems like a step in the right direction unless I'm misunderstanding something here.


  • Rebel Alliance Global Moderator

    see hop 3.. Did you set a gateway on your lan or something?

    You do not set gateways on lan interfaces!



  • There was one set and I removed it after finding out that I shouldn't.


  • Rebel Alliance Global Moderator

    Dude there is no possible way it should ever say that in a traceroute - unless it thought it needed to go out that interface to get somewhere.  Can you post up your route table – here is mine.

    See the default going out my ISP connection.




  • Sure, give me a few mins.



  • Ok so, I'm currently posting while online from PfSense.

    Even though there were no gateways set on the LAN1 or LAN2 pages, on the Gateways page there were two different LAN gateways there for some reason.  I deleted those, rebooted PfSense to continue towards getting you that screenshot and I hear Teamspeak say "Connected."

    I'm going to connect my wrt54gs and see if all my stuff on my home network comes online.  Brb.



  • My home network is now online and working.  Full network and internet connectivity.

    Still need to set those firewall rules for the second LAN which I'm not really sure how to do yet, but I'll cross that bridge when I get to it.  I don't even have my switches here yet and currently have no hardware to play with on that "lab network" so it's no big deal right now.  I just wanted my home network running and not being hindered by that firebox's 12 user limit.

    Thank you so much for your help!



  • Meh, I was on a roll so I decided to do the LAN2 firewall rules before bed.  Do these look ok?  I want internet connectivity on LAN2 but I don't want it to see or access LAN1.  However, I do want LAN1 to see and access LAN2 since I administrate most stuff on LAN2 from my rig on LAN1.



  • You only need one rule on LAN2:

    PASS
    Proto:  IPv4+IPv6
    Source: ANY
    S/Port: ANY
    Dest:  NOT LAN net
    D/Port: ANY

    This allows through any traffic coming in on pfSense's LAN2 interface as long as it does not have a destination address somewhere in the LAN subnet.



  • Are my rules set up properly for what I wanted on LAN2?  Curious to know if I got anywhere close on those since I've never messed with firewall rules before, outside of Windows.  :-[

    Here is what I have now for LAN2…

    Pass
    ID: None
    Proto: IPv4+6 TCP
    Source: *
    Port: *
    Destination: ! LAN net
    Port: *
    Gateway: *
    Queue: None
    Schedule:
    Description: Allow LAN2 to any except LAN.


  • Rebel Alliance Global Moderator

    So does your lan2 have ipv6 on it, if not prob want to just say ipv4, your source could be just lan2 net since what else would be coming into your lan2 interface?

    But other than that looks right..

    So example here are my wlan rules, this is is like a lan2 in your case - but I have another segment dmz besides my lan.

    So the couple of pin holes I have - so ipad at the 2.230 address can go anywhere, lan, internet and dmz
    so wlan can talk to my printer on lan at 1.50
    so wlan can talk to my ntp server on lan at 1.40
    so wlan can talk to internet and dmz, but not the lan network.

    Hope that gives you some ideas how you would do rules that allow some traffic to start from your lan2 into your lan for exceptions, etc.

    Now from my other segment the dmz, I have an alias called locals which has my lan and wlan network segments 192.168.1.0/24 and 192.168.2.0/24 in it

    So this rules says dmz can go anywhere as long as its not either of those networks.  Now if I want to allow dmz to talk to my printer or ntp I could put the same kind of rules I have in my wlan segment above that rule.  So currently it can go to the internet, it could talk to my openvpn clients etc.  But could not create traffic to either my lan or wlan segments.

    Keep in mind both wlan and dmz could answer traffic that comes from lan..  Since my lan rules are allow any.






  • I'm curious about adding wifi(for home network) in to this thing and possibly doing away with my wrt54gs.  My PE860 has one empty PCI slot.  What are my options?  Is there a compatible card list?  The wrt54gs is B/G and WPA2/AES so I'd at least want those specs with the card.  I'd lose my 'switch' on my home network but I've got two Netgear GS108's coming in that'll go on each network.

    This weekend I'll be attempting a PfSense install and config on a Watchguard Firebox X-Core x700 for a friend.  Currently trying to find a good mini-PCI wireless adapter to throw in it so I can do wireless for him as well.  He was the original owner of this PE860 and ran PfSense on it(standalone, not VM) but his wife didn't like the noisy fans in it.  This x700 should work nicely for him especially if I can get wireless going on it. ;)


  • Rebel Alliance Global Moderator

    I would never do wifi directly on pfsense - doing wireless in my mind is not the role of firewall/router - its is to firewall traffic and route..  Wireless is job of AP..



  • @johnpoz:

    Wireless is job of AP..

    Agreed.  You should be able to turn your WRT into an AP for the short term.


  • Netgate Administrator

    I have a mini-pci adapter in my firebox at home. It's this:
    http://wikidevi.com/wiki/Toshiba_PA3458U-1MPC
    Any similar Atheros card of that age should be good. It cost me nothing, I had it gathering dust, it provides useful out of band access when I unplug the wrong cable somewhere and also means I can see channel usage via the webgui. But.. I also have external APs that are much faster and give much better coverage.

    Steve