PfSense/ESXi newbie config advice/questions

dxun

As a small side/hobby-project, I am developing a small home network and I would like to ask for advice and your insights.

This is the first time I am setting up something like this so don't be surprised with n00b questions :)

Here's the story so far.
As you can see on (LAN_overview_to_be.png), this is my current state of affairs: I have two LAN subnets and a virtualized pfSense 2.1.5 acting as classic router/firewall/dns/dhcp machine. Since I want to separate my roomates' network (LAN subnet 2 - L2) from my own (LAN subnet 1 - L1) as much as possible, L2 has a number of blacklisted IPs on L1. This is currently the only rule in place, L2 can currently access everything else and so can L1.

My network hardware config can be seen on (ESXi_network_adapters.PNG) - a quad-nic setup, with one NIC currently unused. You can also see my current ESXi networking setup (ESXi_networking.PNG - note the currently dangling DMZ down on vSwitch3), as well as legend to the right.

I've sketched out where what I eventually want to achieve on (LAN_overview_to_be.PNG) - essentially I'd like to have:

a web VM mostly dedicated to web-serving duties (so it would be essentially publicly accessible on ports 80, 8080 and perhaps a few others, depending on what else would end up there) - I am guessing it would be in the DMZ
a database VM - accessible only from web VM and L1 - so my guess is it would be outside DMZ and protected by firewall rules
a sandbox Windows VM - accessible from L1 only - again, outside DMZ
a dedicated proxy VM - this would be in the DMZ, if I chose to go this way

I am still kinda on the fence with 4) - my current options are either running squid on pfSense (to which I have no special objections) or separating the proxy out (which would be more "the-proper-way-to-it").
In addition, I am not sure how to implement the DMZ in this setup - I have read the pfsense doc on this but it only explains how to create the DMZ and "add" it to pfSense but I don't understand the way I should configure nor how would I "join" VMs to it or prevent them from participating in DMZ.

I'll appreciate any help, thoughts, opinions you may have - I could hack my way through all this but I would rather ask before I spend more time on this because I have a feeling things are going to be progressively more complicated to fix as I go down the path.

Finally, a disclaimer - yes, I am aware that running pfSense on ESXi is not industrial-grade safety; I acknowledge that and find this a sufficient level of safety for my home purposes.

Thanks in advance

LAN_overview_as_is.png_thumb

ESXi_network_adapters.PNG_thumb

ESXi_networking.PNG_thumb

LAN_overview_to_be.png_thumb

johnpoz

More than happy to help you with any questions you might have with running pfsense and other vms on a esxi box. I been running such a setup for years.

I do have one question - why do you have vmkern on multiple interfaces? you have one at 1.100 and 2.10 - Do you really have to manage the esxi host from both those networks, why can you not hit the 1.100 from the 2.0/24 segment through pfsense?

I would also break out the vmkern to its own interface - from what I have seen sharing vmkern with normal port group restricts performance to and from the datastore for example with uploading files.

See for example my networking in esxi, I also have 4 physical interfaces - one for wan (comcast cable) 1 for my wired network, and then 1 for my 2nd segment that I call wlan since really only thing on this is my access point, other then a wired printer just to make it easier for wireless clients to get to it via airprint (mdns). And the wireless controller (unifi) sets on that segment as well. Since I don't have need for 3rd actual segment I broke out vmkern to its own interface. Seems to me your in the same setup - doesn't look like your using the 4 physical interface actually?

As to DMZ - its just another interface on pfsense. Create a new vswitch, call it dmz either connect it to real world with a physical interface or just connect vms too it.. Se my example couple boxes connected to the dmz segment - the win7 and xp box - and then the dmz leg for pfsense.

esxinetwork.png_thumb

dxun

Ah, good catch. It doesn't, not any longer - it was a leftover since my initial experiments; I tried to understand what actually influences the Observerd IP Ranges on physical adapters (link).

I understand vmnic3 has gotten its IP range from my ISP but I cannot understand what had configured vmnic0-2 to exactly those ranges, should I change that and where would I change it at all. That's why at one point I surmised perhaps it's VM Kernel network that influences that (I understand now it doesn't and it only specifies the address at which vSphere is available to be connected with vSphere Client). I still don't know what the correct answer is here.

This is what I ended up configuring - as far as I could see this isn't exactly textbook config but I believe it should suffice for now. The main criticism I'd have in this setup would be the database and web server are in the same (permiter) subnet. My understanding from what I've seen is to have DB server (and probably code repo) not exposed to any publicly available subnets but to create a separate subnet only accessible from my internal subnets. Then web server would point to db server in that new subnet and so would any clients wanting to commit code to the code repo. Would you agree with with that? If yes, how do you propose I implement it? If not, what would be the alternative?

However, I ran out of physical NICs for such a setup and I cannot grasp how would I make such a setup with vSwitches only (as far as I understand, it should be possible). I don't understand how such a setup would ever work since there is no physical NIC:

where does the actual traffic go?
does the ESXi itself act here as the router? do I need physical router (pfSense) at all then?
how is this new connection represented on router (pfSense) and do I have to treat is specially?

johnpoz

Why do you need a physical nic to connect VMs? think of your vswitch as just a normal switch with the physical nic just being a connection to the real world switch.

All your VMs can talk as long as they are connected to the same vswitch, or if there is a router connected between the vswitches - pfsense with a vnic in connected to each switch. As long as one of the legs as tied to real world with physical nic, then even the physical world can connect to the virtual connected only vms via pfsense.

I wouldn't worry too much about the discovered IP ranges. Kind of a useless feature if you ask me ;) But it determines it by broadcast
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006744

If you don't have cdp or llmr switch you could do this to get it set how you want it to your network, etc..
http://sostechblog.com/2012/08/13/vsphere5-setting-the-observed-ip-range/

So back to your physical nic - do you have boxes that you want in this DMZ that are physical? If not then why do have physical nic on that vswitch? See my w7 box there in my dmz..

I can ping it from my lan segment from a physical machine


C:\>ipconfig                                                            

Windows IP Configuration                                                

Ethernet adapter Local:                                                 

   Connection-specific DNS Suffix  . :                                  
   IPv4 Address. . . . . . . . . . . : 192.168.1.100                    
   Subnet Mask . . . . . . . . . . . : 255.255.255.0                    
   Default Gateway . . . . . . . . . : 192.168.1.253                    

C:\>ping w7x64-vm                                                       

Pinging w7x64-vm.local.lan [192.168.3.206] with 32 bytes of data:       
Reply from 192.168.3.206: bytes=32 time=1ms TTL=127                     
Reply from 192.168.3.206: bytes=32 time=6ms TTL=127                     
Reply from 192.168.3.206: bytes=32 time<1ms TTL=127                     
Reply from 192.168.3.206: bytes=32 time=1ms TTL=127                     

Ping statistics for 192.168.3.206:                                      
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),                
Approximate round trip times in milli-seconds:                          
    Minimum = 0ms, Maximum = 6ms, Average = 2ms                         

C:\>tracert w7x64-vm

Tracing route to w7x64-vm.local.lan [192.168.3.206]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  pfsense.local.lan [192.168.1.253]
  2    <1 ms    <1 ms    <1 ms  W7X64-VM.local.lan [192.168.3.206]

And it can talk to the internet - but it can not talk to my other segments because I have that blocked.

If you don't have a need for a physical devices to be on a specific segment you don't need a physical nic on it. Which you could then prob put lan on its own vs sharing with your vmkern