Newbie to pfSense and ESXi, need network setup recommendation

tedSmith

Hi all,

I am a complete newbie, so please bear with me. I am trying to setup an ESXi All-In-One box and currently have these available
a). 2 external (dynamic) IP addresses ( consider this as two different ISPs )
b). One ESXi server with 2 physical NICs
c). One retail router
d). One VM running OpenSolaris hosting my DIY NAS
e). One VM running pfSense

What I would like to achieve is …
a). The retail router takes one of the external IP and handle all internet traffic connected to that router ( eg. Laptop via WiFi and PC via LAN )
b). Access my NAS hosted on the VM on my laptop and PC
c). One NIC connected to the second external IP and handle all internet traffic for all VMs running on ESXi
d). VPN access via pfSense to access my NAS when I am aboard
e). Add an additional VM to be my web server that can be accessed via the second external IP where some of the files are hosted on data shares that I can also access from my laptop and PC
f). Add additional VM that I can access via VNC externally via VPN
g). vSphere and pfSense management not accessible externally for security

What I have figured so far …
a). one ISP being used by the retail router as 192.168.1.1
b). one NIC of the ESXi connected to the router as static 192.168.1.200 as the Management Network
c). the same NIC in (b) is also on vSwitch0 ( my LAN Virtual Machine Port Group )
d). OpenSolaris VM is connected to vSwitch0
e). second NIC of the ESXi is connected to vSwitch1 ( my WAN Virtual Machine Port Group )
f). pfSense VM is connected to both vSwitch0 and vSwitch1. Connection to vSwitch0 ( LAN ) is assuaged as 192.168.1.205

I haven't figured out how to do the remaining and maybe I have got down the wrong path already. Would be great if someone can give me some advice on how best to achieve what I wanted. Ultimately, I would like this to be my platform for my small home business, therefore, I would very much like one ISP being used for business traffics and one for home traffic, but yet able to share the cost of the NAS disks. Having said that, I am also worry about if the sharing of the NAS would create security for my home data.

Thanks in advance.

Ted

biggsy

Hi Ted,

Not sure if it will help but have you seen this: http://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5

A couple of questions:

Can you squeeze another NIC (or two) into your ESXi host? I think this would make the whole thing a lot easier to set up.

Are the two external IPs actually from two different ISPs? The first ISP is handled by your external router but how are you connected to the second?

biggsy

tedSmith

Hi Biggsy,

Yeah, I was actually using that guide to help me install pfSense on ESXi, that is how I figured out the vSwitch0 and vSwitch1 idea. I got confused with the DMZ part, not knowing where it exactly fits in my case. I am guessing if I create a VM to be my web server, it will sit in there, but I am unsure if it will be able to get access to my NAS as the way I wanted and how I need to config pfSense for that ( which the guide hasn't written ).
When you say squeezing more NIC in the ESXi, do you mean physical or virtual ? I am happy to add as many virtual NICs necessary to get the job done, but probably not for physical NIC. In case it is physical NIC, how would that work ? Effectly, all the "business side" of what I want to do is virtual on the ESXi, so you are saying that I should use a physical bridge to gap between "home side" and "business side" with physical NIC ? Sorry, I am confused.
As for the IPs, they are technically the same ISP, just different plans. The "business side" has more upload limits than the "home side". They are physically two separate gigabit ethernet ins, which I though by saying two ISPs is easer for people to understand. I did consider the idea of just making them two separated network connected together via VPN but that is just a waste of resources ( they are physically one body length apart ! ) plus it will slow down how I transfer files to my NAS in the home use case.

Hope I cleared up my issue a little. Thanks

Ted

biggsy

Sorry, maybe I misunderstood your initial post but I'm still a bit confused.

As I read it, you would have to deal with double-NAT in your current design - that's an unnecessary complication that could see you spending a lot of time figuring out why some thing doesn't work.

If both ISP connections are just gigabit Ethernet connections (with the retail router attached to one of them) I would be inclined to add a NIC to the ESXi host and take those two ISP connections in to the pfSense VM as WAN1 and WAN2 - i.e., no retail router in the way. Each of those WANs would then have one of the dynamic IP addresses. That would give you the flexibility to route (and secure) everything using the pfSense VM. If you want to have a Home LAN and Business LAN they would probably be best on separate interfaces as well. Hence the question about squeezing in a NIC or two.

I'm guessing that the retail router is providing the wifi connection for your laptop . It could probably continue to do that but, as an Access Point on either your business or home LAN, not as a router.

The DMZ is just another network that you can make accessible from both LAN and WAN so, for example, you can fully manage the web server from the LAN but on the WAN (Internet) side it's only serving content (with firewall rules to allow, say, only HTTP and HTTPS from the Internet). With a virtual DMZ you don't need physical interfaces to set up that network. Your NAS could sit in the same DMZ network or a separate one. If it was separate you can control what access is available between it and the web server, again by using firewall rules.

biggsy

A picture is worth a thousand words. Here's one I prepared earlier ;)

Two WANs (red), one LAN (green) and one DMZ (orange) in this diagram but it might help.

Ignore the management NIC.

![2013-01-09 22-41-49.png](/public/imported_attachments/1/2013-01-09 22-41-49.png)
![2013-01-09 22-41-49.png_thumb](/public/imported_attachments/1/2013-01-09 22-41-49.png_thumb)

tedSmith

Thanks biggsy. Yeah, I think 10mins after I send out my last reply, I figured out how you might want to use the extra NICs.
Thanks for all the info, I was hoping to keep the "home network" as is without doing much, but I guess that is not really possible then.
Just to follow up from your design ( since you are so nice :P ), would each of the vSwitch be running off different subnet (eg. one be 192.168.1.x, another in 192.168.2.x …etc. ) and pfSense can handle the cross routing ? Are there any doco that I can reference for something like that ?
Thanks again.

Slam

I would agree with what Biggsy said.

I started off with a similar setup a while ago, after weeks of head scratching and trying to visualise how things would be and possibly expand, I ended up getting a quad NIC for my ESXI host, however I went one step further and also got a 8 port managed pswitch, this might not be suitable for you, but Ill explain.

I also have 2 wan's and it was simply a case of configuring vlans on the pswitch and also on my ESXI host, vswitch0 for wan1 and vswitch1 for wan2 (I didnt bother with vlans on pfsense) and then plugged my modems directly to the pswitch (I can add more wans later if I want) , I had another vswitch for my DMZ - webserver and freenas and a vswitch for my LAN.

My current setup consists now of 2 pswitches with intervlan configured, 2 ESXI hosts (7 NIC each) and a cheap mans SAN, not to forget also 8 vm's on 24/7 and three times as much vm's for testing/fun.

Now with your setup, as suggested get at least another NIC, get rid of double NAT and set the modems in bridge mode if possible, all this will save a lot of headache/trouble shooting later on when you encounter problems. Also you dont have to have a managed pswitch for your own setup for this to work or vlans, but I mentioned it to give you a scenario and its something to consider for the future if you envisage your network growing, having one just saves a lot of work later on.

Yes, each vswitch would be running off a different subnet thats configured within pfsense, of course you would have to create all the vswitches that you need within ESXI and add these to the pfsense vm under "hardware".

biggsy

I was hoping to keep the "home network" as is without doing much, but I guess that is not really possible then.

Not so much "not really possible" but certainly more complicated and likely to be a source of problems. There is a small disruption factor in taking the retail router out of the picture but, with planning, it can be minimized.

In the diagram, the two pfSense WAN interfaces would get your two ISP-provided dynamic IP addresses. The LAN could be 192.168.1.0/24 (pfSense LAN being 192.168.1.1 by default) and the DMZ could be 192.168.2.0/24. You would manage routing and filtering of all traffic between networks and virtual hosts through pfSense - that's what it's designed to do.

Abdsalem's point is good in that VLANs could save you the cost of real NICs - but at the cost of a managed switch and some extra configuration. Last time I looked, a pair of gigabit Intel NICs would cost you less than a decent managed switch. If you don't have the spare slots for additional NICs, though, VLANs and a managed switch might be the way to go. However, dual-port and quad-port NICs can be relatively expensive.

As they're from the same ISP, check that your two WAN links don't use the same gateway IP address. That could be an issue.

tedSmith

Thanks guys. Sorry, just wanted to confirm that I understand this correctly. In the diagram "MGMT" is the management network to ESXi and that is a physical NIC but not hooked to pfsense ? If that is the case, does that mean the only way I get access to it is by physical connection to that port with a laptop ?
Also, considering if this is too complicated for me. What if I drop my "home side" ISP ? So only one external IP, the setup would be exactly the same without only on WAN and my router will only be a switch for my physical LAN and Wifi ?
Sorry another very newbie question … what is a pswitch ? How is that different from a normal switch ?

biggsy

In the diagram the MGMT network is a separate NIC that's connected to the VMkernel Port Group (the management network) and nothing else. That's just the way I did it because I had a spare NIC. It's very common to leave the VMkernel Port Group and the VM Port Group (LAN) on the same vSwitch.

You shouldn't be thinking of pfSense as a switch, it's a router/firewall. If you have multiple physical devices (wifi access point, PC, etc) to connect to the LAN you will need a physical switch, which I think is what Abdsalem referred to as a "pswitch".