One server running multiple OSs along with pfSesne



  • Hello everyone!

    I am new here, and since my query is a bit broad I decided to post it in this section of the forum, so I am sorry if I have decided wrongly.

    I am thinking of setting up a box as a home server and the plan is to run VMware ESXi on it, which in turn would virtualise a pfSense distribution, a freeNAS distribution, and a third something (not decided what exactly).
    The box is going to benefit from 4 Gigabit Ethernet ports, and my idea is to assign two of those to pfSense (WAN and LAN), and the other two to the freeNAS and the "something" respectively. The LAN port of pfSense will connect to an 8 port gigabit switch, and the other two interfaces (freeNAS and "something") also. A wireless router will also be connected to the switch to provide wireless connectivity (I am not using the router's built in switch, because more than one PC will require a connection and the router has only 4 ports). Everything will have its gateway set to the pfSense's LAN interface. So basically pfSense will act as a router and a firewall.

    Finally, my question is: will this work and is it a good idea?
    I have scribbled a rough visual representation of the topology:

    https://www.dropbox.com/s/cz451ox2k16x7kg/top1.jpg

    Thank you very much!



  • It will work, and i do it in my setup.

    I have two ESXi boxes in a fail-over DRS cluster and a seperate freenas machine that the VMs reside on, they run pfsense (two actually for a totally separate network for hosting stuff), windows server, multiple linux servers, and much more. couple of things to note though:

    You will want a static IP on ESXi, it will be the first thing to boot up on your network and if you have DHCP or DHCP reservation you will have issues.

    Do you plan to use the freenas distro to mount storage to ESXi? if so than there are other things you will want to consider, such as: if your VMs for pfsense is on your freenas distro than it will need to boot up first on the esxi machine, in order for pfsense to boot, it will also need a static ip address.

    Another thing to keep in mind is that your setup will get complicated, i myself have multiple networks, multiple vlans, multiple pfsense VMs and lots of ways to screw things up, but it is tons of fun doing everything virtual.

    Edit: as an example of what problems you may run into:

    Just the other day my systems lost power and my ESXi machines NFS mount storage from freenas with a DNS name, but my DNS server is on my NAS, which i obviously couldn't connect to and boot up. So the easiest way to overcome that was to ssh into ESXi and modify the hosts file so it didn't need to talk to a DNS server in order to get the freenas machine's IP.



  • Thank you very much for your thorough response, Atlantisman!

    The box will have 2 4TB HDDs and a single 240GB SSD.
    My idea was to install ESXi on the SSD and then install pfSense to use storage from the SSD (So, no, the pfsense distro will not reside on the freenas). The same goes for the "something" OS, and as for the freeNAS, since I would use it mostly as a media server, I'd like it to use the 2 4TB HDDs which will be setup with RAID, using a controller.
    Each OS will be on its own VLAN.

    I saw something on the ESXi documentation:
    "ESXi does not support using local, internal SATA drives on the host server to create VMFS datastores that are shared across multiple ESXi hosts"
    What exactly doest that mean?

    Also, do I need to reserve one physical port on the box, on which to assign the static IP for the ESXi for management? Because I did say I would have 4 interfaces, 2 of which will be assigned to pfSense and the other two to the other OSs. Pardon my, obviously noob question, but it's going to be my first time doing this.



  • Something that i would worry about is putting pfsense on that SSD, unless you're installing the embedded version you may end up reducing the life of the SSD greatly because pfsense will write a lot to the drive with logs and etc.

    @kambiz:

    I saw something on the ESXi documentation:
    "ESXi does not support using local, internal SATA drives on the host server to create VMFS datastores that are shared across multiple ESXi hosts"
    What exactly doest that mean?

    Also, do I need to reserve one physical port on the box, on which to assign the static IP for the ESXi for management? Because I did say I would have 4 interfaces, 2 of which will be assigned to pfSense and the other two to the other OSs. Pardon my, obviously noob question, but it's going to be my first time doing this.

    First thing, that means that you can't share a virtual machine volume to another ESXi machine. So if you add another host later than it will have to have its own storage or use other network storage. This is more of a concern if you have more than one host, such as with my system, virtual machines cannot migrate between esxi hosts if the storage is local only, this is why my storage is accessed over the network with a NFS mount.

    Second, you do not need to reserve a NIC just for ESXi, you don't even need to reserve NICs for pfsense. In the case of my setup pfsense and all the other VMs on my LAN use the same LAN NIC and i have no problems with this.