Hardware vs Virtual: best choice?
-
I have been using a virtual (xen) pfsense since 2.0.1 and it seems to be fine. But as b0rman said if something goes wrong I always do wonder if visualization is the cause. The 2.1 RC seems to be the working pretty well when virtualized but personally I like the HVM route and doing PCI pass through for the network cards as it ends up being the most stable.
-
So your only choice for visualization is hyper-v?
Is this a production, lab, home setup?
I have been running pfsense on esxi for quite some time esxi4, 5, 5.1 and not had any issues. Before had it running on vmware server 2, even played with it on virtualbox, etc. until I went with the esxi host setup.
I don't think I would go back to running it physical - since I play with the development branch 2.1 its nice to be able to roll back in a click if current snap has any sort of issue. Saves space not having to have extra box, saves power, better util of resources vs multiple boxes sitting there at 2% util all the time, etc. Put them all on 1 piece of hardware and actually use the resources you paid for ;)
-
Thanks for Your answers :)
There are a lot of pros and cons in any option. I think if there is no possibility to migrate VMs - it's better to go hardware, because there is double risk to became without firewall/gateway for LAN etc. -
Double risk??
-
Like johnpoz, I've been using ESXi for a while - Jan '08 in fact. I started out migrating Smoothwall to a VM but wasn't happy with the seeming lack of development in that product. Then I discovered pfSense around June '08 and haven't looked back.
Virtualization is good if you want to run other VMs beside pfSense - mail server, Web server, whatever - on the same physical machine.
If all you want is pfSense, save yourself the learning curve, time and effort and install it on hardware. If you have the right hardware (which will need to be much more capable to run ESXi - e.g., min 4GB memory) you can learn the virtualization part later.
-
Double risk??
Yes, because host's hardware can fail OR virtualization software can fail OR host's OS can fail OR pfSense can fail => pfSense is DOWN
vs
pfSense's hardware can fail OR pfSense can fail => pfSense is DOWNI'm not telling it will happen but it can!
Of course if all Your services are also in VMs - it's not a problem, because in case of host's hardware/OS failure OR virtualization software failure (if You can't migrate VMs) all Your VMs will became down too :)If You can migrate VMs - it's more reasonable to use pfSence as VM, because You can utilize your hardware more efficient, it's more flexible solution and host's hardware failure OR virtualization software failure OR host's OS failure will not be critical for You.
-
What???
"host's hardware failure OR virtualization software failure OR host's OS failure will not be critical for You."
You not had your coffee yet this morning? Your not thinking clearly.. ;)
No it is not "double" risk because you run something in a VM..
-
He is correct as far as I see it. While virtualization does maximize you use of the hardware you paid good money for, it does create a single point of failure for everything and 2 layers of compatibility / stability to worry about. I don't think this any great revelation.
-
What?? So you have 2 internet connections, do you have multiple paths for your local network connections, do you have multiple nics in your hardware for your multipaths, do you have multiple hard disks, do you have multiple hdd controllers?
All of which if you don't are single points of failure.. You have drivers that work with your OS that that controls the hardware your using, which are all single points of failure.
You have hardware, you have software - these are parts of the system your using to connect you. Yes you could have a hardware failure, yes you could have software failure. Because your router is running in software be it the OS running on the hardware or software running on hardware that runs your "application" pfsense.
Saying you double the risk of loosing your router because it runs in vm vs on the hardware directly is nonsense.. Like saying your driver that controls your nic doubles your risk.. because not only could the hardware fail, but the OS could fail, or the driver could fail.
Your hardware could fail, or your software could fail - vm's do not significantly increase risk just because they are virtual.
-
Honestly I don't think my Xen server has crashed once in the last year and a half. Even if I mess up an update or something all I need to do is install xen again and as long as I have the configs backed up and the VMs on lvm, everything can be up and running again pretty quickly. Actually being able to run CARP with 2 virtual routers probably reduces down time more than anything.
-
So you have 2 internet connections, do you have multiple paths for your local network connections, do you have multiple nics in your hardware for your multipaths, do you have multiple hard disks, do you have multiple hdd controllers?
Yes ;D
-
So your fully redundant in hardware - so then run multiple Hosts for your VMs– this is the another great aspect of VM.. You can move a VM to new hardware if one host fails, without even dropping the connection ;)
-
Honestly I don't think my Xen server has crashed once in the last year and a half.
Quick one for you guys…. Im seriously considering getting rid of my current hardware platform for pfSense and virtualize it with XenServer (better choice for free home usage out there?) onto my new server (Supermicro with Opteron CPUs and hardware RAID1 hardware )...
I however read on thread http://forum.pfsense.org/index.php?topic=62034.0 that:
pfsense runs on FreeBSD… XenServer does not support FreeBSD at this time, therefore XenTools will not work.
I highly recommend against virtualizing pfsense in a XenServer environment as you will encounter performance degradation from the kernel running in an emulated state.
Is virtualizing pfsense with Xenserver going to cause me troubles or severe drawbacks? Whats the current state of support between pfsense and xenserver ???
PLease excuse with me… Im totally new to virtualization and I am trying to grasp the concepts. As a matter of fact, I haven't even decided which virtualization platform I will use (must be free and significantly feature rich, and have free management tools) but I am leaning toward xenserver as of now..
Thanks!
-
I am working through the same issue – whether to virtualize pfSense, or run it on dedicated hardware -- and I agree with the concern b0rman raised:
I plan to perform remote support through the internet connection. If pfSense is down, I can't connect. And if there is any kind of problem with the virtualization host - hardware, hypervisor, the pfSense VM -- or with pfSense itself, I will not be able to connect to resolve problems.
If I move pfSense to a dedicated computer some (roughly half) of those problems disappear.
The hardware becomes simpler, too, and for that reason, perhaps less likely to fail. (OTOH a lot of effort is put into the virtualization platform to ensure it is reliable.)
My current thinking is, I don't want to virtualize pfSense until I have more confidence in my virtualization setup. After it runs trouble-free for six months I will consider virtualizing pfSense.
Unless I learn something new here.
-
I am working through the same issue – whether to virtualize pfSense, or run it on dedicated hardware -- and I agree with the concern b0rman raised:
I plan to perform remote support through the internet connection. If pfSense is down, I can't connect. And if there is any kind of problem with the virtualization host - hardware, hypervisor, the pfSense VM -- or with pfSense itself, I will not be able to connect to resolve problems.
If I move pfSense to a dedicated computer some (roughly half) of those problems disappear.
The hardware becomes simpler, too, and for that reason, perhaps less likely to fail. (OTOH a lot of effort is put into the virtualization platform to ensure it is reliable.)
My current thinking is, I don't want to virtualize pfSense until I have more confidence in my virtualization setup. After it runs trouble-free for six months I will consider virtualizing pfSense.
Unless I learn something new here.
All depends, IMO what you are doing with pfsense. I do not run a business with it, so if it goes down, its a bummer but not catastrophic..
The way I see it, if I virtualize it, it should run smoothly since I have a server grade machine with server grade components. Also, if it cease to work properly, I can always get an old used machine (P4 or so) and get back in business pretty quickly.
To me, the benefit of electricity savings and less heat output is primordial over "reliability". If reliability was VERY critical, I'd virtualize 2 machines and setup a failover between them.
Other thing to consider, if you dont run a server 100% of the time, why let a big hungry computer run 24/7 if you can build/buy a small machine to run pfsense. After all, you dont need a dual socket Xeon or Opteron server with 128GB RAM to run pfsense….
I recommend you factor in all of your expectations and requirements and take a decision based on that.
-
@lpallard:
Honestly I don't think my Xen server has crashed once in the last year and a half.
Quick one for you guys…. Im seriously considering getting rid of my current hardware platform for pfSense and virtualize it with XenServer (better choice for free home usage out there?) onto my new server (Supermicro with Opteron CPUs and hardware RAID1 hardware )...
I however read on thread http://forum.pfsense.org/index.php?topic=62034.0 that:
pfsense runs on FreeBSD… XenServer does not support FreeBSD at this time, therefore XenTools will not work.
I highly recommend against virtualizing pfsense in a XenServer environment as you will encounter performance degradation from the kernel running in an emulated state.
Is virtualizing pfsense with Xenserver going to cause me troubles or severe drawbacks? Whats the current state of support between pfsense and xenserver ???
PLease excuse with me… Im totally new to virtualization and I am trying to grasp the concepts. As a matter of fact, I haven't even decided which virtualization platform I will use (must be free and significantly feature rich, and have free management tools) but I am leaning toward xenserver as of now..
Thanks!
Personally, I use ESXi (the free edition). Make sure to download 5.1 and not 5.5 since 5.5 requires VCenter for a lot of stuff and VCenter isn't free. I've been running a virtualized pfSense instance for a long time with no issues.
Here's my setup:
[Internet]<===>[pfSense VM]<===>[LAN]
||
====>[DMZ]I have two NICs in the physical machine, one connects to the WAN port and the other is the LAN port (which connects to my wireless router). pfSense is in charge of DHCP, DNS, IPS/IDS, OpenVPN, etc. The installation is the same as you would install on a physical hardware, you just need to remember to install the vm-tools package and to give your ESXi host a static IP (if you set the host for DHCP, it might not get an IP when you reboot it since the pfSense VM will come up after the ESXi networking).
I am working through the same issue – whether to virtualize pfSense, or run it on dedicated hardware -- and I agree with the concern b0rman raised:
I plan to perform remote support through the internet connection. If pfSense is down, I can't connect. And if there is any kind of problem with the virtualization host - hardware, hypervisor, the pfSense VM -- or with pfSense itself, I will not be able to connect to resolve problems.
If I move pfSense to a dedicated computer some (roughly half) of those problems disappear.
The hardware becomes simpler, too, and for that reason, perhaps less likely to fail. (OTOH a lot of effort is put into the virtualization platform to ensure it is reliable.)
My current thinking is, I don't want to virtualize pfSense until I have more confidence in my virtualization setup. After it runs trouble-free for six months I will consider virtualizing pfSense.
Unless I learn something new here.
There are a lot of pros and cons to running a virtualized pfSense system (the same way that there are a lot of pros and cons to running a hardware system). Personally, after a LOT of research and personal experience, I found that the pros of virtualization outweigh the cons. ESXi makes VM management a breeze and the ability to create snapshots means that if you mess something up, you can quickly revert everything to a previous known state.
-
Here's my setup:
[Internet]<===>[pfSense VM]<===>[LAN]
||
====>[DMZ]This is pretty much what I want to do! I now need to purchase a second hand PCIE quad port adapter on fleabay.. I suppose a Intel PRO/1000 PT is ok??? What are you using? You said you only had 2 NICs on that machine so I suppose you are not using afdditional NICs?
Regarding ESXi, have you tried the other big ones? Proxmox, Xenserver?
Some say ESXi is "gimped" to the maximum possible extent. Again if true, I dont like that. I want a full featured virtualization platform. Thats why after ESXi I was going toward Proxmox or Xenserver.
I know ESXi is very popular and must (or maybe not?) get the most driver development, etc… If its locked somehow or limited in any way, I may opt for another platform..
Please share your thoughts!
Thanks a lot my friend!
-
This is pretty much what I want to do! I now need to purchase a second hand PCIE quad port adapter on fleabay.. I suppose a Intel PRO/1000 PT is ok??? What are you using? You said you only had 2 NICs on that machine so I suppose you are not using afdditional NICs?
I would highly recommend an Intel card. They are considered the most stable ones for virtualization. People that use other cards are usually the ones that you see in the forums asking for help due to network issues.
The machine that I'm using is a re-purposed desktop that I had. I use the built-in NIC for the WAN port and I installed an additional NIC for the LAN port (which connects to my switch/wireless router). The computer came with only one NIC. You probably won't need a quad-card. Since everything is virtualized, you can just add virtual switches and bind them to the virtual NICs on your pfSense VM. That's what I did with my DMZ. I've added a vSwitch that's not connected to any NIC and added another virtual NIC to the pfSense machine. I let pfSense do all the routing.
Here's a good guide that will get you started: https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5
Regarding ESXi, have you tried the other big ones? Proxmox, Xenserver?
Some say ESXi is "gimped" to the maximum possible extent. Again if true, I dont like that. I want a full featured virtualization platform. Thats why after ESXi I was going toward Proxmox or Xenserver.
I know ESXi is very popular and must (or maybe not?) get the most driver development, etc… If its locked somehow or limited in any way, I may opt for another platform..
I used ESXi because that's the thing I was familiar with. There are a lot of other alternatives, but I found more people familiar with VMWare products so it's much easier to find help. I would recommend getting ESXi 5.1 instead of 5.5 since 5.5 has a lot of features that require VCenter (which isn't a free product). From my experience, ESXi gives you everything you need, but it will also give you a lot of stuff that you don't so don't get carried away before you have a basic system up and running. Get the basics running and go from there.
When in doubt, ask for help! People on this forum are very helpful and if you can't find the answer here, from my experience, after some Googling, you'll find a web/blog post with the answer.
Please share your thoughts!
Thanks a lot my friend!
-
With the free version of ESXi if you lose power, when power is restored and your Hyper Visor is rebooted your VMs won't start automatically. This could potentially be a problem (What if you are not home) unless someone has figured something out here. I am running Cisco Call Manager in a VM and this is a problem that I have. My Fix was to put my ESXi server on an UPs to tolerate temporary power outages.
-
With the free version of ESXi if you lose power, when power is restored and your Hyper Visor is rebooted your VMs won't start automatically. This could potentially be a problem (What if you are not home) unless someone has figured something out here.
I'm running the free version of ESXi 5.1 and the VM's start automatically. See the attached image for auto start up….
