100 US dollars for working bhyve instructions on pfsense 2.2
-
Thank you John, I have been around ESX since the early 2000's, I actually have one of the first VCP certs. I am looking to run a hypervisor on pfsense. Thats what my goal here is. Type 1 hypervisor, I assume you mean jails? The problem with that would be kernel versions. I have a specifc application that I would like to virtualize that runs a different kernel (an older freebsd one). But I do not want to limit myself there. The ideal situation for my vision is to run this app and others directly on a highpower pfsense box. It may not be easy to do, it may be, i don't know. This is the purpose for the bounty :)
-
I'm pretty sure ESXI counts as type 1.
I'd roll with that. Works well for me.
Take your high power box, install ESXi directly.
Install all your other OSs, including pfsense, on ESXi as VMs.
I'm not sure why you would want to do it any other way?
-
thank you kejianshi, I do not want to use esxi. I am looking to run the hypervisor on pfsense.
-
It's easy.
-
I guess what he means is he likes jails…
-
@gonzopancho:
It's easy.
Well if it is easy then go back to the subject and make a cool 100$.
-
@gonzopancho:
It's easy.
Well if it is easy then go back to the subject and make a cool 100$.
Not look for jails, looking to run the hypervisor on pfsense. Not looking for any other things.
-
I've decided you don't know what you are talking about. (-:
-
lol-Geeze
Not wanting to run pfsense in a vm. Wanting to run pfsense as a vm host.
Weather ESXi is a type 1 or 2 has nothing to do with this.
Jails either.I am not wanting to go the other route you are suggesting, I have a few deployed that way and they work fantastic! I want to run a hypervisor on pfsense.
-
Technically bhyve is neither a Type1 or Type2, it's a hybrid. It runs everything the same way as a Type1, but because it runs a full fledged OS as the host, it's like a Type2.
-
bhyve (also written as BHyVe, and pronounced "bee hive") is a type-2 hypervisor that runs on FreeBSD.[1] It runs FreeBSD 9+, OpenBSD and Linux guests. Current development efforts aim at widening support for other x86-64 operating systems.
Either the title has change or I was smacked out of my mind when I replied….
One might think I would have looked into BhyVe before replying if it were right in front of my eyes.
Other than now knowing the its pronounced Bee Hive, I'm completely clueless on this.
Sorry for my previous reply - Obviously there is an option I was unaware of.
Sooooo you must have a reason to want to use BHyVe. What might that be? Just interested.
-
Thank you for that info.
-
I watched some videos on it and I was left with the impression that bhyve is a serious work in progress with alot of serious limitations.
-
So basically as a "proof of concept" or… say just for the lulz you want to run bhyve on pfsense as a HOST.
May I ask if you also have any special interest on any bhyve feature or guest (Ex: do you want to test if linux guests would work)?
From you're previous post I understand the main issue was that you could not load vmm?
Any error in particular?
I haven't tried pfsense2.2 on a physical box yet (no even production - still on 2.1) only VM's and if that's for example you're case you need to have "nested virtualization support"... I'd have to look at doing that with Xen (where I'm running a local pfsense 2.2 install for test).
Anyway assuming you're running pfsense on bare metal, and since pfsense is using FreeBSD 10 as base, in theory it should work BUT, I'm not sure if they even ported that part of the code or if they removed any parts that might make it impossible... and also as some pointed out bhyve is highly experimental and being developed on current (but it was release indeed for 10.0)
Could you possible post the error you get when trying to load vmm? cause if that doesn't work, no tutorial will help.
In any case I'll leave this link (https://www.monkeybrains.net/support/bhyve) How to for Ubuntu/Centos (not sure if its up to date, just took a quick look)
The bhyve faq also explain how to start a freebsd-11 guest.
Hope that helps, but again posting the error you get loading "vmm" would surely, help us - help you :)
-
Actually a kldload vmm in my box gives file not found and indeed the module is not included .
-
Ok, …full fledged OS :) - Start another thread about that on bud.
Bhyve is considered type 2
Hw-hypervisor-OS = type 1
Hw-OS-hypervisor = type 2
Not that a 1 is better than a 2 or vice versa, that's just a 50,000 ft view
..now can it run of pfsense 2.2 with guest? That's the purpose here.
-
The IMPRESSION I got is that it depends on what you are running, what I/O is involved etc. Seems like It might run a server with SSH access just fine but maybe not be great at something that needs alot of access to display or direct access to hardware etc. They say its alot like KVM but I was left with the impression that KVM was more feature rich.
-
Thats an idea. Maybe run KVM instead. - or virtual box headless (phpvirtualbox for management).
-
I got bhyve working on a clean 2.2 install. Here are the basics:
1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*
/boot/kernel/vmm.ko
/usr/lib/libvmmapi.so.52) Install the missing "/usr/sbin/iasl" file:
Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
Let it boot to the Welcome screen of the installer
At the installer "Welcome" dialog, select "Shell"
Run DHCP client to get an IP address: dhclient em0
Copy "iasl" file to pfsense box: scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/3) Setup network bridge for VM in pfsense box:
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
Now in pfsense GUI:
add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
Create bridge0 with LAN and VM_BHYVE members
Assign bridge0 as VM_BRIDGE and enable
Create rules on LAN and VM_BHYVE to "allow all" access
(you can create more restrictive rules later once you get it working)4) Load VMM kernel module:
kldload vmm5) Make things happen at boot:
ee /boot/loader.conf
add line: vmm_load="YES"
ee /etc/sysctl.conf
add line: net.link.tap.up_on_open=1The typical instructions for working with bhyve work from here, but here are a few more rough notes:
A) Use "vmrun.sh" helper script for launching a bhyve VM:
http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=coB) Install pkg support in pfsense and "screen" utility:
Install pkg support in pfsense
https://doc.pfsense.org/index.php/Installing_FreeBSD_Packagespkg (answer yes to install question)
pkg update
pkg install screenUse "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.
C) Create a sparse disk image file for installing a VM into:
dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15GD) Example to run a VM and install FreeBSD 10.1 into it:
screen
vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_nameCouple of issues I still have:
- I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds. So I have to login to pfsense and manually run the VM.
- Exiting the VM and relaunching also does not work. I have to reboot pfsense to be able to run the VM a second time.
More info on Bhyve can be found here:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
https://www.freebsd.org/cgi/man.cgi?query=bhyve
https://wiki.freebsd.org/bhyve
http://www.bsdnow.tv/tutorials/bhyveI hope some of this is helpful.
Wayne
-
Thanks, Interesting thread, was looking for this type of info to run Wanos as a VM on pfSense.
What is the conclusion: which is the best option to run VM's on pfSense as the host, bhyve or kvm?
From a stability point of view and virtual networking e.g. need to route/bridge/redirect traffic to the guest OS.