100 US dollars for working bhyve instructions on pfsense 2.2

johnpoz · Dec 16, 2014, 4:30 PM

I run my pfsense in vm, I see the point of leveraging hardware to run vms. If what you want to do is run vms - why not run a type 1 hypervisor on the hardware, and then run whatever vms you need and or the hardware can support. Pfsense is a trimmed/tweaked/custom version of freebsd to be a firewall/router - not an ideal setup for running other vms under that.

You will have much better luck, ease of use running type 1 and then your vms under that to be honest vs installing other stuff on your actual firewall/router. While yes cisco and juniper allow vms on their hardware, its not like the vms are running under the operating system of the switch/firewall OS.

Just my 2 cents on how I would go about it. I have no exp with bhyve - but the instructions seem pretty straight forward. The trick might be installing the portions that may or may not have been removed from pfsense to make it pfsense vs just plain jane freebsd running routing and firewall software. Also the way that pfsense starts up stuff vs out of the box freebsd has been modified as well - having to work within those modification to get other vms up and running might also be pita. Where if you went the other direction with say esxi, you could be up and running in a few minutes to be honest.

cdburgess75 · Dec 17, 2014, 12:26 AM

Thank you John, I have been around ESX since the early 2000's, I actually have one of the first VCP certs. I am looking to run a hypervisor on pfsense. Thats what my goal here is. Type 1 hypervisor, I assume you mean jails? The problem with that would be kernel versions. I have a specifc application that I would like to virtualize that runs a different kernel (an older freebsd one). But I do not want to limit myself there. The ideal situation for my vision is to run this app and others directly on a highpower pfsense box. It may not be easy to do, it may be, i don't know. This is the purpose for the bounty :)

kejianshi · Dec 17, 2014, 12:57 AM

I'm pretty sure ESXI counts as type 1.

I'd roll with that. Works well for me.

Take your high power box, install ESXi directly.

Install all your other OSs, including pfsense, on ESXi as VMs.

I'm not sure why you would want to do it any other way?

cdburgess75 · Dec 22, 2014, 2:12 PM

thank you kejianshi, I do not want to use esxi. I am looking to run the hypervisor on pfsense.

Jan 10, 2015, 6:35 AM

It's easy.

kejianshi · Jan 10, 2015, 7:07 AM

I guess what he means is he likes jails…

cdburgess75 · Jan 22, 2015, 3:13 AM

@gonzopancho:

It's easy.

Well if it is easy then go back to the subject and make a cool 100$.

cdburgess75 · Jan 22, 2015, 3:15 AM

@cdburgess75:

@gonzopancho:

It's easy.

Well if it is easy then go back to the subject and make a cool 100$.

Not look for jails, looking to run the hypervisor on pfsense. Not looking for any other things.

kejianshi · Jan 22, 2015, 4:21 AM

I've decided you don't know what you are talking about. (-:

cdburgess75 · Jan 23, 2015, 1:57 AM

lol-Geeze

Not wanting to run pfsense in a vm. Wanting to run pfsense as a vm host.

Weather ESXi is a type 1 or 2 has nothing to do with this.
Jails either.

I am not wanting to go the other route you are suggesting, I have a few deployed that way and they work fantastic! I want to run a hypervisor on pfsense.

Harvy66 · Jan 26, 2015, 1:11 AM

Technically bhyve is neither a Type1 or Type2, it's a hybrid. It runs everything the same way as a Type1, but because it runs a full fledged OS as the host, it's like a Type2.

kejianshi · Jan 26, 2015, 9:36 AM

bhyve (also written as BHyVe, and pronounced "bee hive") is a type-2 hypervisor that runs on FreeBSD.[1] It runs FreeBSD 9+, OpenBSD and Linux guests. Current development efforts aim at widening support for other x86-64 operating systems.

Either the title has change or I was smacked out of my mind when I replied….

One might think I would have looked into BhyVe before replying if it were right in front of my eyes.

Other than now knowing the its pronounced Bee Hive, I'm completely clueless on this.

Sorry for my previous reply - Obviously there is an option I was unaware of.

Sooooo you must have a reason to want to use BHyVe. What might that be? Just interested.

cdburgess75 · Jan 26, 2015, 8:46 PM

Thank you for that info.

kejianshi · Jan 26, 2015, 9:13 PM

I watched some videos on it and I was left with the impression that bhyve is a serious work in progress with alot of serious limitations.

mikecpt · Jan 28, 2015, 3:51 AM

So basically as a "proof of concept" or… say just for the lulz you want to run bhyve on pfsense as a HOST.

May I ask if you also have any special interest on any bhyve feature or guest (Ex: do you want to test if linux guests would work)?

From you're previous post I understand the main issue was that you could not load vmm?

Any error in particular?

I haven't tried pfsense2.2 on a physical box yet (no even production - still on 2.1) only VM's and if that's for example you're case you need to have "nested virtualization support"... I'd have to look at doing that with Xen (where I'm running a local pfsense 2.2 install for test).

Anyway assuming you're running pfsense on bare metal, and since pfsense is using FreeBSD 10 as base, in theory it should work BUT, I'm not sure if they even ported that part of the code or if they removed any parts that might make it impossible... and also as some pointed out bhyve is highly experimental and being developed on current (but it was release indeed for 10.0)

Could you possible post the error you get when trying to load vmm? cause if that doesn't work, no tutorial will help.

In any case I'll leave this link (https://www.monkeybrains.net/support/bhyve) How to for Ubuntu/Centos (not sure if its up to date, just took a quick look)

The bhyve faq also explain how to start a freebsd-11 guest.

Hope that helps, but again posting the error you get loading "vmm" would surely, help us - help you :)

mikecpt · Jan 28, 2015, 4:36 AM

Actually a kldload vmm in my box gives file not found and indeed the module is not included .

cdburgess75 · Jan 29, 2015, 12:38 PM

Ok, …full fledged OS :) - Start another thread about that on bud.

Bhyve is considered type 2

Hw-hypervisor-OS = type 1

Hw-OS-hypervisor = type 2

Not that a 1 is better than a 2 or vice versa, that's just a 50,000 ft view

..now can it run of pfsense 2.2 with guest? That's the purpose here.

kejianshi · Jan 29, 2015, 12:45 PM

The IMPRESSION I got is that it depends on what you are running, what I/O is involved etc. Seems like It might run a server with SSH access just fine but maybe not be great at something that needs alot of access to display or direct access to hardware etc. They say its alot like KVM but I was left with the impression that KVM was more feature rich.

cdburgess75 · Feb 6, 2015, 7:53 PM

Thats an idea. Maybe run KVM instead. - or virtual box headless (phpvirtualbox for management).

WayneSherman · Feb 26, 2015, 6:59 AM

I got bhyve working on a clean 2.2 install. Here are the basics:

1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*

/boot/kernel/vmm.ko
/usr/lib/libvmmapi.so.5

*ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-bootonly.iso

2) Install the missing "/usr/sbin/iasl" file:
Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
Let it boot to the Welcome screen of the installer
At the installer "Welcome" dialog, select "Shell"
Run DHCP client to get an IP address: dhclient em0
Copy "iasl" file to pfsense box: scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/

3) Setup network bridge for VM in pfsense box:
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
Now in pfsense GUI:
add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
Create bridge0 with LAN and VM_BHYVE members
Assign bridge0 as VM_BRIDGE and enable
Create rules on LAN and VM_BHYVE to "allow all" access
(you can create more restrictive rules later once you get it working)

4) Load VMM kernel module:
kldload vmm

5) Make things happen at boot:
ee /boot/loader.conf
add line: vmm_load="YES"
ee /etc/sysctl.conf
add line: net.link.tap.up_on_open=1

The typical instructions for working with bhyve work from here, but here are a few more rough notes:

A) Use "vmrun.sh" helper script for launching a bhyve VM:
http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=co

B) Install pkg support in pfsense and "screen" utility:
Install pkg support in pfsense
https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

pkg (answer yes to install question)
pkg update
pkg install screen

Use "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.

C) Create a sparse disk image file for installing a VM into:
dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15G

D) Example to run a VM and install FreeBSD 10.1 into it:
screen
vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_name

Couple of issues I still have:

I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds. So I have to login to pfsense and manually run the VM.
Exiting the VM and relaunching also does not work. I have to reboot pfsense to be able to run the VM a second time.

More info on Bhyve can be found here:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
https://www.freebsd.org/cgi/man.cgi?query=bhyve
https://wiki.freebsd.org/bhyve
http://www.bsdnow.tv/tutorials/bhyve

I hope some of this is helpful.

Wayne