100 US dollars for working bhyve instructions on pfsense 2.2

kejianshi

The IMPRESSION I got is that it depends on what you are running, what I/O is involved etc. Seems like It might run a server with SSH access just fine but maybe not be great at something that needs alot of access to display or direct access to hardware etc. They say its alot like KVM but I was left with the impression that KVM was more feature rich.

cdburgess75

Thats an idea. Maybe run KVM instead. - or virtual box headless (phpvirtualbox for management).

WayneSherman

I got bhyve working on a clean 2.2 install. Here are the basics:

1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*

/boot/kernel/vmm.ko
/usr/lib/libvmmapi.so.5

*ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-bootonly.iso

2) Install the missing "/usr/sbin/iasl" file:
Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
Let it boot to the Welcome screen of the installer
At the installer "Welcome" dialog, select "Shell"
Run DHCP client to get an IP address: dhclient em0
Copy "iasl" file to pfsense box: scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/

3) Setup network bridge for VM in pfsense box:
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
Now in pfsense GUI:
add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
Create bridge0 with LAN and VM_BHYVE members
Assign bridge0 as VM_BRIDGE and enable
Create rules on LAN and VM_BHYVE to "allow all" access
(you can create more restrictive rules later once you get it working)

4) Load VMM kernel module:
kldload vmm

5) Make things happen at boot:
ee /boot/loader.conf
add line: vmm_load="YES"
ee /etc/sysctl.conf
add line: net.link.tap.up_on_open=1

The typical instructions for working with bhyve work from here, but here are a few more rough notes:

A) Use "vmrun.sh" helper script for launching a bhyve VM:
http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=co

B) Install pkg support in pfsense and "screen" utility:
Install pkg support in pfsense
https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

pkg (answer yes to install question)
pkg update
pkg install screen

Use "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.

C) Create a sparse disk image file for installing a VM into:
dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15G

D) Example to run a VM and install FreeBSD 10.1 into it:
screen
vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_name

Couple of issues I still have:

I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds. So I have to login to pfsense and manually run the VM.
Exiting the VM and relaunching also does not work. I have to reboot pfsense to be able to run the VM a second time.

More info on Bhyve can be found here:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
https://www.freebsd.org/cgi/man.cgi?query=bhyve
https://wiki.freebsd.org/bhyve
http://www.bsdnow.tv/tutorials/bhyve

I hope some of this is helpful.

Wayne

ahenning

Thanks, Interesting thread, was looking for this type of info to run Wanos as a VM on pfSense.

What is the conclusion: which is the best option to run VM's on pfSense as the host, bhyve or kvm?

From a stability point of view and virtual networking e.g. need to route/bridge/redirect traffic to the guest OS.

WayneSherman

@ahenning:

What is the conclusion: which is the best option to run VM's on pfSense as the host, bhyve or kvm?
From a stability point of view…

I get the impression that bhyve isn't very mature yet, so I suspect KVM might be more stable but I haven't tried it. I started with bhyve since it is more "native" to FreeBSD.

WayneSherman

@WayneSherman:

…I suspect KVM might be more stable but I haven't tried it.

I was getting ready to test KVM/QEMU on pfsense, but after I saw that QEMU wanted to pull in 49 packages and use 436MB of additional space, I decided I would be content with bhyve for now. Although bhyve has a couple of issues (which I mentioned above), it does work for my intended usage of running a Ubiquiti UniFi controller in a light FreeBSD VM.

kejianshi

So the check is in the mail then?

jwt

@kejianshi:

So the check is in the mail then?

Probably not, but we did just enable netmap and bhyve on 2.3 snapshots.


Jims-MBP:~ jim$ ssh <nope>-l root
Password for root@<nope>:
*** Welcome to pfSense 2.3-ALPHA-pfSense (amd64) on tonkawa-gw ***

 WAN (wan)       -> igb3       -> v4/DHCP4: <nope>LAN (lan)       -> igb2       -> v4: <nope>v6: <nope>OPT1 (opt1)     -> igb4       -> 
 OPT2 (opt2)     -> igb5       -> 
 HENETIPV6 (opt3) -> gif0       -> v6: <nope>0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) pfSense Developer Shell
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8 ) Shell      <– space added so you don't see a smiley.

Enter an option: 8

[2.3-ALPHA][root@<nope>]/root: kldload vmm
[2.3-ALPHA][root@<nope>]/root: ifconfig tap0 create
[2.3-ALPHA][root@<nope>]/root: sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
[2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 create
[2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 addm igb2 addm tap0
[2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 up
[2.3-ALPHA][root@<nope>]/root: truncate -s 16G guest.img
[2.3-ALPHA][root@<nope>]/root: fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.2/FreeBSD-10.2-RELEASE-amd64-bootonly.iso
FreeBSD-10.2-RELEASE-amd64-bootonly.iso       100% of  230 MB   22 Mbps 00m11s

 <copied in="" usr="" share="" examples="" bhyve="" vmrun.sh="" from="" a="" stock="" freebsd="" machine="">[2.3-ALPHA][root@<nope>]/root: chmod 755 /tmp/vmrun.sh 
[2.3-ALPHA][root@<nope>]/root: mv /tmp/vmrun.sh /usr/share/examples/bhyve/
[2.3-ALPHA][root@<nope>]/root: sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.2-RELEASE-amd64-bootonly.iso new-machine
Launching virtual machine "new-machine" …
Consoles: userboot  

FreeBSD/amd64 User boot, Revision 1.1
(root@pfs23-amd64-builder, Sat Nov 21 14:30:41 CST 2015)
Loading /boot/defaults/loader.conf 
-
  ______               ____   _____ _____  
 |  ____|             |  _ \ / ____|  __ \ 
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/

 <text eliminated="" because="" it="" screws="" up="" smf's="" rendering="">/boot/kernel/kernel text=0xfc8de8 data=0x1283b0+0x207880 syms=[0x8+0x145350+0x8+0x15fe20]
Booting…
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2399.68-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0x9f83fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt,pbe>Features2=0xc2d86217 <sse3,pclmulqdq,dtes64,ds_cpl,ssse3,cx16,xtpr,sse4.1,sse4.2,movbe,popcnt,aesni,rdrand,hv>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x200 <erms>TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory  = 1073741824 (1024 MB)
avail memory = 1010425856 (963 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <bhyve  bvmadt ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 4 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
random device not loaded; using insecure entropy
ioapic0 <version 1.1="">irqs 0-23 on motherboard
random: <software, yarrow="">initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80db8eb0, 0) error 19
kbd1 at kbdmux0
acpi0: <bhyve bvxsdt="">on motherboard
acpi0: Power Button (fixed)
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 10000000 Hz quality 950
Event timer "HPET" frequency 10000000 Hz quality 550
Event timer "HPET1" frequency 10000000 Hz quality 450
Event timer "HPET2" frequency 10000000 Hz quality 450
Event timer "HPET3" frequency 10000000 Hz quality 450
Event timer "HPET4" frequency 10000000 Hz quality 450
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
isab0: <pci-isa bridge="">at device 1.0 on pci0
isa0: <isa bus="">on isab0
virtio_pci0: <virtio pci="" network="" adapter="">port 0x2000-0x201f mem 0xc0000000-0xc0001fff irq 16 at device 2.0 on pci0
vtnet0: <virtio networking="" adapter="">on virtio_pci0
vtnet0: Ethernet address: 00:a0:98:f0:f8:5a
virtio_pci1: <virtio pci="" block="" adapter="">port 0x2040-0x207f mem 0xc0002000-0xc0003fff irq 17 at device 3.0 on pci0
vtblk0: <virtio block="" adapter="">on virtio_pci1
vtblk0: 16384MB (33554432 512 byte sectors)
ahci0: <intel ich8="" ahci="" sata="" controller="">mem 0xc0004000-0xc00043ff irq 18 at device 31.0 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <ahci channel="">at channel 0 on ahci0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
sc0: <system console="">at flags 0x100 on isa0
sc0: MDA <16 virtual consoles, flags=0x100>
vga0: <generic isa="" vga="">at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
cd0: <bhyve bhyve="" dvd-rom="" 001="">Removable CD-ROM SCSI device
cd0: Serial Number BHYVE-1C79-402D-6445
cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
cd0: cd present [118134 x 2048 byte records]
random: unblocking device.
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from cd9660:/dev/iso9660/10_2_RELEASE_AMD64_BO [ro]…
Entropy harvesting: interrupts ethernet point_to_point swi.
Starting file system checks:
Mounting local file systems:.
Writing entropy file:.
/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
Starting Network: lo0 vtnet0.
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21 <performnud,auto_linklocal>vtnet0: flags=8902 <broadcast,promisc,simplex,multicast>metric 0 mtu 1500
	options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 00:a0:98:f0:f8:5a
	nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
Starting devd.
Starting Network: vtnet0.
vtnet0: flags=8902 <broadcast,promisc,simplex,multicast>metric 0 mtu 1500
	options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 00:a0:98:f0:f8:5a
	nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Generating host.conf.
eval: cannot create /etc/host.conf: Read-only file system
eval: cannot create /etc/host.conf: Read-only file system
eval: cannot create /etc/host.conf: Read-only file system
Creating and/or trimming log files.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Clearing /tmp (X related).
Starting local daemons:
Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)
   cons25w  cons25w terminal

Console type [vt100]: 

 <gibberish ansi="" escape="" codes…="" i'm="" sure="" you="" can="" work="" it="" out="">mkdir: /tmp/bsdinstall_etc: File exists

Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)
   cons25w  cons25w terminal

 <gibberish ansi="" escape="" codes...="" i'm="" sure="" you="" can="" work="" it="" out="">Updating motd: /etc/motd is not writable, update failed.
Mounting late file systems:.
Configuring syscons: blanktime.
Starting cron.
Starting background file system checks in 60 seconds.

Sat Nov 21 23:55:01 UTC 2015

FreeBSD/amd64 (new-machine) (ttyu0)

login: root
Nov 21 23:55:36  login: ROOT LOGIN (root) ON ttyu0
FreeBSD ?.?.?  (UNKNOWN)

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
root@new-machine:~ # halt
Nov 21 23:55:39  halt: halted by root
Nov 21 23:55:39  syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 done
All buffers synced.
Uptime: 2m53s

The operating system has halted.
Please press any key to reboot.

[2.3-ALPHA][root@<nope>]/root: 
[2.3-ALPHA][root@<nope>]/root:</nope></nope></gibberish></gibberish></full-duplex></performnud,ifdisabled,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></broadcast,promisc,simplex,multicast></full-duplex></performnud,ifdisabled,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></broadcast,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></bhyve></at></keyboard></generic></system></ahci></intel></virtio></virtio></virtio></virtio></isa></pci-isa></acpi></acpi></high></at></at></bhyve></software,></version></bhyve ></erms></lahf,prefetch></syscall,nx,lm></sse3,pclmulqdq,dtes64,ds_cpl,ssse3,cx16,xtpr,sse4.1,sse4.2,movbe,popcnt,aesni,rdrand,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt,pbe></text></nope></nope></nope></copied></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope>

Thanks for the Thanksgiving Assignment!!! Looks like fun…

cdburgess75

Hey man. I forgot all about this. I had an email from an admin reminding me. Pm me PayPal if you have it. Good job. I know that they included it in dev and I can't help but think you may have been motivating a bit!

@WayneSherman:

I got bhyve working on a clean 2.2 install. Here are the basics:

1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*

/boot/kernel/vmm.ko
/usr/lib/libvmmapi.so.5

*ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-bootonly.iso

2) Install the missing "/usr/sbin/iasl" file:
Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
Let it boot to the Welcome screen of the installer
At the installer "Welcome" dialog, select "Shell"
Run DHCP client to get an IP address: dhclient em0
Copy "iasl" file to pfsense box: scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/

3) Setup network bridge for VM in pfsense box:
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
Now in pfsense GUI:
add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
Create bridge0 with LAN and VM_BHYVE members
Assign bridge0 as VM_BRIDGE and enable
Create rules on LAN and VM_BHYVE to "allow all" access
(you can create more restrictive rules later once you get it working)

4) Load VMM kernel module:
kldload vmm

5) Make things happen at boot:
ee /boot/loader.conf
add line: vmm_load="YES"
ee /etc/sysctl.conf
add line: net.link.tap.up_on_open=1

The typical instructions for working with bhyve work from here, but here are a few more rough notes:

A) Use "vmrun.sh" helper script for launching a bhyve VM:
http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=co

B) Install pkg support in pfsense and "screen" utility:
Install pkg support in pfsense
https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

pkg (answer yes to install question)
pkg update
pkg install screen

Use "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.

C) Create a sparse disk image file for installing a VM into:
dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15G

D) Example to run a VM and install FreeBSD 10.1 into it:
screen
vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_name

Couple of issues I still have:

I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds. So I have to login to pfsense and manually run the VM.

Exiting the VM and relaunching also does not work. I have to reboot pfsense to be able to run the VM a second time.

More info on Bhyve can be found here:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
https://www.freebsd.org/cgi/man.cgi?query=bhyve
https://wiki.freebsd.org/bhyve
http://www.bsdnow.tv/tutorials/bhyve

I hope some of this is helpful.

Wayne

viniciusferrao

I followed the guide and everything worked fine. But there's on curious thing. I can access the WAN and other computers on the LAN but I can't talk with the firewall, except for ICMP ping and for receiving DHCP addresses.

DNS don't get resolved, HTTP either since I'm running Squid on the firewall.

I've checked all the rules, done an allow all rule just to be sure, but I still can't talk with the firewall.

I'm not sure if it's a bhyve issue or not, perhaps whille forwarding packages on the same machine over tap0. I don't know... just wondering.

n8henrie

@viniciusferrao it's been a few years, but I followed this thread and ran into a similar issue that was ultimately due to hardware checksum offloading: https://docs.netgate.com/pfsense/en/latest/virtualization/virtio.html

I wrote up a more complete post on the steps I took to get it working: https://n8henrie.com/2023/03/running-nixos-and-ubuntu-vms-on-pfsense-via-bhyve/, hope it's hopeful to someone!