100 US dollars for working bhyve instructions on pfsense 2.2



  • I would really like to see this done.  Just looking for a simple set of instructions to use bhyve to run an older freebsd version (i386-6.4) in pfsense 2.2

    Any takers?



  • follow the README.  it's pretty simple.


  • LAYER 8 Global Moderator

    Out of curiosity - why, what are you wanting run in vm on your router/firewall?



  • Gonzo,  I get stuck because I cannot kldload vmm.  But like I said,  Im willing to pay for a step by step.  I had ok luck with virtual box but got stuck.  I would prefer bhyve.  If you can do it,  show me how and make a quick 100$

    John,  I want to run a guest operating system or 2–Its a project (proof of concept more or less)  FYI:  Cisco and Juniper do it all the time

    /dave


  • LAYER 8 Global Moderator

    I run my pfsense in vm, I see the point of leveraging hardware to run vms.  If what you want to do is run vms - why not run a type 1 hypervisor on the hardware, and then run whatever vms you need and or the hardware can support.  Pfsense is a trimmed/tweaked/custom version of freebsd to be a firewall/router - not an ideal setup for running other vms under that.

    You will have much better luck, ease of use running type 1 and then your vms under that to be honest vs installing other stuff on your actual firewall/router.  While yes cisco and juniper allow vms on their hardware, its not like the vms are running under the operating system of the switch/firewall OS.

    Just my 2 cents on how I would go about it.  I have no exp with bhyve - but the instructions seem pretty straight forward.  The trick might be installing the portions that may or may not have been removed from pfsense to make it pfsense vs just plain jane freebsd running routing and firewall software.  Also the way that pfsense starts up stuff vs out of the box freebsd has been modified as well - having to work within those modification to get other vms up and running might also be pita.  Where if you went the other direction with say esxi, you could be up and running in a few minutes to be honest.



  • Thank you John,  I have been around ESX since the early 2000's,  I actually have one of the first VCP certs.  I am looking to run a hypervisor on pfsense.  Thats what my goal here is.  Type 1 hypervisor, I assume you mean jails?  The problem with that would be kernel versions.  I have a specifc application that I would like to virtualize that runs a different kernel (an older freebsd one).  But I do not want to limit myself there.  The ideal situation for my vision is to run this app and others directly on a highpower pfsense box.  It may not be easy to do,  it may be,  i don't know.  This is the purpose for the bounty :)



  • I'm pretty sure ESXI counts as type 1.

    I'd roll with that.  Works well for me.

    Take your high power box, install ESXi directly.

    Install all your other OSs, including pfsense, on ESXi as VMs.

    I'm not sure why you would want to do it any other way?



  • thank you kejianshi,  I do not want to use esxi.  I am looking to run the hypervisor on pfsense.



  • It's easy.



  • I guess what he means is he likes jails…



  • @gonzopancho:

    It's easy.

    Well if it is easy then go back to the subject and make a cool 100$.



  • @cdburgess75:

    @gonzopancho:

    It's easy.

    Well if it is easy then go back to the subject and make a cool 100$.

    Not look for jails, looking to run the hypervisor on pfsense. Not looking for any other things.



  • I've decided you don't know what you are talking about.  (-:



  • lol-Geeze

    Not wanting to run pfsense in a vm.  Wanting to run pfsense as a vm host.

    Weather ESXi is a type 1 or 2 has nothing to do with this.
    Jails either.

    I am not wanting to go the other route you are suggesting,  I have a few deployed that way and they work fantastic!  I want to run a hypervisor on pfsense.



  • Technically bhyve is neither a Type1 or Type2, it's a hybrid. It runs everything the same way as a Type1, but because it runs a full fledged OS as the host, it's like a Type2.



  • bhyve (also written as BHyVe, and pronounced "bee hive") is a type-2 hypervisor that runs on FreeBSD.[1] It runs FreeBSD 9+, OpenBSD and Linux guests. Current development efforts aim at widening support for other x86-64 operating systems.

    Either the title has change or I was smacked out of my mind when I replied….

    One might think I would have looked into BhyVe before replying if it were right in front of my eyes.

    Other than now knowing the its pronounced Bee Hive, I'm completely clueless on this.

    Sorry for my previous reply - Obviously there is an option I was unaware of.

    Sooooo you must have a reason to want to use BHyVe.  What might that be?  Just interested.



  • Thank you for that info.



  • I watched some videos on it and I was left with the impression that bhyve is a serious work in progress with alot of serious limitations.



  • So basically as a "proof of concept" or… say just for the lulz you want to run bhyve on pfsense as a HOST.

    May I ask if you also have any special interest on any bhyve feature or guest (Ex: do you want to test if linux guests would work)?

    From you're previous post I understand the main issue was that you could not load vmm?

    Any error in particular?

    I haven't tried pfsense2.2 on a physical box yet (no even production - still on 2.1) only VM's and if that's for example you're case you need to have "nested virtualization support"... I'd have to look at doing that with Xen (where I'm running a local pfsense 2.2 install for test).

    Anyway assuming you're running pfsense on bare metal, and since pfsense is using FreeBSD 10 as base, in theory it should work BUT, I'm not sure if they even ported that part of the code or if they removed any parts that might make it impossible... and also as some pointed out bhyve is highly experimental and being developed on current (but it was release indeed for 10.0)

    Could you possible post the error you get when trying to load vmm? cause if that doesn't work, no tutorial will help.

    In any case I'll leave this link (https://www.monkeybrains.net/support/bhyve) How to for Ubuntu/Centos (not sure if its up to date, just took a quick look)

    The bhyve faq also explain how to start a freebsd-11 guest.

    Hope that helps, but again posting the error you get loading "vmm" would surely, help us - help you :)



  • Actually a kldload vmm in my box gives file not found and indeed the module is not included .



  • Ok, …full fledged OS :) - Start another thread about that on bud.

    Bhyve is considered type 2

    Hw-hypervisor-OS = type 1

    Hw-OS-hypervisor = type 2

    Not that a 1 is better than a 2 or vice versa, that's just a 50,000 ft view

    ..now can it run of pfsense 2.2 with guest?  That's the purpose here.



  • The IMPRESSION I got is that it depends on what you are running, what I/O is involved etc.  Seems like It might run a server with SSH access just fine but maybe not be great at something that needs alot of access to display or direct access to hardware etc.  They say its alot like KVM but I was left with the impression that KVM was more feature rich.



  • Thats an idea.  Maybe run KVM instead.  -  or virtual box headless (phpvirtualbox for management).



  • I got bhyve working on a clean 2.2 install.  Here are the basics:

    1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*

    /boot/kernel/vmm.ko
    /usr/lib/libvmmapi.so.5

    *ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-bootonly.iso

    2) Install the missing "/usr/sbin/iasl" file:
    Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
    Let it boot to the Welcome screen of the installer
    At the installer "Welcome" dialog, select "Shell"
    Run DHCP client to get an IP address:  dhclient em0
    Copy "iasl" file to pfsense box:  scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/

    3) Setup network bridge for VM in pfsense box:
    ifconfig tap0 create
    sysctl net.link.tap.up_on_open=1
    Now in pfsense GUI:
      add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
      Create bridge0 with LAN and VM_BHYVE members
      Assign bridge0  as VM_BRIDGE and enable
      Create rules on LAN and VM_BHYVE to "allow all" access
      (you can create more restrictive rules later once you get it working)

    4) Load VMM kernel module:
    kldload vmm

    5) Make things happen at boot:
    ee /boot/loader.conf
    add line:  vmm_load="YES"
    ee /etc/sysctl.conf
    add line:  net.link.tap.up_on_open=1

    The typical instructions for working with bhyve work from here, but here are a few more rough notes:

    A) Use "vmrun.sh" helper script for launching a bhyve VM:
    http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=co

    B) Install pkg support in pfsense and "screen" utility:
    Install pkg support in pfsense
    https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

    pkg  (answer yes to install question)
    pkg update
    pkg install screen

    Use "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.

    C) Create a sparse disk image file for installing a VM into:
    dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15G

    D) Example to run a VM and install FreeBSD 10.1 into it:
    screen
    vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_name

    Couple of issues I still have:

    • I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds.  So I have to login to pfsense and manually run the VM.
    • Exiting the VM and relaunching also does not work.  I have to reboot pfsense to be able to run the VM a second time.

    More info on Bhyve can be found here:
    https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
    https://www.freebsd.org/cgi/man.cgi?query=bhyve
    https://wiki.freebsd.org/bhyve
    http://www.bsdnow.tv/tutorials/bhyve

    I hope some of this is helpful.

    Wayne



  • Thanks, Interesting thread, was looking for this type of info to run Wanos as a VM on pfSense.

    What is the conclusion: which is the best option to run VM's on pfSense as the host, bhyve or kvm?

    From a stability point of view and virtual networking e.g. need to route/bridge/redirect traffic to the guest OS.



  • @ahenning:

    What is the conclusion: which is the best option to run VM's on pfSense as the host, bhyve or kvm?
    From a stability point of view…

    I get the impression that bhyve isn't very mature yet, so I suspect KVM might be more stable but I haven't tried it.  I started with bhyve since it is more "native" to FreeBSD.



  • @WayneSherman:

    …I suspect KVM might be more stable but I haven't tried it.

    I was getting ready to test KVM/QEMU on pfsense, but after I saw that QEMU wanted to pull in 49 packages and use 436MB of additional space, I decided I would be content with bhyve for now.  Although bhyve has a couple of issues (which I mentioned above), it does work for my intended usage of running a Ubiquiti UniFi controller in a light FreeBSD VM.



  • So the check is in the mail then?


  • Netgate

    @kejianshi:

    So the check is in the mail then?

    Probably not, but we did just enable netmap and bhyve on 2.3 snapshots.

    
    Jims-MBP:~ jim$ ssh <nope>-l root
    Password for root@<nope>:
    *** Welcome to pfSense 2.3-ALPHA-pfSense (amd64) on tonkawa-gw ***
    
     WAN (wan)       -> igb3       -> v4/DHCP4: <nope>LAN (lan)       -> igb2       -> v4: <nope>v6: <nope>OPT1 (opt1)     -> igb4       -> 
     OPT2 (opt2)     -> igb5       -> 
     HENETIPV6 (opt3) -> gif0       -> v6: <nope>0) Logout (SSH only)                  9) pfTop
     1) Assign Interfaces                 10) Filter Logs
     2) Set interface(s) IP address       11) Restart webConfigurator
     3) Reset webConfigurator password    12) pfSense Developer Shell
     4) Reset to factory defaults         13) Update from console
     5) Reboot system                     14) Disable Secure Shell (sshd)
     6) Halt system                       15) Restore recent configuration
     7) Ping host                         16) Restart PHP-FPM
     8 ) Shell      <– space added so you don't see a smiley.
    
    Enter an option: 8
    
    [2.3-ALPHA][root@<nope>]/root: kldload vmm
    [2.3-ALPHA][root@<nope>]/root: ifconfig tap0 create
    [2.3-ALPHA][root@<nope>]/root: sysctl net.link.tap.up_on_open=1
    net.link.tap.up_on_open: 0 -> 1
    [2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 create
    [2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 addm igb2 addm tap0
    [2.3-ALPHA][root@<nope>]/root: ifconfig bridge0 up
    [2.3-ALPHA][root@<nope>]/root: truncate -s 16G guest.img
    [2.3-ALPHA][root@<nope>]/root: fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.2/FreeBSD-10.2-RELEASE-amd64-bootonly.iso
    FreeBSD-10.2-RELEASE-amd64-bootonly.iso       100% of  230 MB   22 Mbps 00m11s
    
     <copied in="" usr="" share="" examples="" bhyve="" vmrun.sh="" from="" a="" stock="" freebsd="" machine="">[2.3-ALPHA][root@<nope>]/root: chmod 755 /tmp/vmrun.sh 
    [2.3-ALPHA][root@<nope>]/root: mv /tmp/vmrun.sh /usr/share/examples/bhyve/
    [2.3-ALPHA][root@<nope>]/root: sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.2-RELEASE-amd64-bootonly.iso new-machine
    Launching virtual machine "new-machine" …
    Consoles: userboot  
    
    FreeBSD/amd64 User boot, Revision 1.1
    (root@pfs23-amd64-builder, Sat Nov 21 14:30:41 CST 2015)
    Loading /boot/defaults/loader.conf 
    -
      ______               ____   _____ _____  
     |  ____|             |  _ \ / ____|  __ \ 
     | |___ _ __ ___  ___ | |_) | (___ | |  | |
     |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
     | |   | | |  __/  __/| |_) |____) | |__| |
     | |   | | |    |    ||     |      |      |
     |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                                 s` `.....---.......--.```   -/
    
     <text eliminated="" because="" it="" screws="" up="" smf's="" rendering="">/boot/kernel/kernel text=0xfc8de8 data=0x1283b0+0x207880 syms=[0x8+0x145350+0x8+0x15fe20]
    Booting…
    Copyright (c) 1992-2015 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015
        root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2399.68-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
      Features=0x9f83fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt,pbe>Features2=0xc2d86217 <sse3,pclmulqdq,dtes64,ds_cpl,ssse3,cx16,xtpr,sse4.1,sse4.2,movbe,popcnt,aesni,rdrand,hv>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x200 <erms>TSC: P-state invariant
    Hypervisor: Origin = "bhyve bhyve "
    real memory  = 1073741824 (1024 MB)
    avail memory = 1010425856 (963 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <bhyve  bvmadt ="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 4 package(s) x 1 core(s)
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP): APIC ID:  1
     cpu2 (AP): APIC ID:  2
     cpu3 (AP): APIC ID:  3
    random device not loaded; using insecure entropy
    ioapic0 <version 1.1="">irqs 0-23 on motherboard
    random: <software, yarrow="">initialized
    module_register_init: MOD_LOAD (vesa, 0xffffffff80db8eb0, 0) error 19
    kbd1 at kbdmux0
    acpi0: <bhyve bvxsdt="">on motherboard
    acpi0: Power Button (fixed)
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 10000000 Hz quality 950
    Event timer "HPET" frequency 10000000 Hz quality 550
    Event timer "HPET1" frequency 10000000 Hz quality 450
    Event timer "HPET2" frequency 10000000 Hz quality 450
    Event timer "HPET3" frequency 10000000 Hz quality 450
    Event timer "HPET4" frequency 10000000 Hz quality 450
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    isab0: <pci-isa bridge="">at device 1.0 on pci0
    isa0: <isa bus="">on isab0
    virtio_pci0: <virtio pci="" network="" adapter="">port 0x2000-0x201f mem 0xc0000000-0xc0001fff irq 16 at device 2.0 on pci0
    vtnet0: <virtio networking="" adapter="">on virtio_pci0
    vtnet0: Ethernet address: 00:a0:98:f0:f8:5a
    virtio_pci1: <virtio pci="" block="" adapter="">port 0x2040-0x207f mem 0xc0002000-0xc0003fff irq 17 at device 3.0 on pci0
    vtblk0: <virtio block="" adapter="">on virtio_pci1
    vtblk0: 16384MB (33554432 512 byte sectors)
    ahci0: <intel ich8="" ahci="" sata="" controller="">mem 0xc0004000-0xc00043ff irq 18 at device 31.0 on pci0
    ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
    ahcich0: <ahci channel="">at channel 0 on ahci0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart0: console (9600,n,8,1)
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: MDA <16 virtual consoles, flags=0x100>
    vga0: <generic isa="" vga="">at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
    atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    ppc0: cannot reserve I/O port range
    Timecounters tick every 1.000 msec
    cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
    cd0: <bhyve bhyve="" dvd-rom="" 001="">Removable CD-ROM SCSI device
    cd0: Serial Number BHYVE-1C79-402D-6445
    cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
    cd0: cd present [118134 x 2048 byte records]
    random: unblocking device.
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #3 Launched!
    Trying to mount root from cd9660:/dev/iso9660/10_2_RELEASE_AMD64_BO [ro]…
    Entropy harvesting: interrupts ethernet point_to_point swi.
    Starting file system checks:
    Mounting local file systems:.
    Writing entropy file:.
    /etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
    Starting Network: lo0 vtnet0.
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
    	options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 
    	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
    	inet 127.0.0.1 netmask 0xff000000 
    	nd6 options=21 <performnud,auto_linklocal>vtnet0: flags=8902 <broadcast,promisc,simplex,multicast>metric 0 mtu 1500
    	options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 00:a0:98:f0:f8:5a
    	nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
    Starting devd.
    Starting Network: vtnet0.
    vtnet0: flags=8902 <broadcast,promisc,simplex,multicast>metric 0 mtu 1500
    	options=80028 <vlan_mtu,jumbo_mtu,linkstate>ether 00:a0:98:f0:f8:5a
    	nd6 options=29 <performnud,ifdisabled,auto_linklocal>media: Ethernet 10Gbase-T <full-duplex>status: active
    add net fe80::: gateway ::1
    add net ff02::: gateway ::1
    add net ::ffff:0.0.0.0: gateway ::1
    add net ::0.0.0.0: gateway ::1
    Generating host.conf.
    eval: cannot create /etc/host.conf: Read-only file system
    eval: cannot create /etc/host.conf: Read-only file system
    eval: cannot create /etc/host.conf: Read-only file system
    Creating and/or trimming log files.
    Starting syslogd.
    ELF ldconfig path: /lib /usr/lib /usr/lib/compat
    32-bit compatibility ldconfig path: /usr/lib32
    Clearing /tmp (X related).
    Starting local daemons:
    Welcome to FreeBSD!
    
    Please choose the appropriate terminal type for your system.
    Common console types are:
       ansi     Standard ANSI terminal
       vt100    VT100 or compatible terminal
       xterm    xterm terminal emulator (or compatible)
       cons25w  cons25w terminal
    
    Console type [vt100]: 
    
     <gibberish ansi="" escape="" codes…="" i'm="" sure="" you="" can="" work="" it="" out="">mkdir: /tmp/bsdinstall_etc: File exists
    
    Welcome to FreeBSD!
    
    Please choose the appropriate terminal type for your system.
    Common console types are:
       ansi     Standard ANSI terminal
       vt100    VT100 or compatible terminal
       xterm    xterm terminal emulator (or compatible)
       cons25w  cons25w terminal
    
     <gibberish ansi="" escape="" codes...="" i'm="" sure="" you="" can="" work="" it="" out="">Updating motd: /etc/motd is not writable, update failed.
    Mounting late file systems:.
    Configuring syscons: blanktime.
    Starting cron.
    Starting background file system checks in 60 seconds.
    
    Sat Nov 21 23:55:01 UTC 2015
    
    FreeBSD/amd64 (new-machine) (ttyu0)
    
    login: root
    Nov 21 23:55:36  login: ROOT LOGIN (root) ON ttyu0
    FreeBSD ?.?.?  (UNKNOWN)
    
    Welcome to FreeBSD!
    
    Release Notes, Errata: https://www.FreeBSD.org/releases/
    Security Advisories:   https://www.FreeBSD.org/security/
    FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
    FreeBSD FAQ:           https://www.FreeBSD.org/faq/
    Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
    FreeBSD Forums:        https://forums.FreeBSD.org/
    
    Documents installed with the system are in the /usr/local/share/doc/freebsd/
    directory, or can be installed later with:  pkg install en-freebsd-doc
    For other languages, replace "en" with a language code like de or fr.
    
    Show the version of FreeBSD installed:  freebsd-version ; uname -a
    Please include that output and any error messages when posting questions.
    Introduction to manual pages:  man man
    FreeBSD directory layout:      man hier
    
    Edit /etc/motd to change this login announcement.
    root@new-machine:~ # halt
    Nov 21 23:55:39  halt: halted by root
    Nov 21 23:55:39  syslogd: exiting on signal 15
    Waiting (max 60 seconds) for system process `vnlru' to stop...done
    Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
    Waiting (max 60 seconds) for system process `syncer' to stop...
    Syncing disks, vnodes remaining...0 0 done
    All buffers synced.
    Uptime: 2m53s
    
    The operating system has halted.
    Please press any key to reboot.
    
    [2.3-ALPHA][root@<nope>]/root: 
    [2.3-ALPHA][root@<nope>]/root:</nope></nope></gibberish></gibberish></full-duplex></performnud,ifdisabled,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></broadcast,promisc,simplex,multicast></full-duplex></performnud,ifdisabled,auto_linklocal></vlan_mtu,jumbo_mtu,linkstate></broadcast,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></bhyve></at></keyboard></generic></system></ahci></intel></virtio></virtio></virtio></virtio></isa></pci-isa></acpi></acpi></high></at></at></bhyve></software,></version></bhyve ></erms></lahf,prefetch></syscall,nx,lm></sse3,pclmulqdq,dtes64,ds_cpl,ssse3,cx16,xtpr,sse4.1,sse4.2,movbe,popcnt,aesni,rdrand,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt,pbe></text></nope></nope></nope></copied></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope></nope> 
    


  • Thanks for the Thanksgiving Assignment!!! Looks like fun…



  • Hey man. I forgot all about this. I had an email from an admin reminding me. Pm me PayPal if you have it. Good job. I know that they included it in dev and I can't help but think you may have been motivating a bit!

    @WayneSherman:

    I got bhyve working on a clean 2.2 install.  Here are the basics:

    1) Install these two files on pfsense by extracting them from the FreeBSD 10.1 boot-only ISO:*

    /boot/kernel/vmm.ko
    /usr/lib/libvmmapi.so.5

    *ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-bootonly.iso

    2) Install the missing "/usr/sbin/iasl" file:
    Boot FreeBSD-10.1-RELEASE-amd64-bootonly.iso ISO image (on a real computer or in a VM such as Qemu)
    Let it boot to the Welcome screen of the installer
    At the installer "Welcome" dialog, select "Shell"
    Run DHCP client to get an IP address:  dhclient em0
    Copy "iasl" file to pfsense box:  scp /usr/sbin/iasl root@pfsensebox:/usr/sbin/

    3) Setup network bridge for VM in pfsense box:
    ifconfig tap0 create
    sysctl net.link.tap.up_on_open=1
    Now in pfsense GUI:
      add TAP0 as VM_BHYVE interface and enable it (don't give it an IP address)
      Create bridge0 with LAN and VM_BHYVE members
      Assign bridge0  as VM_BRIDGE and enable
      Create rules on LAN and VM_BHYVE to "allow all" access
      (you can create more restrictive rules later once you get it working)

    4) Load VMM kernel module:
    kldload vmm

    5) Make things happen at boot:
    ee /boot/loader.conf
    add line:  vmm_load="YES"
    ee /etc/sysctl.conf
    add line:  net.link.tap.up_on_open=1

    The typical instructions for working with bhyve work from here, but here are a few more rough notes:

    A) Use "vmrun.sh" helper script for launching a bhyve VM:
    http://svnweb.freebsd.org/base/head/share/examples/bhyve/vmrun.sh?view=co

    B) Install pkg support in pfsense and "screen" utility:
    Install pkg support in pfsense
    https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages

    pkg  (answer yes to install question)
    pkg update
    pkg install screen

    Use "screen" to run the VM and interact with it's terminal, and be able to switch to another terminal.

    C) Create a sparse disk image file for installing a VM into:
    dd if=/dev/zero of=/vm_image/fbsd_unifi.img bs=1 count=0 seek=15G

    D) Example to run a VM and install FreeBSD 10.1 into it:
    screen
    vmrun.sh -i -c 2 -m 1024M -t tap0 -d /vm_image/fbsd_unifi.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso a_vm_name

    Couple of issues I still have:

    • I wasn't successful a starting a VM at bootup, it crashes after about 30 seconds.  So I have to login to pfsense and manually run the VM.
    • Exiting the VM and relaunching also does not work.  I have to reboot pfsense to be able to run the VM a second time.

    More info on Bhyve can be found here:
    https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html
    https://www.freebsd.org/cgi/man.cgi?query=bhyve
    https://wiki.freebsd.org/bhyve
    http://www.bsdnow.tv/tutorials/bhyve

    I hope some of this is helpful.

    Wayne



  • I followed the guide and everything worked fine. But there's on curious thing. I can access the WAN and other computers on the LAN but I can't talk with the firewall, except for ICMP ping and for receiving DHCP addresses.

    DNS don't get resolved, HTTP either since I'm running Squid on the firewall.

    I've checked all the rules, done an allow all rule just to be sure, but I still can't talk with the firewall.

    I'm not sure if it's a bhyve issue or not, perhaps whille forwarding packages on the same machine over tap0. I don't know... just wondering.


Log in to reply