Any way to build pfSense2.1 i386 for XEN4 PV Paravirt mode?

  • Hi,

    i am trying to make an pfSense2.1-i386(nano) Image for XEN4 in PV Paravirtual mode.

    making an normal nano image is no problem.

    but i a run into many compilings errors if i try to build the nano image with a modified  Kernel config.
    trying to mix the original FreeBSD8.3 XEN Kernel config with the pfSense_wrap.8.i386 breaks the compilation.

    any way to get pfSense2.1 with XEN PV mode to run ?

    later if i am on compiling machine again then i would post the error messages while compiling.

    best regards
    second attemp to post

  • hello ren22, I was able to compile a pfSense i386 Xen PV image but never was successful in making it boot properly. I will have some time to mess around with the configuration again next week and see if I can help.

  • Sabre - you might find the config info helpful towards the end of the post in that other thread.  Could you also post your updated "/home/pfsense/tools/builder_scripts/conf/kernel/pfSense_wrap.8.i386" with the XEN options?

    I made a diff of the BSD i386 vs. XEN conf files, and used them as the basis for updating the current pfSense_wrap.8.i386, but I'm getting a compile error building the kernel.

    Good to know I'm not the only one trying to build a Paravirt-capable kernel for pfSense.  I'm new to BSD, but have a lot of Linux experience, so I'm working through this.  I'm building an all-in-one box and would like to use pfSense as the firewall/gateway (with dedicated NIC for external interface) to replace my old iptables-based linux firewall/gateway/proxy.

    There's an old thread about doing this, based on BSD8.1, found here:,37693.0.html

    I've been working through that, adapting to pfsense-2-1 as I go.

    If either of you (or anyone else makes progress, please post here).

  • Ok, I spent a chunk of today working through this, and now have a PV domain up and running.  I converted a PVHVM install to paravirt as it was an image I had conveniently available, but you can just upgrade a regular HVM install the same way (I haven't tried working through a PV install yet).

    Attached is my modified pfSense_SMP.8 file.

    You'll need a build environment set up per the devwiki, and then drop the attached pfSense_SMP.8 file into /home/pfsense/tools/builder_scripts/conf/kernel

    After you've built your iso or whatever (I used the script in  /home/pfsense/tools/builder_scripts ) , you need to copy this kernel file from the build environment to the dom0:


    Then do the following:

    cd /tmp/kernels/pfSense_SMP.8
    tar zcvf boot.tgz boot

    With your HVM domU running, go to its shell, and use scp to copy the boot.tgz file you just made to the host, unpack it, and copy it over the HVM kernel etc:

    scp root@ boot.tgz
    tar zxvf boot.tgz
    cp -a boot /

    Now with the new kernel there, you can shut down the pfsense HVM host, and make your xen config file.

    Create your paravirt xen config file, something like this:

    name = 'pfsense21'
    kernel = '/root/kernels/pfsense21/SMP/kernel'
    extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
    disk = ['phy:/dev/vg_hdd/pfsense21pv,xvda,w']
    memory = 512
    bootload = 'pygrub'
    vcpus = 1
    vif = [ 'bridge=xenbr0, mac=00:aa:0a:14:01:97', 'bridge=xenbr0, mac=00:aa:0a:14:01:96' ]

    (where the disk is pointing to an existing HVM install of pfSense - I like to use LVM)

    I just tried passing through a pci device and it hasn't worked on the first attempt, but its late so I'll have a look into that tomorrow evening.


  • So a quick play and I can't get a PCI device passed through - but this could be my complete inexperience with freebsd, or it could be a lack of support for pcifront (the xen pci frontend).
    I've got passthrough working with the same device on the same dom0 to other VM's, including a freeBSD HVM and a Linux paravirt.

    "xl pci-list pfsense21" shows the device attached to the PV pfsense VM, but "pciconf -l" shows nothing (on paravirt hosts it usually just shows only the passed-through devices - unlike HVM).

    The device is an Intel 82574L NIC - which uses the "em0"" device ID ("driver em" in the GENERIC BSD kernel conf file) - I've added it to my pfSense_SMP.8 but it still isn't being picked up.

    Does anyone with more Xen on BSD experience know which driver package adds support for Xen's pcifront to a domU?  (xenpci is for HVM only I believe?)

    I'll have to find a xen/bsd mailing list - I suspect I'm pushing the boundaries here of BSD xen support.

  • did you use the permissive option when making the VM?

    eg. pci=['08:00.0,permissive=1']

    only works with xl toolstack, xm required you to go it differently.

  • Thanks for the idea - I'm using the xl toolset and had tried the "pci_permissive=1" general option which applies to all pci devices, with no luck.  Same goes for the single device type like you listed.

    I also noticed that the config I built doesn't support SMP, so while rebuilding another PV kernel I added the SMP options.  They cause a kernel panic which bumps me out to the kernel debugger (panic: HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, &ctxt): /usr/pfSensesrc/src/sys/i386/xen/mp_machdep.c:930).

    At this stage I can either accept that pfsense2-1 works only with a pair of virtualised network interfaces (which means no traffic shaping, and no physically separate DMZ subnet), or find another solution.  As this is for my home server and a hobby, I'll spend some more time on it  :D

    So the next thing I'm trying is to get a freeBSD 10 BETA paravirt machine up and running to test the xen status of the latest build, as pfsense2.2 is, I believe, moving to BSD 10?  As an aside - FreeBSD10 includes the XENHVM stuff as a kernel module in the default build.. so it creates xn0 etc. with the default kernel.

    If I can get plain freeBSD 10 working with pci passthrough, then I'll have a go at building pfsense on bsd10 (presumably thats what the .10 files in the kernel conf directory are for?) .

    If anyone makes progress and gets further than me please update this thread!

  • Ok, so it turns out that PV is not worth bothering with on freeBSD at present..

    No PCI passthrough
    No SMP support
    Memory limit around 700 megs

    So I'm back on the PVHVM track.

  • sorry for my absence :(

    i got one "alpha-hacking-version" running with Pfsense 2.1 and Freebsd 8.3 i386 in PV mode  .. but some error are there .

    i am not sure what all i did but i will write what i mostly remeber its 8am xd .

    the pfSense image was made under freebsd8.1 i386
    the kernel was made under freebsd8.3 i386

    under /usr/pfSensesrc/src/sys/modules  and /usr/pfSensesrc/src/sys/modules/netgraph, there are one Makefile per Folder (the Makefiles without endings).
    inside of the Makefile there are the Modules and stuff, i removed one by one if the compile process gave me an error, and i remeber one file was missing but i dont know the name if you looking for there are 3 candites of search results, i think i took the closest one to xen or pci .. i kdont know.

    the part of compiling i used under /usr/pfSensesrc/src

    export MAKEOBJDIRPREFIX=/other/dir

    csh users use setenv

    Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):

    make buildworld && make buildkernel KERNCONF=XEN

    Our file-backed virtual disk should still be mounted, so now we can install to it:

    export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XEN

    after that i got the kernel from freebsd 8.3 "xenified" :D
    to find under  /usr/obj/usr/pfSensesrc/src/sys/XEN

    so far my info right know .. i will more test around and if i got a clear result how to do then i will post :)

    i put my stuff in one file maybe some one can need it
    btw use ufs:/dev/xbd0s1a at the prompt while pfsense is halting at boot i did not set the right parameter in the .cfg (~150MB) build env(~660MB)

    i need more testing cause i just got the build finished right now :)


  • Official support for Xen (and HyperV) will be forthcoming.  I can't say exactly 'when'.

    VMware (officially) and EC2 are up first.

  • That sounds awesome gonzopancho! Please look into adding ALTQ support to the Xen drivers for FreeBSD if possible. Everything else seems to be working silky smooth for PVHVM atleast.

  • Gonzopancho, thats great news thanks!

    There are a couple of use-cases where pfSense/XEN makes a really good combo - one is secure firewall/VPN/single point of access for a cloud-style cluster of virtaul machines (I just saw you guys have released an Amazon AMI, well done!).

    The other is as part of a consolidated server solution which incorporates gateway/proxy/vpn/file/mail/backup/application servers as VM's running on a single-box for small businesses, reducing hardware costs.

    One comment on Xen support - I asked the xen-bsd mailing list, was advised that pure paravirt is taking a backseat to XenHVM and dom0 development currently, and that there's no support for pci-passthrough on full paravirt freeBSD domUI's.  The implication for pfSense is that until bsd support for paravirt domU's expands, its not possible to use a physically separate subnet for DMZ or external interfaces on a pure paravirt domU, only on XENHVM ones.

    Great news that Xen is on your radar - I'll be happy to help with testing.

  • hi all

    i spent some time to get pfSense2.1 in Paravirtualization (PV) Mode running succesfull on a i386 machine :)

    build the image as written in the

    i build an nano image (.img), if everything goes well to create the usually nano image, then copy the XEN KERNELCONF from pfSense to the pfSensesrc folder

    cause i need a kernel to get pfSense running from this kernel in PV mode.

    cp /home/pfsense/tools/builder_scripts/conf/kernel/pfSense_XEN.8  /usr/pfSensesrc/src/sys/i386/conf

    and comment out this values in the KERNELCONF, (some values are twice inside the KERNCONF):

    ##options 	PREEMPTION		# Enable kernel thread preemption
    ##options		KDB
    ##nooptions       KDB_TRACE 
    ##options		DDB                     # Support DDB.
    ##nooptions       GDB                     # Support remote GDB.
    ##nooptions	INVARIANTS
    ##nooptions	INVARIANT_SUPPORT
    ##nooptions	WITNESS
    ##nooptions	WITNESS_SKIPSPIN
    ##options		GEOM_PART_MBR
    ##options		GEOM_PART_BSD
    ##options		NETGRAPH_VLAN
    ##options         ALTQ
    ##options         ALTQ_CBQ
    ##options         ALTQ_RED
    ##options         ALTQ_RIO
    ##options         ALTQ_HFSC
    ##options         ALTQ_PRIQ
    ##device		bktr			# bktr -- Brooktree Bt848/849/878/879 and Pinnacle PCTV video capture
    ##device		ale				# ale -- Atheros AR8121/AR8113/AR8114 Gigabit/Fast Ethernet driver
    ##device		et				# et(4) for AGERE ET1310 fastE and gigE
    ##device		ed				# NE[12]000, SMC Ultra, 3c503, DS8390 cards
    ##device		mxge			# mxge - Myricom Myri10GE 10 Gigabit Ethernet adapter driver
    ##device		cxgb			# cxgb -- Chelsio T3 10 Gigabit Ethernet adapter driver
    ##device		ae				# ae -- Attansic/Atheros L2 FastEthernet controller driver
    ##device		cas		# Sun Cassini/Cassini+ and National Semiconductor DP83065 Saturn
    ##device		hifn            # Hifn 7951, 7781, etc.
    ##device		ubsec           # Broadcom 5501, 5601, 58xx
    ##device		udav            # Davicom DM9601 USB Ethernet driver
    ##options		ALTQ_FAIRQ

    this is my /etc/make.conf
    i dont really need to build all:

    MODULES_OVERRIDE = ipfw ipdivert dummynet fdescfs runfw if_stf
    WITHOUT_MODULES= aha ahb amd cxgb dpt drm hptmv ida malo mps mwl nve sound sym trm xfs

    going to build now the kernel:

    mkdir /root/myboot
    cd /usr/pfSensesrc/src/
    make KERNCONF=pfSense_XEN.8 DESTDIR=/root/myboot kernel

    if all went good then we have now a PV Kernel under /root/myboot/boot/kernel/kernel

    then just copy the nano image and the kernel it to your xendomain folder

    and this is my pvsense.cfg configuration for XEN what i use to run the image

    kernel = "/home/xendomains/pfsensetest/kernel"
    extra = "vfs.root.mountfrom=ufs:/dev/xbd0s1a"
    #bootload = '/usr/bin/pygrub'
    memory = 512
    name = "sen"
    vcpus = 1
    nics = 2
    #vif = [ 'mac=aa:00:00:50:02:f1, bridge=bridge0' ]
    vif = ['mac=00:16:3e:0f:12:df, bridge=bridge0,model=ne2k_pci', 'mac=00:16:3e:45:18:2a, bridge=bridge1,model=ne2k_pci']
    disk = [ 'file:/home/xendomains/pfsensetest/disk.img,0x01,w' ]
    #root = "xbd0s1"

    i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D

    i have attached 2xen patches, just replace the files from the archive, run ./ and then build the kernel as written above


  • @ren22:

    i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D

    I think the real strategy here is to wait until pfSense 2.2 (based on FreeBSD 10) for real Xen support.