Any way to build pfSense2.1 i386 for XEN4 PV Paravirt mode?
i am trying to make an pfSense2.1-i386(nano) Image for XEN4 in PV Paravirtual mode.
making an normal nano image is no problem.
but i a run into many compilings errors if i try to build the nano image with a modified Kernel config.
trying to mix the original FreeBSD8.3 XEN Kernel config with the pfSense_wrap.8.i386 breaks the compilation.
any way to get pfSense2.1 with XEN PV mode to run ?
later if i am on compiling machine again then i would post the error messages while compiling.
second attemp to post
hello ren22, I was able to compile a pfSense i386 Xen PV image but never was successful in making it boot properly. I will have some time to mess around with the configuration again next week and see if I can help.
Sabre - you might find the config info helpful towards the end of the post in that other thread. Could you also post your updated "/home/pfsense/tools/builder_scripts/conf/kernel/pfSense_wrap.8.i386" with the XEN options?
I made a diff of the BSD i386 vs. XEN conf files, and used them as the basis for updating the current pfSense_wrap.8.i386, but I'm getting a compile error building the kernel.
Good to know I'm not the only one trying to build a Paravirt-capable kernel for pfSense. I'm new to BSD, but have a lot of Linux experience, so I'm working through this. I'm building an all-in-one box and would like to use pfSense as the firewall/gateway (with dedicated NIC for external interface) to replace my old iptables-based linux firewall/gateway/proxy.
There's an old thread about doing this, based on BSD8.1, found here:
I've been working through that, adapting to pfsense-2-1 as I go.
If either of you (or anyone else makes progress, please post here).
Ok, I spent a chunk of today working through this, and now have a PV domain up and running. I converted a PVHVM install to paravirt as it was an image I had conveniently available, but you can just upgrade a regular HVM install the same way (I haven't tried working through a PV install yet).
Attached is my modified pfSense_SMP.8 file.
You'll need a build environment set up per the devwiki, and then drop the attached pfSense_SMP.8 file into /home/pfsense/tools/builder_scripts/conf/kernel
After you've built your iso or whatever (I used the script build_kernels.sh in /home/pfsense/tools/builder_scripts ) , you need to copy this kernel file from the build environment to the dom0:
Then do the following:
cd /tmp/kernels/pfSense_SMP.8 tar zcvf boot.tgz boot
With your HVM domU running, go to its shell, and use scp to copy the boot.tgz file you just made to the host, unpack it, and copy it over the HVM kernel etc:
scp firstname.lastname@example.org:/tmp/kernels/pfSense_SMP.8/boot.tgz boot.tgz tar zxvf boot.tgz cp -a boot /
Now with the new kernel there, you can shut down the pfsense HVM host, and make your xen config file.
Create your paravirt xen config file, something like this:
name = 'pfsense21' kernel = '/root/kernels/pfsense21/SMP/kernel' extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a' disk = ['phy:/dev/vg_hdd/pfsense21pv,xvda,w'] memory = 512 bootload = 'pygrub' vcpus = 1 vif = [ 'bridge=xenbr0, mac=00:aa:0a:14:01:97', 'bridge=xenbr0, mac=00:aa:0a:14:01:96' ]
(where the disk is pointing to an existing HVM install of pfSense - I like to use LVM)
I just tried passing through a pci device and it hasn't worked on the first attempt, but its late so I'll have a look into that tomorrow evening.
So a quick play and I can't get a PCI device passed through - but this could be my complete inexperience with freebsd, or it could be a lack of support for pcifront (the xen pci frontend).
I've got passthrough working with the same device on the same dom0 to other VM's, including a freeBSD HVM and a Linux paravirt.
"xl pci-list pfsense21" shows the device attached to the PV pfsense VM, but "pciconf -l" shows nothing (on paravirt hosts it usually just shows only the passed-through devices - unlike HVM).
The device is an Intel 82574L NIC - which uses the "em0"" device ID ("driver em" in the GENERIC BSD kernel conf file) - I've added it to my pfSense_SMP.8 but it still isn't being picked up.
Does anyone with more Xen on BSD experience know which driver package adds support for Xen's pcifront to a domU? (xenpci is for HVM only I believe?)
I'll have to find a xen/bsd mailing list - I suspect I'm pushing the boundaries here of BSD xen support.
did you use the permissive option when making the VM?
only works with xl toolstack, xm required you to go it differently.
Thanks for the idea - I'm using the xl toolset and had tried the "pci_permissive=1" general option which applies to all pci devices, with no luck. Same goes for the single device type like you listed.
I also noticed that the config I built doesn't support SMP, so while rebuilding another PV kernel I added the SMP options. They cause a kernel panic which bumps me out to the kernel debugger (panic: HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, &ctxt): /usr/pfSensesrc/src/sys/i386/xen/mp_machdep.c:930).
At this stage I can either accept that pfsense2-1 works only with a pair of virtualised network interfaces (which means no traffic shaping, and no physically separate DMZ subnet), or find another solution. As this is for my home server and a hobby, I'll spend some more time on it :D
So the next thing I'm trying is to get a freeBSD 10 BETA paravirt machine up and running to test the xen status of the latest build, as pfsense2.2 is, I believe, moving to BSD 10? As an aside - FreeBSD10 includes the XENHVM stuff as a kernel module in the default build.. so it creates xn0 etc. with the default kernel.
If I can get plain freeBSD 10 working with pci passthrough, then I'll have a go at building pfsense on bsd10 (presumably thats what the .10 files in the kernel conf directory are for?) .
If anyone makes progress and gets further than me please update this thread!
Ok, so it turns out that PV is not worth bothering with on freeBSD at present..
No PCI passthrough
No SMP support
Memory limit around 700 megs
So I'm back on the PVHVM track.
sorry for my absence :(
i got one "alpha-hacking-version" running with Pfsense 2.1 and Freebsd 8.3 i386 in PV mode .. but some error are there .
i am not sure what all i did but i will write what i mostly remeber its 8am xd .
the pfSense image was made under freebsd8.1 i386
the kernel was made under freebsd8.3 i386
under /usr/pfSensesrc/src/sys/modules and /usr/pfSensesrc/src/sys/modules/netgraph, there are one Makefile per Folder (the Makefiles without endings).
inside of the Makefile there are the Modules and stuff, i removed one by one if the compile process gave me an error, and i remeber one file was missing but i dont know the name if you looking for there are 3 candites of search results, i think i took the closest one to xen or pci .. i kdont know.
the part of compiling i used under /usr/pfSensesrc/src
csh users use setenv
Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):
Our file-backed virtual disk should still be mounted, so now we can install to it:
after that i got the kernel from freebsd 8.3 "xenified" :D
to find under /usr/obj/usr/pfSensesrc/src/sys/XEN
so far my info right know .. i will more test around and if i got a clear result how to do then i will post :)
i put my stuff in one file maybe some one can need it
btw use ufs:/dev/xbd0s1a at the prompt while pfsense is halting at boot i did not set the right parameter in the .cfg
http://www.gigasize.com/get/rx6ls9d0gzd build env(~660MB)
i need more testing cause i just got the build finished right now :)
Official support for Xen (and HyperV) will be forthcoming. I can't say exactly 'when'.
VMware (officially) and EC2 are up first.
That sounds awesome gonzopancho! Please look into adding ALTQ support to the Xen drivers for FreeBSD if possible. Everything else seems to be working silky smooth for PVHVM atleast.
Gonzopancho, thats great news thanks!
There are a couple of use-cases where pfSense/XEN makes a really good combo - one is secure firewall/VPN/single point of access for a cloud-style cluster of virtaul machines (I just saw you guys have released an Amazon AMI, well done!).
The other is as part of a consolidated server solution which incorporates gateway/proxy/vpn/file/mail/backup/application servers as VM's running on a single-box for small businesses, reducing hardware costs.
One comment on Xen support - I asked the xen-bsd mailing list, was advised that pure paravirt is taking a backseat to XenHVM and dom0 development currently, and that there's no support for pci-passthrough on full paravirt freeBSD domUI's. The implication for pfSense is that until bsd support for paravirt domU's expands, its not possible to use a physically separate subnet for DMZ or external interfaces on a pure paravirt domU, only on XENHVM ones.
Great news that Xen is on your radar - I'll be happy to help with testing.
i spent some time to get pfSense2.1 in Paravirtualization (PV) Mode running succesfull on a i386 machine :)
build the image as written in the https://devwiki.pfsense.org/DevelopersBootStrapAndDevIso
i build an nano image (.img), if everything goes well to create the usually nano image, then copy the XEN KERNELCONF from pfSense to the pfSensesrc folder
cause i need a kernel to get pfSense running from this kernel in PV mode.
cp /home/pfsense/tools/builder_scripts/conf/kernel/pfSense_XEN.8 /usr/pfSensesrc/src/sys/i386/conf
and comment out this values in the KERNELCONF, (some values are twice inside the KERNCONF):
##options PREEMPTION # Enable kernel thread preemption ##options KDB ##nooptions KDB_TRACE ##options DDB # Support DDB. ##nooptions GDB # Support remote GDB. ##nooptions INVARIANTS ##nooptions INVARIANT_SUPPORT ##nooptions WITNESS ##nooptions WITNESS_SKIPSPIN ##options GEOM_PART_MBR ##options GEOM_PART_BSD ##options NETGRAPH_VLAN ##options ALTQ ##options ALTQ_CBQ ##options ALTQ_RED ##options ALTQ_RIO ##options ALTQ_HFSC ##options ALTQ_PRIQ ##device bktr # bktr -- Brooktree Bt848/849/878/879 and Pinnacle PCTV video capture ##device ale # ale -- Atheros AR8121/AR8113/AR8114 Gigabit/Fast Ethernet driver ##device et # et(4) for AGERE ET1310 fastE and gigE ##device ed # NE000, SMC Ultra, 3c503, DS8390 cards ##device mxge # mxge - Myricom Myri10GE 10 Gigabit Ethernet adapter driver ##device cxgb # cxgb -- Chelsio T3 10 Gigabit Ethernet adapter driver ##device ae # ae -- Attansic/Atheros L2 FastEthernet controller driver ##device cas # Sun Cassini/Cassini+ and National Semiconductor DP83065 Saturn ##device hifn # Hifn 7951, 7781, etc. ##device ubsec # Broadcom 5501, 5601, 58xx ##device udav # Davicom DM9601 USB Ethernet driver ##options ALTQ_FAIRQ
this is my /etc/make.conf
i dont really need to build all:
MODULES_OVERRIDE = ipfw ipdivert dummynet fdescfs runfw if_stf WITHOUT_MODULES= aha ahb amd cxgb dpt drm hptmv ida malo mps mwl nve sound sym trm xfs
going to build now the kernel:
mkdir /root/myboot cd /usr/pfSensesrc/src/ make KERNCONF=pfSense_XEN.8 DESTDIR=/root/myboot kernel
if all went good then we have now a PV Kernel under /root/myboot/boot/kernel/kernel
then just copy the nano image and the kernel it to your xendomain folder
and this is my pvsense.cfg configuration for XEN what i use to run the image
kernel = "/home/xendomains/pfsensetest/kernel" extra = "vfs.root.mountfrom=ufs:/dev/xbd0s1a" #bootload = '/usr/bin/pygrub' memory = 512 name = "sen" vcpus = 1 nics = 2 #vif = [ 'mac=aa:00:00:50:02:f1, bridge=bridge0' ] vif = ['mac=00:16:3e:0f:12:df, bridge=bridge0,model=ne2k_pci', 'mac=00:16:3e:45:18:2a, bridge=bridge1,model=ne2k_pci'] disk = [ 'file:/home/xendomains/pfsensetest/disk.img,0x01,w' ] #root = "xbd0s1" #bootloader="pygrub"
i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D
i have attached 2xen patches, just replace the files from the archive, run ./apply_kernel_patches.sh and then build the kernel as written above
i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D
I think the real strategy here is to wait until pfSense 2.2 (based on FreeBSD 10) for real Xen support.