Any way to build pfSense2.1 i386 for XEN4 PV Paravirt mode?

Ok, I spent a chunk of today working through this, and now have a PV domain up and running. I converted a PVHVM install to paravirt as it was an image I had conveniently available, but you can just upgrade a regular HVM install the same way (I haven't tried working through a PV install yet).

Attached is my modified pfSense_SMP.8 file.

You'll need a build environment set up per the devwiki, and then drop the attached pfSense_SMP.8 file into /home/pfsense/tools/builder_scripts/conf/kernel

After you've built your iso or whatever (I used the script build_kernels.sh in /home/pfsense/tools/builder_scripts ) , you need to copy this kernel file from the build environment to the dom0:

/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8/kernel

Then do the following:


cd /tmp/kernels/pfSense_SMP.8
tar zcvf boot.tgz boot

With your HVM domU running, go to its shell, and use scp to copy the boot.tgz file you just made to the host, unpack it, and copy it over the HVM kernel etc:


scp root@192.168.0.35:/tmp/kernels/pfSense_SMP.8/boot.tgz boot.tgz
tar zxvf boot.tgz
cp -a boot /

Now with the new kernel there, you can shut down the pfsense HVM host, and make your xen config file.

Create your paravirt xen config file, something like this:


name = 'pfsense21'

kernel = '/root/kernels/pfsense21/SMP/kernel'
extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
disk = ['phy:/dev/vg_hdd/pfsense21pv,xvda,w']
memory = 512
bootload = 'pygrub'
vcpus = 1
vif = [ 'bridge=xenbr0, mac=00:aa:0a:14:01:97', 'bridge=xenbr0, mac=00:aa:0a:14:01:96' ]

(where the disk is pointing to an existing HVM install of pfSense - I like to use LVM)

I just tried passing through a pci device and it hasn't worked on the first attempt, but its late so I'll have a look into that tomorrow evening.

pfSense_SMP.8.txt

So a quick play and I can't get a PCI device passed through - but this could be my complete inexperience with freebsd, or it could be a lack of support for pcifront (the xen pci frontend).
I've got passthrough working with the same device on the same dom0 to other VM's, including a freeBSD HVM and a Linux paravirt.

"xl pci-list pfsense21" shows the device attached to the PV pfsense VM, but "pciconf -l" shows nothing (on paravirt hosts it usually just shows only the passed-through devices - unlike HVM).

The device is an Intel 82574L NIC - which uses the "em0"" device ID ("driver em" in the GENERIC BSD kernel conf file) - I've added it to my pfSense_SMP.8 but it still isn't being picked up.

Does anyone with more Xen on BSD experience know which driver package adds support for Xen's pcifront to a domU? (xenpci is for HVM only I believe?)

I'll have to find a xen/bsd mailing list - I suspect I'm pushing the boundaries here of BSD xen support.

Sabrewarrior

did you use the permissive option when making the VM?

eg. pci=['08:00.0,permissive=1']

only works with xl toolstack, xm required you to go it differently.

Thanks for the idea - I'm using the xl toolset and had tried the "pci_permissive=1" general option which applies to all pci devices, with no luck. Same goes for the single device type like you listed.

I also noticed that the config I built doesn't support SMP, so while rebuilding another PV kernel I added the SMP options. They cause a kernel panic which bumps me out to the kernel debugger (panic: HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, &ctxt): /usr/pfSensesrc/src/sys/i386/xen/mp_machdep.c:930).

At this stage I can either accept that pfsense2-1 works only with a pair of virtualised network interfaces (which means no traffic shaping, and no physically separate DMZ subnet), or find another solution. As this is for my home server and a hobby, I'll spend some more time on it :D

So the next thing I'm trying is to get a freeBSD 10 BETA paravirt machine up and running to test the xen status of the latest build, as pfsense2.2 is, I believe, moving to BSD 10? As an aside - FreeBSD10 includes the XENHVM stuff as a kernel module in the default build.. so it creates xn0 etc. with the default kernel.

If I can get plain freeBSD 10 working with pci passthrough, then I'll have a go at building pfsense on bsd10 (presumably thats what the .10 files in the kernel conf directory are for?) .

If anyone makes progress and gets further than me please update this thread!

Ok, so it turns out that PV is not worth bothering with on freeBSD at present..

No PCI passthrough
No SMP support
Memory limit around 700 megs

So I'm back on the PVHVM track.

ren22

sorry for my absence :(

i got one "alpha-hacking-version" running with Pfsense 2.1 and Freebsd 8.3 i386 in PV mode .. but some error are there .

i am not sure what all i did but i will write what i mostly remeber its 8am xd .

the pfSense image was made under freebsd8.1 i386
the kernel was made under freebsd8.3 i386

under /usr/pfSensesrc/src/sys/modules and /usr/pfSensesrc/src/sys/modules/netgraph, there are one Makefile per Folder (the Makefiles without endings).
inside of the Makefile there are the Modules and stuff, i removed one by one if the compile process gave me an error, and i remeber one file was missing but i dont know the name if you looking for there are 3 candites of search results, i think i took the closest one to xen or pci .. i kdont know.

the part of compiling i used under /usr/pfSensesrc/src

export MAKEOBJDIRPREFIX=/other/dir

csh users use setenv

Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):

make buildworld && make buildkernel KERNCONF=XEN

Our file-backed virtual disk should still be mounted, so now we can install to it:

export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XEN

after that i got the kernel from freebsd 8.3 "xenified" :D
to find under /usr/obj/usr/pfSensesrc/src/sys/XEN

so far my info right know .. i will more test around and if i got a clear result how to do then i will post :)

i put my stuff in one file maybe some one can need it
btw use ufs:/dev/xbd0s1a at the prompt while pfsense is halting at boot i did not set the right parameter in the .cfg

http://d01.megashares.com/dl/KYkRoDA/pfsense21-pv.tar.gz (~150MB)
http://www.gigasize.com/get/rx6ls9d0gzd build env(~660MB)

i need more testing cause i just got the build finished right now :)

sources:
http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso
http://forum.pfsense.org/index.php?topic=37693.0
http://forums.freebsd.org/showthread.php?t=10268

Official support for Xen (and HyperV) will be forthcoming. I can't say exactly 'when'.

VMware (officially) and EC2 are up first.

Sabrewarrior

That sounds awesome gonzopancho! Please look into adding ALTQ support to the Xen drivers for FreeBSD if possible. Everything else seems to be working silky smooth for PVHVM atleast.

Gonzopancho, thats great news thanks!

There are a couple of use-cases where pfSense/XEN makes a really good combo - one is secure firewall/VPN/single point of access for a cloud-style cluster of virtaul machines (I just saw you guys have released an Amazon AMI, well done!).

The other is as part of a consolidated server solution which incorporates gateway/proxy/vpn/file/mail/backup/application servers as VM's running on a single-box for small businesses, reducing hardware costs.

One comment on Xen support - I asked the xen-bsd mailing list, was advised that pure paravirt is taking a backseat to XenHVM and dom0 development currently, and that there's no support for pci-passthrough on full paravirt freeBSD domUI's. The implication for pfSense is that until bsd support for paravirt domU's expands, its not possible to use a physically separate subnet for DMZ or external interfaces on a pure paravirt domU, only on XENHVM ones.

Great news that Xen is on your radar - I'll be happy to help with testing.

ren22

hi all

i spent some time to get pfSense2.1 in Paravirtualization (PV) Mode running succesfull on a i386 machine :)

build the image as written in the https://devwiki.pfsense.org/DevelopersBootStrapAndDevIso

i build an nano image (.img), if everything goes well to create the usually nano image, then copy the XEN KERNELCONF from pfSense to the pfSensesrc folder

cause i need a kernel to get pfSense running from this kernel in PV mode.

cp /home/pfsense/tools/builder_scripts/conf/kernel/pfSense_XEN.8  /usr/pfSensesrc/src/sys/i386/conf

and comment out this values in the KERNELCONF, (some values are twice inside the KERNCONF):

##options 	PREEMPTION		# Enable kernel thread preemption
##options		KDB
##nooptions       KDB_TRACE 
##options		DDB                     # Support DDB.
##nooptions       GDB                     # Support remote GDB.
##nooptions	INVARIANTS
##nooptions	INVARIANT_SUPPORT
##nooptions	WITNESS
##nooptions	WITNESS_SKIPSPIN
##options		GEOM_PART_MBR
##options		GEOM_PART_BSD
##options		NETGRAPH_VLAN
##options         ALTQ
##options         ALTQ_CBQ
##options         ALTQ_RED
##options         ALTQ_RIO
##options         ALTQ_HFSC
##options         ALTQ_PRIQ
##device		bktr			# bktr -- Brooktree Bt848/849/878/879 and Pinnacle PCTV video capture
##device		ale				# ale -- Atheros AR8121/AR8113/AR8114 Gigabit/Fast Ethernet driver
##device		et				# et(4) for AGERE ET1310 fastE and gigE
##device		ed				# NE[12]000, SMC Ultra, 3c503, DS8390 cards
##device		mxge			# mxge - Myricom Myri10GE 10 Gigabit Ethernet adapter driver
##device		cxgb			# cxgb -- Chelsio T3 10 Gigabit Ethernet adapter driver
##device		ae				# ae -- Attansic/Atheros L2 FastEthernet controller driver
##device		cas		# Sun Cassini/Cassini+ and National Semiconductor DP83065 Saturn
##device		hifn            # Hifn 7951, 7781, etc.
##device		ubsec           # Broadcom 5501, 5601, 58xx
##device		udav            # Davicom DM9601 USB Ethernet driver
##options		ALTQ_FAIRQ

this is my /etc/make.conf
i dont really need to build all:

MODULES_OVERRIDE = ipfw ipdivert dummynet fdescfs runfw if_stf
WITHOUT_MODULES= aha ahb amd cxgb dpt drm hptmv ida malo mps mwl nve sound sym trm xfs

going to build now the kernel:

mkdir /root/myboot
cd /usr/pfSensesrc/src/
make KERNCONF=pfSense_XEN.8 DESTDIR=/root/myboot kernel

if all went good then we have now a PV Kernel under /root/myboot/boot/kernel/kernel

then just copy the nano image and the kernel it to your xendomain folder

and this is my pvsense.cfg configuration for XEN what i use to run the image

kernel = "/home/xendomains/pfsensetest/kernel"
extra = "vfs.root.mountfrom=ufs:/dev/xbd0s1a"
#bootload = '/usr/bin/pygrub'
memory = 512
name = "sen"
vcpus = 1
nics = 2
#vif = [ 'mac=aa:00:00:50:02:f1, bridge=bridge0' ]
vif = ['mac=00:16:3e:0f:12:df, bridge=bridge0,model=ne2k_pci', 'mac=00:16:3e:45:18:2a, bridge=bridge1,model=ne2k_pci']
disk = [ 'file:/home/xendomains/pfsensetest/disk.img,0x01,w' ]
#root = "xbd0s1"
#bootloader="pygrub"

i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D
thanks

i have attached 2xen patches, just replace the files from the archive, run ./apply_kernel_patches.sh and then build the kernel as written above

pfsense2.1.1pre-xen.patches.tar.gz.txt

@ren22:

i hope we get soon pfSense on freebsd9 or 10 running with better XEN support :D
thanks

I think the real strategy here is to wait until pfSense 2.2 (based on FreeBSD 10) for real Xen support.