Xen PV domU - How to run pfSense in paravirtulised mode

nexus

Abstract
The main purpose of this topic is to provide a simple way and share patches to build a pfSense DomU in paravirtualised mode for Xen Hypervisor.

Why I'm doing that : For the challenge and because my personnal Xen Hypervisor Host does'nt have any Virtualisation set built in. (Atom)

Good to know before building
Xen Kernel for domU in RELENG_8_1 (FreeBSD 8.1) is not really stable, I got a lot of panic and a lot of feature where not present at this stage of freeze in Xen code.

I started in two way :

Try to backport Xen kernel code the the RELENG_8_1 kernel, hard work and I'm not familliar with code versionning for FreeBSD so I give up.
Build pfSense based on RELENG_8_2 (Xen kernel is more complete at this stage).

Building your own pfSense domU

1 - Before anything else, you must install a freebsd8.1 from scrach (I run it on another VM)
2 - Follow the instruction to setup your own pfSense build scratchbox ( http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso )
3 - Make sure that the building process is working by following the instruction in the wiki
4 - Now we need to make some custom things in order to make it build for RELENG_8_2

Customisation 1 : Make the Kernel Xen Aware
Create your own KERNCONF file to add XEN support (and keeping pfsense stuff in it). For this tack I took the pfsense_wrap.8.i386 file and the XEN one from freebsd sources and mixed them.

Warning XEN is including PAE so there some device to remove (scsi stuff, and some others) because they stop when compiling complaining of memory. In my file I remove all the devices including wireless because I don't need them at the moment. Feel free to custom yourself if you need specific device with PCIpasstrough from Xen (excep the need of wireless device, I don't see others to keep in domU).

See below for my pfSense_wrap.8.XEN file.

To use your custom file, override the existing one (make a backup before) in /home/pfsense/tools/builder_script/conf/pfSense_wrap.8.i386

Next you have to modify /home/pfsense/tools/builder_scripts/conf/src.conf.embedded.8 to add the module we don't want to build (using makeoption WITHOUT_MODULES in KERNCONF seems not working fine with RELENG_8_2). So add :

WITHOUT_MODULES= aha ahb amd cxgb dpt drm hptmv ida malo mps mwl nve sound sym trm xfs

Customisation 2 : Modify some scripts to allow Kernel get compiled

Next we need to remove a line in ome/pfsense/tools/builder_scripts/pfsense_local.sh because we add an override witch is not compatible with XEN Kernel. Modify the line :

export MODULES_OVERRIDE=${MODULES_OVERRIDE:-"i2c ipmi acpi ndis ipfw ipdivert dummynet fdescfs cpufreq opensolaris zfs glxsb runfw if_stf"}

by

export MODULES_OVERRIDE=${MODULES_OVERRIDE:-"ipfw ipdivert dummynet fdescfs runfw if_stf"}

Customisation 3 : Modify patches to work with RELENG_8_2

You can find attached the modified diff file to work with RELENG_8_2, copy/replace them to /home/pfsense/tools/patches/RELENG_8_2

bsnmp.diff
carp_correct_mac.diff
divert.RELENG_8.diff
if_lagg_vlan_handling.diff
ipfw.diff
mbuf_tag.diff

OK we are almost done now !

Customisation 4 : Set the right release to build

Use the menu.sh and go to version to configure building pfSense_2_0 with FreeBSD_8_1

Exit the menu.sh and go to modify /home/pfsense/tools/builder_script/pfsense-locale.conf and replace any RELENG_8_1 occurences with RELENG_8_2.

We are ready to build !

Go to the menu.sh and select build nanobsd image, if I don't miss anything at this point, it should compile and build without errors.

How to install my fresh image as domU ?

Is fairly simple, locate the img you just build (normaly /tmp/builder/nanobsd.full.img) and copy it on your dom0.

Locate the kernel in building tree /usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386/kernel and copy it on your dom0

In my configuration I use lvm as disk management for my domU but it should work with any other disk method.

Dump the nanobsd.img with dd as below :

dd if=nanobsd.full.img of=/dev/myVG/pfsense_rootfs bs=16k

and setup you domU according to

kernel = '/boot/pfsense/kernel-pfsense-xen-domU-8.2'
extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
disk = ['phy:/dev/myVG/pfSense_rootfs,xvda,w']
memory = 256
bootload = '/usr/bin/pygrub'
vcpus = 1
vif = ['mac=00:16:3e:0f:12:df, bridge=br8021q,model=', 'mac=00:16:3e:45:18:2a, bridge=eth0,model=']

Your pfsense domU should start !

Some notes and todo …

Using vlan with xen network interface need a patch :

--- sys/dev/xen/netfront/netfront.c  2010-11-24 02:03:03.000000000 +0100
+++ sys/dev/xen/netfront/netfront.c  2011-05-29 17:30:30.000000000 +0200
@@ -2067,6 +2067,10 @@
        }
 #endif
        ifp->if_capenable = ifp->if_capabilities;
+
+       /* Add VLAN support capabilite for xn if
+        ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU;
+        ifp->if_capenable |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU;

        ether_ifattach(ifp, np->mac);
        callout_init(&np->xn_stat_ch, CALLOUT_MPSAFE);

If you got some error like : xennet_get_responses: too many frags 6 > max 5

here is the patch


--- head/sys/dev/xen/netfront/netfront.c	2011/01/29 00:53:58	218055
+++ head/sys/dev/xen/netfront/netfront.c	2011/01/29 02:36:45	218056
@@ -1273,7 +1273,6 @@
 	struct mbuf *m, *m0, *m_prev;
 	grant_ref_t ref = xennet_get_rx_ref(np, *cons);
 	RING_IDX ref_cons = *cons;
-	int max = 5 /* MAX_TX_REQ_FRAGS + (rx->status <= RX_COPY_THRESHOLD) */;
 	int frags = 1;
 	int err = 0;
 	u_long ret;
@@ -1416,20 +1415,10 @@
 		frags++;
 	}
 	*list = m0;
-
-	if (unlikely(frags > max)) {
-		if (net_ratelimit())
-			WPRINTK("Too many frags\n");
-		printf("%s: too many frags %d > max %d\n", __func__, frags,
-		       max);
-		err = E2BIG;
-	}
-
 	*cons += frags;
-
 	*pages_flipped_p = pages_flipped;

-	return err;
+	return (err);
 }

 static void

If you need to use the shapping, you will get an error with the wizard like "You do not have 2 of local interfaces!".

To fix it (not tested yet) you need to add the xen network device in a /etc/inc/interface.inc

function is_altq_capable($int) {
        /* Per:
         * http://www.freebsd.org/cgi/man.cgi?query=altq&manpath=FreeBSD+7.2-current&format=html
                        pfSense_interface_mtu($realhwif, $mtu);
         * Only the following drivers have ALTQ support
         */
        $capable = array("age", "alc", "ale", "an", "ath", "aue", "awi", "bce", "xn"

The wizard and interface will be ok, but you need to add support of ALTQ to xen xn device.

I don't have any patch yet but you can follow in the following man page : http://www.freebsd.org/cgi/man.cgi?query=altq&sektion=9&apropos=0&manpath=FreeBSD+8.2-RELEASE

Feel free to comment or add your own experiments !
Note : All the attachement have been suffixed with txt in order to be attached.

pfSense_wrap.8.XEN.txt
bsnmp.diff.txt
carp_correct_mac.diff.txt
divert.RELENG_8.diff.txt
if_lagg_vlan_handling.diff.txt
ipfw.diff.txt
mbuf_tag.diff.txt

vsnine

How stable is this setup? Could you provide the disk image?

cyruspy

I would really love Xen domU support on PFSense. I've read that 2.0 final won't upgrade to 8.2, so having an official build won't be possible. I'll test this and provide some feedback when possible..

lvvloten

Hi,

Thanks for this very useful post! I am a total newbie in the field of pfSense and/or freeBSD, but your post really helped me to get started.

I want to implement pfSense in a domU on my home server, which has a simple Pentium IV processor without virtualization support. I overcame quite a few problems but now I seem to be stuck. I will start with give a short overview of some of the problems I solved, once I manage (if I manage…) to complete the procedure I will post my solutions in more detail.
Disclaimer: I hardly know what I am doing. My solutions need to be reviewed by someone a bit more exprienced in this matter before it can be called 'reliable'!

What did I encounter?

Even with the modified patches that you provided, I encountered quite a few patches that didn't work. I analyzed and modified those patches by hand and I managed to make all patches work without modifying any of the code itself. See attachments.
One patch turned out to lead to a compile error, I modified the code so it would work. (This might need some review from someone who, unlike me, would actually know what he is doing)
Specifically: in if_rum_pr_144642.diff I changed "return 0;" to "return;". This solved the compile error.
Three pfPorts did not build due to compile errors. I solved two. So far I did not yet manage to install php52-pfSense-module; however, this does not seem to be related to the issue below.

One important addition to the procedure above is that in my pfSense_wrap.8.i386 file (see attachment), I explicitly excluded everything that I could encounter that might be related to USB, SCSI or wireless support. To achieve this I used the 'nodevice' and 'nooptions'. My purpose is that no USB (or SCSI or wireless) related materials will be built - not directly into the kernel, and not as modules. However, as can be seen in the problems I encounter, this does not work as I expected.

So where does it go wrong?
I start the procedure using the /usr/home/pfsense/toold/builder_scripts/build_nano.sh script. I get the same result when I start the procedure from the menu, as described above.

Everything seems to run fine up to the time that it starts buidling the kernel. At "stage 3.1: making dependencies", the following error occurs:

/usr/pfSensesrc/src/sys/dev/usb/net/if_udav.c:72:21: error: usbdevs.h: No such file or directory

I have attached the full output of the build_nano.sh command; normal (build_nano_output-normal.txt) and after uncommenting 'set -x' in the build_nano.sh script (build_nano_output-debug.txt).

I manage to pass this stage by manually making some changes:

Modify if_udav.c line 72: change 'include "usbdevs.h"' to 'include <dev usb="" usbdevs.h="">'</dev>
Manually run the following command:

cd /usr/pfSensesrc/src/sys/dev/usb && /usr/bin/awk -f /usr/pfSensesrc/src/sys/tools/usbdevs2h.awk usbdevs -h

Although I don't have a clue what I am doing here I manage to get past the above error, only to end up in some other compile error in a USB related program.

So my question is:
How can I avoid that any USB related software will be built in the kernel build process? Clearly, excluding it in my pfSense_wrap.8.i386 configuration file is not the way to do it.

Any help would be greatly appreciated!

Kind regards,
Lucas

build_nano_output-normal.txt
build_nano_output-debug.txt
pfSense-diffs.tar.gz.txt
pfSense_wrap.8.i386.txt

CDuv

Hello,

I'm also trying to put a pfSense VM into Xen (running on a Debian 6).

By following the "DevelopersBootStrapAndDevIso" guide (on a VirtualBox-ed FreeBSD VM), I'm running into the following issue when running ./build_iso.sh:

chmod: /usr/local/pfsense-fs/usr/local/lib/php/20060613/ioncube/ioncube_loader_fre_5.2_ts.so: No such file or directory

Installation collected library information (usr/local), please wait…
chroot'ing and running /etc/rc.php_ini_setup
Copying config.xml from conf.default/ to cf/conf/
WARNING: attempt to domain_add(netgraph) after domainfinalize()
Loaded /boot/kernel/ng_socket.ko, id=2
Testing PHP installation in /usr/local/pfsense-fs:
Fatal error: Call to undefined function pfSense_get_interface_addresses() in /etc/inc/globals.inc on line 49

An error occured while testing the php installation in /usr/local/pfsense-fs

####################################
Something went wront, check errors!
####################################

NOTE: a lot of time you can run ./clean_build.sh to resolve.

Press enter to continue.

Clearly, the builder is lacking some PHP components but why?

nexus

cd /home/pfsense/tools/pfPorts/php5/ && make all install clean
cd /home/pfsense/tools/pfPorts/php5-pfSense-module/ && make all install clean

Then run the build again it should fix.

cougarmaster

Please make this post a sticky ;D

nexus

To get sticked I think there is some work to be done by other member. I fell alone with this topic so I give up for the moment (other things to do like my wedding for example).

I think the best way is to use the appliance part of pfSense build tools, we also need a lot of patches from last kernel and xen devices (the xn network device for example that need some improvement if we need to manage VLAN inside pfSense or QOS with bandwidth tweaks).

Feel free to try and Improve the receipe, if this topic is moving on, I will do my best to continue my work.