PfSense 2.0 - Building custom kernel and image for appliance
-
Maybe found a way using extra_options and noextra_options in pfsense-build.conf
and for target nanobsd could be ok I think for deployment
-
It gets me nuts ! Module_override always apply.
From another way, what options and modules are mandatory for pfsense ?
Can I build the kernel from /u/pfsense/src with make build kernel kernconf=xen-conf ?
Thx
-
Ok, I make it works, but have some panic with network, needs to be debbuged. ;D ;D ;D
Step :
-Use the XEN KERNCONF from FreeBSD8.1 and add device and options from pfsense_wrap.8
-Strip some VAR in script (MODULE_OVERRIDE…)
-Build a Nanobsd img
-Extract the kernel from image
-Dump the image in a LV
-Make it bootSome proof :
xm console pfsense WARNING: loader(8) metadata is missing! KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RC1 #15: Tue May 17 13:30:17 CEST 2011 nexus@freebsdhvm.inetpsa.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386 i386 Xen reported: 1600.043 MHz processor. Timecounter "ixen" frequency 1000000000 Hz quality 0 CPU: Intel(R) Atom(TM) CPU N270 @ 1.60GHz (1600.04-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x106c2 Family = 6 Model = 1c Stepping = 2 Features=0xbfe9fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40c39d <sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe>AMD Features=0x100000 <nx>AMD Features2=0x1 <lahf>TSC: P-state invariant real memory = 268435456 (256 MB) avail memory = 254332928 (242 MB) [XEN] IPI cpu=0 irq=128 vector=RESCHEDULE_VECTOR (0) [XEN] IPI cpu=0 irq=129 vector=CALL_FUNCTION_VECTOR (1) xenbus0: <xen devices="">on motherboard [XEN] xen_rtc_probe: probing Hypervisor RTC clock rtc0: <xen hypervisor="" clock="">on motherboard [XEN] xen_rtc_attach: attaching Hypervisor RTC clock xc0: <xen console="">on motherboard padlock0: No ACE support. cryptosoft0: <software crypto="">on motherboard Timecounters tick every 10.000 msec [XEN] hypervisor wallclock nudged; nudging TOD. IPsec: Initialized Security Association Processing. xbd0: 10240MB <virtual block="" device="">at device/vbd/51712 on xenbus0 xn0: <virtual network="" interface="">at device/vif/0 on xenbus0 GEOM: xbd0s1: geometry does not match label (16h,63s != 255h,63s). GEOM: xbd0s2: geometry does not match label (16h,63s != 255h,63s). Trying to mount root from ufs:/dev/xbd0s1a WARNING: / was not properly dismounted rtc0: [XEN] xen_rtc_gettime rtc0: [XEN] xen_rtc_gettime: wallclock 1304589572 sec; 999999753 nsec rtc0: [XEN] xen_rtc_gettime: uptime 1062266 sec; 328577741 nsec rtc0: [XEN] xen_rtc_gettime: TOD 1305651839 sec; 328577494 nsec Configuring crash dumps... Mounting filesystems... Setting up embedded specific environment... done. WARNING: R/W mount of / denied. Filesystem is not clean - run fsck mount: /dev/xbd0s1a : Operation not permitted ** /dev/xbd0s1a ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes INCORRECT BLOCK COUNT I=44371 (80 should be 0) CORRECT? yes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=44371 OWNER=root MODE=100600 SIZE=0 MTIME=May 17 17:03 2011 CLEAR? yes LINK COUNT FILE I=44553 OWNER=root MODE=100600 SIZE=40960 MTIME=May 17 17:03 2011 COUNT 2 SHOULD BE 1 ADJUST? yes ** Phase 5 - Check Cyl groups BLK(S) MISSING IN BIT MAPS SALVAGE? yes 4732 files, 270780 used, 635875 free (379 frags, 79437 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** ** /dev/ufs/cf ** Last Mounted on /cf ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 20 files, 203 used, 100852 free (36 frags, 12602 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** ___ ___/ f \ / p \___/ Sense \___/ \ \___/ Welcome to pfSense 2.0-RC2 ... Creating symlinks......done. External config loader 1.0 is now starting... xbd0s3 appending output to nohup.out Launching the init system... done. Initializing............................ done. Starting device manager (devd)...done. Loading configuration......done. Updating configuration...done. Cleaning backup cache........done. Setting up extended sysctls...done. Setting timezone...done. Starting Secure Shell Services...done. Setting up polling defaults...done. Setting up interfaces microcode...done. Configuring LAGG interfaces...done. Configuring VLAN interfaces...done. Configuring QinQ interfaces...done. Configuring WAN interface...xn0: link state changed to DOWN xn0: link state changed to UP done. Syncing OpenVPN settings...done. Starting syslog...done. Configuring firewall......done. Starting PFLOG...done. Setting up gateway monitors...done. Synchronizing user settings...done. Starting webConfigurator...done. Configuring CRON...done. Starting OpenNTP time client...done. Starting DNS forwarder...done. Configuring firewall......done. Generating RRD graphs...done. Starting CRON... done. Executing rc.d items... Starting /usr/local/etc/rc.d/*.sh...done. Bootup complete FreeBSD/i386 (pfSense.localdomain) (console) *** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfSense *** WAN (wan) -> xn0 -> 10.82.121.21 (DHCP) 0) Logout (SSH only) 8) Shell 1) Assign Interfaces 9) pfTop 2) Set interface(s) IP address 10) Filter Logs 3) Reset webConfigurator password 11) Restart webConfigurator 4) Reset to factory defaults 12) pfSense Developer Shell 5) Reboot system 13) Upgrade from console 6) Halt system 14) Disable Secure Shell (sshd) 7) Ping host</virtual></virtual></software></xen></xen></xen></lahf></nx></sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>And some proof that is not on HVM
root@debian-msi:~# xm info host : debian-msi release : 2.6.32-5-xen-686 version : #1 SMP Wed Mar 9 01:27:54 UTC 2011 machine : i686 nr_cpus : 2 nr_nodes : 1 cores_per_socket : 1 threads_per_core : 2 cpu_mhz : 1600 hw_caps : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000 virt_caps : total_memory : 1013 free_memory : 4 node_to_cpu : node0:0-1 node_to_memory : node0:4 node_to_dma32_mem : node0:4 max_node_id : 0 xen_major : 4 xen_minor : 0 xen_extra : .1 xen_caps : xen-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : placeholder cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Wed Jan 12 15:04:50 UTC 2011 xend_config_format : 4 root@debian-msi:~# uname -a Linux debian-msi 2.6.32-5-xen-686 #1 SMP Wed Mar 9 01:27:54 UTC 2011 i686 GNU/Linux root@debian-msi:~# xm info host : debian-msi release : 2.6.32-5-xen-686 version : #1 SMP Wed Mar 9 01:27:54 UTC 2011 machine : i686 nr_cpus : 2 nr_nodes : 1 cores_per_socket : 1 threads_per_core : 2 cpu_mhz : 1600 hw_caps : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000 virt_caps : total_memory : 1013 free_memory : 4 node_to_cpu : node0:0-1 node_to_memory : node0:4 node_to_dma32_mem : node0:4 max_node_id : 0 xen_major : 4 xen_minor : 0 xen_extra : .1 xen_caps : xen-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : placeholder cc_compiler : gcc version 4.4.5 (Debian 4.4.5-10) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Wed Jan 12 15:04:50 UTC 2011 xend_config_format : 4 root@debian-msi:~# cat /tmp/pfsense.cfg kernel = '/root/pfsense/kernel-pfsense-xen-domU-debug' vif = ['mac=00:16:3e:45:18:2a, bridge=eth0,model='] auto_start_vm = 0 extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a' disk = ['phy:/dev/myVG/pfSense2,xvda,w'] vmname = 'pfsense' uuid = '6a819243-524f-d428-17f9-097bbd7f0abd' platform = 'xen' on_reboot = 'restart' on_shutdown = 'rename-restart' os_version = '11' memory = 256 ramdisk = '' bootloader = '' bootload = '/usr/bin/pygrub' name = 'pfsense' provision_timeout = 300 vcpus = 1 root = ''I can go the Interface without any issue. But sometimes I got a KP. I'll come back with KP details
-
Reply to my self,
After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.
It loads fine in my PV xen serveur, and seems to be stable.
I will stress it for a while and make some perfs tests.
*** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfsense *** WAN (wan) -> xn1 -> 10.42.37.25 (DHCP) LAN (lan) -> xn0 -> NONE 0) Logout (SSH only) 8) Shell 1) Assign Interfaces 9) pfTop 2) Set interface(s) IP address 10) Filter Logs 3) Reset webConfigurator password 11) Restart webConfigurator 4) Reset to factory defaults 12) pfSense Developer Shell 5) Reboot system 13) Upgrade from console 6) Halt system 14) Disable Secure Shell (sshd) 7) Ping host Enter an option: 8 [2.0-RC2][root@pfsense.localdomain]/root(1): uname -a FreeBSD pfsense.localdomain 8.2-STABLE FreeBSD 8.2-STABLE #3: Fri May 27 17:11:42 CEST 2011 user@freebsdhvm:/usr/obj/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386 i386 -
Reply to my self,
After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.
It loads fine in my PV xen serveur, and seems to be stable.
I will stress it for a while and make some perfs tests.
Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.
Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.
First, to clarify:
1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
2. This is NOT a PV kernel.
3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.
There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.
To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):
options NO_ADAPTIVE_MUTEXES
options NO_ADAPTIVE_RWLOCKS
options XENHVM
device xenpciI didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.
Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!
Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).
You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.
A sweet way of replacing the kernel after an install:
1. Inject your new kernel file into the official ISO file using mkisofs or similar
2. Reboot straight back into the CD after the installation and go into LiveCD mode
3. Enter a shell
4. Mount your installation slice and CD
5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.
Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.
-
I will stress it for a while and make some perfs tests. Thanks for sharing links here :)
–---------------------------
-
I made a specific topic on my hacks for PV domU based on RELENG_8_2. For the moment it's fine :D
http://forum.pfsense.org/index.php/topic,37693.0.html
-
Reply to my self,
After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.
It loads fine in my PV xen serveur, and seems to be stable.
I will stress it for a while and make some perfs tests.
Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.
Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.
First, to clarify:
1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
2. This is NOT a PV kernel.
3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.
There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.
To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):
options NO_ADAPTIVE_MUTEXES
options NO_ADAPTIVE_RWLOCKS
options XENHVM
device xenpciI didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.
Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!
Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).
You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.
A sweet way of replacing the kernel after an install:
1. Inject your new kernel file into the official ISO file using mkisofs or similar
2. Reboot straight back into the CD after the installation and go into LiveCD mode
3. Enter a shell
4. Mount your installation slice and CD
5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.
Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.
Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.
-
Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.
As far as the PV drivers are concerned, they're very stable (no issues so far); though I'm not using any traffic shaping or VLANs. Performance wise, its a give and take, though I can't recall the specific differences now. Maybe I'll run some comparisons on my next snapshot update.
Not sure about the PV kernel though, you'll have to ask nexus about that. But I'm pretty sure 64bits isn't possible for now. That doesn't mean you can't follow nexus' guide (linked two posts above) to paravirtualise 32bits pfSense though.
-
AFAIK FreeBSD supports only hardware virtualized (HVM) kernels on amd64; however, PV drivers are supported in this configuration.