PfSense 2.0 - Building custom kernel and image for appliance



  • Hi,

    I try to build pfsense for a specific usage (and testing purpose) :

    -Custom Kernel with PAE/XEN in order to make it works in pv mode (for non HVM machine and performances bench)
    -Custom Image to be installed as a domU

    I've followed the wiki (http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso) to make my own build env.

    1° Now I need to modify KERNCONF in /home/pfsense/tools/builder_scripts/conf but there are a lot of files.

    I want to build 2.0 release for FreeBS8.1 but I'm lost in conf file between FreeBSD.8 or pfsense.8 or pfsense_wrap.8 and sec.conf.8.

    How to apply my changes in one of this file ?

    2° How to craft a custom img or a strapped images (for example a ready to boot raw img), or building with XEN DOmU Option let me boot the ISO and install pfsense on my Dom0 machine (keeping the same kernel) ?

    Thanks for any help



  • Maybe found a way using extra_options and noextra_options in pfsense-build.conf

    and for target nanobsd could be ok I think for deployment



  • It gets me nuts ! Module_override always apply.

    From another way, what options and modules are mandatory for pfsense ?

    Can I build the kernel from /u/pfsense/src with make build kernel kernconf=xen-conf ?

    Thx



  • Ok, I make it works, but have some panic with network, needs to be debbuged. ;D ;D ;D

    Step :

    -Use the XEN KERNCONF from FreeBSD8.1 and add device and options from pfsense_wrap.8
    -Strip some VAR in script (MODULE_OVERRIDE…)
    -Build a Nanobsd img
    -Extract the kernel from image
    -Dump the image in a LV
    -Make it boot

    Some proof :

    xm console pfsense
    WARNING: loader(8) metadata is missing!
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1992-2010 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.1-RC1 #15: Tue May 17 13:30:17 CEST 2011
        nexus@freebsdhvm.inetpsa.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386 i386
    Xen reported: 1600.043 MHz processor.
    Timecounter "ixen" frequency 1000000000 Hz quality 0
    CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz (1600.04-MHz 686-class CPU)
      Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  Stepping = 2
      Features=0xbfe9fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40c39d <sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe>AMD Features=0x100000 <nx>AMD Features2=0x1 <lahf>TSC: P-state invariant
    real memory  = 268435456 (256 MB)
    avail memory = 254332928 (242 MB)
    [XEN] IPI cpu=0 irq=128 vector=RESCHEDULE_VECTOR (0)
    [XEN] IPI cpu=0 irq=129 vector=CALL_FUNCTION_VECTOR (1)
    xenbus0: <xen devices="">on motherboard
    [XEN] xen_rtc_probe: probing Hypervisor RTC clock
    rtc0: <xen hypervisor="" clock="">on motherboard
    [XEN] xen_rtc_attach: attaching Hypervisor RTC clock
    xc0: <xen console="">on motherboard
    padlock0: No ACE support.
    cryptosoft0: <software crypto="">on motherboard
    Timecounters tick every 10.000 msec
    [XEN] hypervisor wallclock nudged; nudging TOD.
    IPsec: Initialized Security Association Processing.
    xbd0: 10240MB <virtual block="" device="">at device/vbd/51712 on xenbus0
    xn0: <virtual network="" interface="">at device/vif/0 on xenbus0
    GEOM: xbd0s1: geometry does not match label (16h,63s != 255h,63s).
    GEOM: xbd0s2: geometry does not match label (16h,63s != 255h,63s).
    Trying to mount root from ufs:/dev/xbd0s1a
    WARNING: / was not properly dismounted
    rtc0: [XEN] xen_rtc_gettime
    rtc0: [XEN] xen_rtc_gettime: wallclock 1304589572 sec; 999999753 nsec
    rtc0: [XEN] xen_rtc_gettime: uptime 1062266 sec; 328577741 nsec
    rtc0: [XEN] xen_rtc_gettime: TOD 1305651839 sec; 328577494 nsec
    Configuring crash dumps...
    Mounting filesystems...
    Setting up embedded specific environment... done.
    WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
    mount: /dev/xbd0s1a : Operation not permitted
    ** /dev/xbd0s1a
    ** Last Mounted on /
    ** Root file system
    ** Phase 1 - Check Blocks and Sizes
    INCORRECT BLOCK COUNT I=44371 (80 should be 0)
    CORRECT? yes
    
    ** Phase 2 - Check Pathnames
    ** Phase 3 - Check Connectivity
    ** Phase 4 - Check Reference Counts
    UNREF FILE I=44371  OWNER=root MODE=100600
    SIZE=0 MTIME=May 17 17:03 2011
    CLEAR? yes
    
    LINK COUNT FILE I=44553  OWNER=root MODE=100600
    SIZE=40960 MTIME=May 17 17:03 2011  COUNT 2 SHOULD BE 1
    ADJUST? yes
    
    ** Phase 5 - Check Cyl groups
    BLK(S) MISSING IN BIT MAPS
    SALVAGE? yes
    
    4732 files, 270780 used, 635875 free (379 frags, 79437 blocks, 0.0% fragmentation)
    
    ***** FILE SYSTEM MARKED CLEAN *****
    
    ***** FILE SYSTEM WAS MODIFIED *****
    ** /dev/ufs/cf
    ** Last Mounted on /cf
    ** Phase 1 - Check Blocks and Sizes
    ** Phase 2 - Check Pathnames
    ** Phase 3 - Check Connectivity
    ** Phase 4 - Check Reference Counts
    ** Phase 5 - Check Cyl groups
    SUMMARY INFORMATION BAD
    SALVAGE? yes
    
    BLK(S) MISSING IN BIT MAPS
    SALVAGE? yes
    
    20 files, 203 used, 100852 free (36 frags, 12602 blocks, 0.0% fragmentation)
    
    ***** FILE SYSTEM MARKED CLEAN *****
    
    ***** FILE SYSTEM WAS MODIFIED *****
    
         ___
     ___/ f \
    / p \___/ Sense
    \___/   \
        \___/
    
    Welcome to pfSense 2.0-RC2  ...
    
    Creating symlinks......done.
    External config loader 1.0 is now starting... xbd0s3
    appending output to nohup.out
    Launching the init system... done.
    Initializing............................ done.
    Starting device manager (devd)...done.
    Loading configuration......done.
    Updating configuration...done.
    Cleaning backup cache........done.
    Setting up extended sysctls...done.
    Setting timezone...done.
    Starting Secure Shell Services...done.
    Setting up polling defaults...done.
    Setting up interfaces microcode...done.
    Configuring LAGG interfaces...done.
    Configuring VLAN interfaces...done.
    Configuring QinQ interfaces...done.
    Configuring WAN interface...xn0: link state changed to DOWN
    xn0: link state changed to UP
    done.
    Syncing OpenVPN settings...done.
    Starting syslog...done.
    Configuring firewall......done.
    Starting PFLOG...done.
    Setting up gateway monitors...done.
    Synchronizing user settings...done.
    Starting webConfigurator...done.
    Configuring CRON...done.
    Starting OpenNTP time client...done.
    Starting DNS forwarder...done.
    Configuring firewall......done.
    Generating RRD graphs...done.
    Starting CRON... done.
    Executing rc.d items...
     Starting /usr/local/etc/rc.d/*.sh...done.
    Bootup complete
    
    FreeBSD/i386 (pfSense.localdomain) (console)
    
    *** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfSense ***
    
      WAN (wan)                 -> xn0        -> 10.82.121.21 (DHCP)
    
     0) Logout (SSH only)                  8) Shell
     1) Assign Interfaces                  9) pfTop
     2) Set interface(s) IP address       10) Filter Logs
     3) Reset webConfigurator password    11) Restart webConfigurator
     4) Reset to factory defaults         12) pfSense Developer Shell
     5) Reboot system                     13) Upgrade from console
     6) Halt system                       14) Disable Secure Shell (sshd)
     7) Ping host</virtual></virtual></software></xen></xen></xen></lahf></nx></sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    

    And some proof that is not on HVM

    
    root@debian-msi:~# xm info
    host                   : debian-msi
    release                : 2.6.32-5-xen-686
    version                : #1 SMP Wed Mar 9 01:27:54 UTC 2011
    machine                : i686
    nr_cpus                : 2
    nr_nodes               : 1
    cores_per_socket       : 1
    threads_per_core       : 2
    cpu_mhz                : 1600
    hw_caps                : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000
    virt_caps              :
    total_memory           : 1013
    free_memory            : 4
    node_to_cpu            : node0:0-1
    node_to_memory         : node0:4
    node_to_dma32_mem      : node0:4
    max_node_id            : 0
    xen_major              : 4
    xen_minor              : 0
    xen_extra              : .1
    xen_caps               : xen-3.0-x86_32p
    xen_scheduler          : credit
    xen_pagesize           : 4096
    platform_params        : virt_start=0xf5800000
    xen_changeset          : unavailable
    xen_commandline        : placeholder
    cc_compiler            : gcc version 4.4.5 (Debian 4.4.5-10)
    cc_compile_by          : waldi
    cc_compile_domain      : debian.org
    cc_compile_date        : Wed Jan 12 15:04:50 UTC 2011
    xend_config_format     : 4
    root@debian-msi:~# uname -a
    Linux debian-msi 2.6.32-5-xen-686 #1 SMP Wed Mar 9 01:27:54 UTC 2011 i686 GNU/Linux
    root@debian-msi:~# xm info
    host                   : debian-msi
    release                : 2.6.32-5-xen-686
    version                : #1 SMP Wed Mar 9 01:27:54 UTC 2011
    machine                : i686
    nr_cpus                : 2
    nr_nodes               : 1
    cores_per_socket       : 1
    threads_per_core       : 2
    cpu_mhz                : 1600
    hw_caps                : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000
    virt_caps              :
    total_memory           : 1013
    free_memory            : 4
    node_to_cpu            : node0:0-1
    node_to_memory         : node0:4
    node_to_dma32_mem      : node0:4
    max_node_id            : 0
    xen_major              : 4
    xen_minor              : 0
    xen_extra              : .1
    xen_caps               : xen-3.0-x86_32p
    xen_scheduler          : credit
    xen_pagesize           : 4096
    platform_params        : virt_start=0xf5800000
    xen_changeset          : unavailable
    xen_commandline        : placeholder
    cc_compiler            : gcc version 4.4.5 (Debian 4.4.5-10)
    cc_compile_by          : waldi
    cc_compile_domain      : debian.org
    cc_compile_date        : Wed Jan 12 15:04:50 UTC 2011
    xend_config_format     : 4
    
    root@debian-msi:~# cat /tmp/pfsense.cfg
    kernel = '/root/pfsense/kernel-pfsense-xen-domU-debug'
    vif = ['mac=00:16:3e:45:18:2a, bridge=eth0,model=']
    auto_start_vm = 0
    extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
    disk = ['phy:/dev/myVG/pfSense2,xvda,w']
    vmname = 'pfsense'
    uuid = '6a819243-524f-d428-17f9-097bbd7f0abd'
    platform = 'xen'
    on_reboot = 'restart'
    on_shutdown = 'rename-restart'
    os_version = '11'
    memory = 256
    ramdisk = ''
    bootloader = ''
    bootload = '/usr/bin/pygrub'
    name = 'pfsense'
    provision_timeout = 300
    vcpus = 1
    root = ''
    
    

    I can go the Interface without any issue. But sometimes I got a KP. I'll come back with KP details



  • Reply to my self,

    After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

    It loads fine in my PV xen serveur, and seems to be stable.

    I will stress it for a while and make some perfs tests.

    *** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfsense ***
    
      WAN (wan)                 -> xn1        -> 10.42.37.25 (DHCP)
      LAN (lan)                 -> xn0        -> NONE
    
     0) Logout (SSH only)                  8) Shell
     1) Assign Interfaces                  9) pfTop
     2) Set interface(s) IP address       10) Filter Logs
     3) Reset webConfigurator password    11) Restart webConfigurator
     4) Reset to factory defaults         12) pfSense Developer Shell
     5) Reboot system                     13) Upgrade from console
     6) Halt system                       14) Disable Secure Shell (sshd)
     7) Ping host
    
    Enter an option: 8
    
    [2.0-RC2][root@pfsense.localdomain]/root(1): uname -a
    FreeBSD pfsense.localdomain 8.2-STABLE FreeBSD 8.2-STABLE #3: Fri May 27 17:11:42 CEST 2011     user@freebsdhvm:/usr/obj/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386  i386
    
    


  • @nexus:

    Reply to my self,

    After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

    It loads fine in my PV xen serveur, and seems to be stable.

    I will stress it for a while and make some perfs tests.

    Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.

    Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.

    First, to clarify:
    1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
    2. This is NOT a PV kernel.
    3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
    4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.

    Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.

    There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.

    To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):

    options         NO_ADAPTIVE_MUTEXES
    options         NO_ADAPTIVE_RWLOCKS
    options         XENHVM
    device          xenpci

    I didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.

    Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!

    Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
    http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056

    Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).

    You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.

    A sweet way of replacing the kernel after an install:
    1. Inject your new kernel file into the official ISO file using mkisofs or similar
    2. Reboot straight back into the CD after the installation and go into LiveCD mode
    3. Enter a shell
    4. Mount your installation slice and CD
    5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/

    That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.

    Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.



  • I will stress it for a while and make some perfs tests. Thanks for sharing links here :)

    –---------------------------

    Tin tuc
    Tin Giao duc



  • I made a specific topic on my hacks for PV domU based on RELENG_8_2. For the moment it's fine :D

    http://forum.pfsense.org/index.php/topic,37693.0.html



  • @xieliwei:

    @nexus:

    Reply to my self,

    After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

    It loads fine in my PV xen serveur, and seems to be stable.

    I will stress it for a while and make some perfs tests.

    Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.

    Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.

    First, to clarify:
    1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
    2. This is NOT a PV kernel.
    3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
    4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.

    Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.

    There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.

    To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):

    options         NO_ADAPTIVE_MUTEXES
    options         NO_ADAPTIVE_RWLOCKS
    options         XENHVM
    device          xenpci

    I didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.

    Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!

    Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
    http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056

    Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).

    You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.

    A sweet way of replacing the kernel after an install:
    1. Inject your new kernel file into the official ISO file using mkisofs or similar
    2. Reboot straight back into the CD after the installation and go into LiveCD mode
    3. Enter a shell
    4. Mount your installation slice and CD
    5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/

    That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.

    Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.

    Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.



  • @cyruspy:

    Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.

    As far as the PV drivers are concerned, they're very stable (no issues so far); though I'm not using any traffic shaping or VLANs. Performance wise, its a give and take, though I can't recall the specific differences now. Maybe I'll run some comparisons on my next snapshot update.

    Not sure about the PV kernel though, you'll have to ask nexus about that. But I'm pretty sure 64bits isn't possible for now. That doesn't mean you can't follow nexus' guide (linked two posts above) to paravirtualise 32bits pfSense though.



  • AFAIK FreeBSD supports only hardware virtualized (HVM) kernels on amd64; however, PV drivers are supported in this configuration.


Locked