Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense 2.0 - Building custom kernel and image for appliance

    Scheduled Pinned Locked Moved Development
    11 Posts 4 Posters 18.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nexus
      last edited by

      Hi,

      I try to build pfsense for a specific usage (and testing purpose) :

      -Custom Kernel with PAE/XEN in order to make it works in pv mode (for non HVM machine and performances bench)
      -Custom Image to be installed as a domU

      I've followed the wiki (http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso) to make my own build env.

      1° Now I need to modify KERNCONF in /home/pfsense/tools/builder_scripts/conf but there are a lot of files.

      I want to build 2.0 release for FreeBS8.1 but I'm lost in conf file between FreeBSD.8 or pfsense.8 or pfsense_wrap.8 and sec.conf.8.

      How to apply my changes in one of this file ?

      2° How to craft a custom img or a strapped images (for example a ready to boot raw img), or building with XEN DOmU Option let me boot the ISO and install pfsense on my Dom0 machine (keeping the same kernel) ?

      Thanks for any help

      1 Reply Last reply Reply Quote 0
      • N
        nexus
        last edited by

        Maybe found a way using extra_options and noextra_options in pfsense-build.conf

        and for target nanobsd could be ok I think for deployment

        1 Reply Last reply Reply Quote 0
        • N
          nexus
          last edited by

          It gets me nuts ! Module_override always apply.

          From another way, what options and modules are mandatory for pfsense ?

          Can I build the kernel from /u/pfsense/src with make build kernel kernconf=xen-conf ?

          Thx

          1 Reply Last reply Reply Quote 0
          • N
            nexus
            last edited by

            Ok, I make it works, but have some panic with network, needs to be debbuged. ;D ;D ;D

            Step :

            -Use the XEN KERNCONF from FreeBSD8.1 and add device and options from pfsense_wrap.8
            -Strip some VAR in script (MODULE_OVERRIDE…)
            -Build a Nanobsd img
            -Extract the kernel from image
            -Dump the image in a LV
            -Make it boot

            Some proof :

            xm console pfsense
            WARNING: loader(8) metadata is missing!
            KDB: debugger backends: ddb
            KDB: current backend: ddb
            Copyright (c) 1992-2010 The FreeBSD Project.
            Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
                    The Regents of the University of California. All rights reserved.
            FreeBSD is a registered trademark of The FreeBSD Foundation.
            FreeBSD 8.1-RC1 #15: Tue May 17 13:30:17 CEST 2011
                nexus@freebsdhvm.inetpsa.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386 i386
            Xen reported: 1600.043 MHz processor.
            Timecounter "ixen" frequency 1000000000 Hz quality 0
            CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz (1600.04-MHz 686-class CPU)
              Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  Stepping = 2
              Features=0xbfe9fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40c39d <sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe>AMD Features=0x100000 <nx>AMD Features2=0x1 <lahf>TSC: P-state invariant
            real memory  = 268435456 (256 MB)
            avail memory = 254332928 (242 MB)
            [XEN] IPI cpu=0 irq=128 vector=RESCHEDULE_VECTOR (0)
            [XEN] IPI cpu=0 irq=129 vector=CALL_FUNCTION_VECTOR (1)
            xenbus0: <xen devices="">on motherboard
            [XEN] xen_rtc_probe: probing Hypervisor RTC clock
            rtc0: <xen hypervisor="" clock="">on motherboard
            [XEN] xen_rtc_attach: attaching Hypervisor RTC clock
            xc0: <xen console="">on motherboard
            padlock0: No ACE support.
            cryptosoft0: <software crypto="">on motherboard
            Timecounters tick every 10.000 msec
            [XEN] hypervisor wallclock nudged; nudging TOD.
            IPsec: Initialized Security Association Processing.
            xbd0: 10240MB <virtual block="" device="">at device/vbd/51712 on xenbus0
            xn0: <virtual network="" interface="">at device/vif/0 on xenbus0
            GEOM: xbd0s1: geometry does not match label (16h,63s != 255h,63s).
            GEOM: xbd0s2: geometry does not match label (16h,63s != 255h,63s).
            Trying to mount root from ufs:/dev/xbd0s1a
            WARNING: / was not properly dismounted
            rtc0: [XEN] xen_rtc_gettime
            rtc0: [XEN] xen_rtc_gettime: wallclock 1304589572 sec; 999999753 nsec
            rtc0: [XEN] xen_rtc_gettime: uptime 1062266 sec; 328577741 nsec
            rtc0: [XEN] xen_rtc_gettime: TOD 1305651839 sec; 328577494 nsec
            Configuring crash dumps...
            Mounting filesystems...
            Setting up embedded specific environment... done.
            WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
            mount: /dev/xbd0s1a : Operation not permitted
            ** /dev/xbd0s1a
            ** Last Mounted on /
            ** Root file system
            ** Phase 1 - Check Blocks and Sizes
            INCORRECT BLOCK COUNT I=44371 (80 should be 0)
            CORRECT? yes
            
            ** Phase 2 - Check Pathnames
            ** Phase 3 - Check Connectivity
            ** Phase 4 - Check Reference Counts
            UNREF FILE I=44371  OWNER=root MODE=100600
            SIZE=0 MTIME=May 17 17:03 2011
            CLEAR? yes
            
            LINK COUNT FILE I=44553  OWNER=root MODE=100600
            SIZE=40960 MTIME=May 17 17:03 2011  COUNT 2 SHOULD BE 1
            ADJUST? yes
            
            ** Phase 5 - Check Cyl groups
            BLK(S) MISSING IN BIT MAPS
            SALVAGE? yes
            
            4732 files, 270780 used, 635875 free (379 frags, 79437 blocks, 0.0% fragmentation)
            
            ***** FILE SYSTEM MARKED CLEAN *****
            
            ***** FILE SYSTEM WAS MODIFIED *****
            ** /dev/ufs/cf
            ** Last Mounted on /cf
            ** Phase 1 - Check Blocks and Sizes
            ** Phase 2 - Check Pathnames
            ** Phase 3 - Check Connectivity
            ** Phase 4 - Check Reference Counts
            ** Phase 5 - Check Cyl groups
            SUMMARY INFORMATION BAD
            SALVAGE? yes
            
            BLK(S) MISSING IN BIT MAPS
            SALVAGE? yes
            
            20 files, 203 used, 100852 free (36 frags, 12602 blocks, 0.0% fragmentation)
            
            ***** FILE SYSTEM MARKED CLEAN *****
            
            ***** FILE SYSTEM WAS MODIFIED *****
            
                 ___
             ___/ f \
            / p \___/ Sense
            \___/   \
                \___/
            
            Welcome to pfSense 2.0-RC2  ...
            
            Creating symlinks......done.
            External config loader 1.0 is now starting... xbd0s3
            appending output to nohup.out
            Launching the init system... done.
            Initializing............................ done.
            Starting device manager (devd)...done.
            Loading configuration......done.
            Updating configuration...done.
            Cleaning backup cache........done.
            Setting up extended sysctls...done.
            Setting timezone...done.
            Starting Secure Shell Services...done.
            Setting up polling defaults...done.
            Setting up interfaces microcode...done.
            Configuring LAGG interfaces...done.
            Configuring VLAN interfaces...done.
            Configuring QinQ interfaces...done.
            Configuring WAN interface...xn0: link state changed to DOWN
            xn0: link state changed to UP
            done.
            Syncing OpenVPN settings...done.
            Starting syslog...done.
            Configuring firewall......done.
            Starting PFLOG...done.
            Setting up gateway monitors...done.
            Synchronizing user settings...done.
            Starting webConfigurator...done.
            Configuring CRON...done.
            Starting OpenNTP time client...done.
            Starting DNS forwarder...done.
            Configuring firewall......done.
            Generating RRD graphs...done.
            Starting CRON... done.
            Executing rc.d items...
             Starting /usr/local/etc/rc.d/*.sh...done.
            Bootup complete
            
            FreeBSD/i386 (pfSense.localdomain) (console)
            
            *** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfSense ***
            
              WAN (wan)                 -> xn0        -> 10.82.121.21 (DHCP)
            
             0) Logout (SSH only)                  8) Shell
             1) Assign Interfaces                  9) pfTop
             2) Set interface(s) IP address       10) Filter Logs
             3) Reset webConfigurator password    11) Restart webConfigurator
             4) Reset to factory defaults         12) pfSense Developer Shell
             5) Reboot system                     13) Upgrade from console
             6) Halt system                       14) Disable Secure Shell (sshd)
             7) Ping host</virtual></virtual></software></xen></xen></xen></lahf></nx></sse3,dtes64,mon,ds_cpl,est,tm2,ssse3,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
            

            And some proof that is not on HVM

            
            root@debian-msi:~# xm info
            host                   : debian-msi
            release                : 2.6.32-5-xen-686
            version                : #1 SMP Wed Mar 9 01:27:54 UTC 2011
            machine                : i686
            nr_cpus                : 2
            nr_nodes               : 1
            cores_per_socket       : 1
            threads_per_core       : 2
            cpu_mhz                : 1600
            hw_caps                : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000
            virt_caps              :
            total_memory           : 1013
            free_memory            : 4
            node_to_cpu            : node0:0-1
            node_to_memory         : node0:4
            node_to_dma32_mem      : node0:4
            max_node_id            : 0
            xen_major              : 4
            xen_minor              : 0
            xen_extra              : .1
            xen_caps               : xen-3.0-x86_32p
            xen_scheduler          : credit
            xen_pagesize           : 4096
            platform_params        : virt_start=0xf5800000
            xen_changeset          : unavailable
            xen_commandline        : placeholder
            cc_compiler            : gcc version 4.4.5 (Debian 4.4.5-10)
            cc_compile_by          : waldi
            cc_compile_domain      : debian.org
            cc_compile_date        : Wed Jan 12 15:04:50 UTC 2011
            xend_config_format     : 4
            root@debian-msi:~# uname -a
            Linux debian-msi 2.6.32-5-xen-686 #1 SMP Wed Mar 9 01:27:54 UTC 2011 i686 GNU/Linux
            root@debian-msi:~# xm info
            host                   : debian-msi
            release                : 2.6.32-5-xen-686
            version                : #1 SMP Wed Mar 9 01:27:54 UTC 2011
            machine                : i686
            nr_cpus                : 2
            nr_nodes               : 1
            cores_per_socket       : 1
            threads_per_core       : 2
            cpu_mhz                : 1600
            hw_caps                : bfe9fbff:00100000:00000000:00000940:0040c39d:00000000:00000001:00000000
            virt_caps              :
            total_memory           : 1013
            free_memory            : 4
            node_to_cpu            : node0:0-1
            node_to_memory         : node0:4
            node_to_dma32_mem      : node0:4
            max_node_id            : 0
            xen_major              : 4
            xen_minor              : 0
            xen_extra              : .1
            xen_caps               : xen-3.0-x86_32p
            xen_scheduler          : credit
            xen_pagesize           : 4096
            platform_params        : virt_start=0xf5800000
            xen_changeset          : unavailable
            xen_commandline        : placeholder
            cc_compiler            : gcc version 4.4.5 (Debian 4.4.5-10)
            cc_compile_by          : waldi
            cc_compile_domain      : debian.org
            cc_compile_date        : Wed Jan 12 15:04:50 UTC 2011
            xend_config_format     : 4
            
            root@debian-msi:~# cat /tmp/pfsense.cfg
            kernel = '/root/pfsense/kernel-pfsense-xen-domU-debug'
            vif = ['mac=00:16:3e:45:18:2a, bridge=eth0,model=']
            auto_start_vm = 0
            extra = 'vfs.root.mountfrom=ufs:/dev/xbd0s1a'
            disk = ['phy:/dev/myVG/pfSense2,xvda,w']
            vmname = 'pfsense'
            uuid = '6a819243-524f-d428-17f9-097bbd7f0abd'
            platform = 'xen'
            on_reboot = 'restart'
            on_shutdown = 'rename-restart'
            os_version = '11'
            memory = 256
            ramdisk = ''
            bootloader = ''
            bootload = '/usr/bin/pygrub'
            name = 'pfsense'
            provision_timeout = 300
            vcpus = 1
            root = ''
            
            

            I can go the Interface without any issue. But sometimes I got a KP. I'll come back with KP details

            1 Reply Last reply Reply Quote 0
            • N
              nexus
              last edited by

              Reply to my self,

              After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

              It loads fine in my PV xen serveur, and seems to be stable.

              I will stress it for a while and make some perfs tests.

              *** Welcome to pfSense 2.0-RC2-nanobsd (i386) on pfsense ***
              
                WAN (wan)                 -> xn1        -> 10.42.37.25 (DHCP)
                LAN (lan)                 -> xn0        -> NONE
              
               0) Logout (SSH only)                  8) Shell
               1) Assign Interfaces                  9) pfTop
               2) Set interface(s) IP address       10) Filter Logs
               3) Reset webConfigurator password    11) Restart webConfigurator
               4) Reset to factory defaults         12) pfSense Developer Shell
               5) Reboot system                     13) Upgrade from console
               6) Halt system                       14) Disable Secure Shell (sshd)
               7) Ping host
              
              Enter an option: 8
              
              [2.0-RC2][root@pfsense.localdomain]/root(1): uname -a
              FreeBSD pfsense.localdomain 8.2-STABLE FreeBSD 8.2-STABLE #3: Fri May 27 17:11:42 CEST 2011     user@freebsdhvm:/usr/obj/usr/pfSensesrc/src/sys/pfSense_wrap.8.i386  i386
              
              
              1 Reply Last reply Reply Quote 0
              • X
                xieliwei
                last edited by

                @nexus:

                Reply to my self,

                After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

                It loads fine in my PV xen serveur, and seems to be stable.

                I will stress it for a while and make some perfs tests.

                Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.

                Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.

                First, to clarify:
                1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
                2. This is NOT a PV kernel.
                3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
                4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.

                Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.

                There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.

                To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):

                options         NO_ADAPTIVE_MUTEXES
                options         NO_ADAPTIVE_RWLOCKS
                options         XENHVM
                device          xenpci

                I didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.

                Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!

                Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
                http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056

                Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).

                You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.

                A sweet way of replacing the kernel after an install:
                1. Inject your new kernel file into the official ISO file using mkisofs or similar
                2. Reboot straight back into the CD after the installation and go into LiveCD mode
                3. Enter a shell
                4. Mount your installation slice and CD
                5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/

                That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.

                Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.

                1 Reply Last reply Reply Quote 0
                • D
                  demsaoroi88
                  last edited by

                  I will stress it for a while and make some perfs tests. Thanks for sharing links here :)

                  –---------------------------

                  Tin tuc
                  Tin Giao duc

                  1 Reply Last reply Reply Quote 0
                  • N
                    nexus
                    last edited by

                    I made a specific topic on my hacks for PV domU based on RELENG_8_2. For the moment it's fine :D

                    http://forum.pfsense.org/index.php/topic,37693.0.html

                    1 Reply Last reply Reply Quote 0
                    • C
                      cyruspy
                      last edited by

                      @xieliwei:

                      @nexus:

                      Reply to my self,

                      After porting pfsenses patches from RELENG_8_1 to RELENG_8_2 (some mistake in patches) ! I've build a kernel based on RELENG_8_2.

                      It loads fine in my PV xen serveur, and seems to be stable.

                      I will stress it for a while and make some perfs tests.

                      Would you be able to detail the steps required to get a working PV kernel? I'm just starting on this and it'll be great if you can save everyone the trouble of going through the problems you encountered while you were working on this.

                      Update: Okay, got it to work based on nexus's posts. Its quite obvious after going through the wiki page nexus pointed out.

                      First, to clarify:
                      1. I'm compiling a new kernel for a HVM install. The new kernel will contain the xen PV drivers.
                      2. This is NOT a PV kernel.
                      3. AFAIK a PV kernel is not available for amd64 until at least RELEND_9_0, which pfSense is not compatible with yet.
                      4. A PV kernel can be compiled for i386, the instructions are similar, but you will have to modify the kernel config differently yourself.

                      Couldn't get RELEND_8_2 to compile though (or rather, the build system is too fragile for my patience and hacking skills). But it does seem that the current patch set for RELEND_8_2 is outdated, most of the patches are already applied in one way or another.

                      There seems to be a bug in the builder scripts with regards to building for amd64. It seems that the shell variable TARGET_ARCH is not defined, and defaults to i386. This causes the kernel compile to use the i386 version of the GENERIC config instead of the amd64 one. Not sure if there's any harm or is intentional, but appending "export TARGET_ARCH amd64" to pfsense-build.conf fixes that.

                      To get xen drivers compiled into the kernel, append the following into the kernel config file (Located in /home/pfsense/tools/builder/scripts/conf/ as pfSense.* or pfSense_.) relevant to the kernel you're going to use (or append to all if you're not sure):

                      options         NO_ADAPTIVE_MUTEXES
                      options         NO_ADAPTIVE_RWLOCKS
                      options         XENHVM
                      device          xenpci

                      I didn't check if the MODULES_OVERRIDE will affect the build, but I did find out that setting ALL_MODULES will override the override, so append "export ALL_MODULES" to pfsense-build.conf as well.

                      Just in case, do a "grep -r MODULES_OVERRIDE" in the /home/pfsense directory and remove anything that seems to apply the module overriding to the compile. NOTE: Don't just remove every instance returned by grep!

                      Also, apply the following patch so that virtual network interfaces don't give you "xennet_get_responses: too many frags 6 > max 5":
                      http://svn.freebsd.org/viewvc/base/head/sys/dev/xen/netfront/netfront.c?view=patch&r1=218056&r2=218055&pathrev=218056

                      Try to build the ISO, you may have to execute ./build_iso.sh a few times to get it to complete without errors. Also, the build system is quite fragile and affects your FreeBSD installation, so its best to start with a fresh FreeBSD install each time you make a mistake (yeah, that'll up your stakes for making mistakes! ;) ).

                      You'll get an ISO which you can install with, but I don't trust the quality of my own build, so I extracted the kernel from /tmp/kernel/[type]/boot/kernel/kernel.gz , installed from a recent official snapshot and replaced the installed kernel with my own.

                      A sweet way of replacing the kernel after an install:
                      1. Inject your new kernel file into the official ISO file using mkisofs or similar
                      2. Reboot straight back into the CD after the installation and go into LiveCD mode
                      3. Enter a shell
                      4. Mount your installation slice and CD
                      5. Copy your new kernel file from the CD into [Where your installation is mounted]/boot/kernel/

                      That way, your installation would never boot from the stock kernel and boots from your new kernel on its first fun. Much cleaner that way.

                      Alternatively, instead of injecting the kernel file into the ISO, you can have it uploaded on a remote FTP server and retrieve it from the LiveCD after manually setting up the network using ifconfig.

                      Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.

                      1 Reply Last reply Reply Quote 0
                      • X
                        xieliwei
                        last edited by

                        @cyruspy:

                        Any update on this?, i would like to run 64bits pfsense PV. Given it's not possible I would like to know how your HVM + PV drivers install is doing.

                        As far as the PV drivers are concerned, they're very stable (no issues so far); though I'm not using any traffic shaping or VLANs. Performance wise, its a give and take, though I can't recall the specific differences now. Maybe I'll run some comparisons on my next snapshot update.

                        Not sure about the PV kernel though, you'll have to ask nexus about that. But I'm pretty sure 64bits isn't possible for now. That doesn't mean you can't follow nexus' guide (linked two posts above) to paravirtualise 32bits pfSense though.

                        1 Reply Last reply Reply Quote 0
                        • N
                          nexus
                          last edited by

                          AFAIK FreeBSD supports only hardware virtualized (HVM) kernels on amd64; however, PV drivers are supported in this configuration.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.