Build 2.1-DEVELOPMENT + RELENG_8_3



  • Hi,

    I followed the steps provided here http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso

    via menu.sh, I choose the version, applied the patches and asked to build a image. After some hours I've got some errors.

    debug and configuration are here http://pastie.org/7462038#

    there is something that I'm missing?

    best regards,

    Silva.

    UPDATE: I tried, via menu.sh to compile just the NanoBSD, and that is what is faling. Any idea?

    UPDATE2: after 8 hours trying to compile, fix the environment the error, looks better:

    I ran ./build_nano.sh

    http://pastie.org/7465906

    the tail of the error looks like that:

    Running plugins: customroot customscripts pkginstall buildmodules Done!
    Using /usr/home/pfsense/tools/builder_scripts/../builder_scripts/copy.list.RELENG_2_1…
    Populating newer binaries found on host jail/os (usr/local)...
    cp: /usr/local/pfsense-fs/usr/local/libdata/pkgconfig/libcrypto.pc: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/libdata/pkgconfig/libssl.pc: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/libdata/pkgconfig/openssl.pc: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/CA.pl: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/CA.sh: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/c_hash: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/c_info: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/c_issuer: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/c_name: No such file or directory
    cp: /usr/local/pfsense-fs/usr/local/openssl/misc/tsget: No such file or directory

    Installing collected library information, please wait...
    Deleting files listed in /usr/home/pfsense/tools/builder_scripts/remove.list.iso.8
    Fixing up NanoBSD Specific items...
    Creating md5 summary of files present...Done.
    Copying config.xml from conf.default/ to cf/conf/
    Testing PHP installation in /usr/local/pfsense-fs:
    Fatal error: Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 169

    An error occured while testing the php installation in /usr/local/pfsense-fs

    any idea?



  • Encountering a similar problem (see attached tail of the compilation log) when executed ./build_nano.sh:

    Two errors are obvious from the attached log:

    1.

    Clog is missing on staging area copying from host
    install: /usr/sbin/clog: No such file or directory

    For this, I installed clog to the host machine using 'pkg_add -r clog', but clog gets installed to /usr/local/sbin/clog instead of the desired /usr/sbin/clog:

    which clog

    /usr/local/sbin/clog

    hmmm..

    2.

    Copying config.xml from conf.default/ to cf/conf/
    Testing PHP installation in /usr/local/pfsense-fs:chroot: /test_php.php: No such file or directory

    An error occured while testing the php installation in /usr/local/pfsense-fs

    ####################################
    Something went wrong, check errors!
    ####################################

    NOTE: a lot of times you can run ./clean_build.sh to resolve.

    Do not know what to do for this? Any pointers? I have tried several times with ./clean_build.sh and recompiled again for the last two days and no matter what I do it ends at the same point of compilation.

    BTW, my pfsense-build.conf looks like:

    export PFSENSE_VERSION=2.1-RC0
    export FREEBSD_VERSION=8
    export FREEBSD_BRANCH=RELENG_8_3
    export PFSENSETAG=RELENG_2_1
    export
    PFSPATCHFILE=/home/pfsense/tools/builder_scripts/conf/patchlist/patches.RELENG_8_3
    export PFSPATCHDIR=/home/pfsense/tools/patches/RELENG_8_3
    export
    SUPFILE=/home/pfsense/tools/builder_scripts/conf/csup/RELENG_8_3-supfile
    export
    CUSTOM_COPY_LIST=/home/pfsense/tools/builder_scripts/conf/copylist/copy.list.RELENG_2_1
    export OVERRIDE_FREEBSD_CVSUP_HOST=cvsup14.freebsd.org
    export BASE_DIR=/home/pfsense/
    export BUILDER_TOOLS=/home/pfsense/tools/
    export BUILDER_SCRIPTS=/home/pfsense/tools/builder_scripts
    export EXTRA_DEVICES=siba_bwn,bwn,run,epair,if_bridge
    export EXTRA_OPTIONS=VIMAGE,NULLFS
    export NOEXTRA_OPTIONS=SCTP
    export FREESBIE_ERROR_MAIL=admin@email.tld
    export PFSPORTSFILE=buildports.RELENG_2_1

    Thanks!

    pfsense-compilation-errors.txt


  • Rebel Alliance Developer Netgate

    Never use the ports version or pkg_add. Anything you need is in ports or pfPorts and must be built by the build system.

    # ls -l /usr/sbin/clog
    -r-xr-xr-x  1 root  wheel   8616 Jun 30 15:12 /usr/sbin/clog
    # cd /home/pfsense/tools/builder_scripts; ./build_pfPorts.sh clog
    # ls -l /usr/sbin/clog
    -r-xr-xr-x  1 root  wheel   8616 Aug  5 15:28 /usr/sbin/clog
    

    Looks like the pfPorts build process is building clog and installing it where it should.

    Your BASE_DIR and BUILDER_TOOLS vars have trailing slashes when they shouldn't, but that shouldn't be fatal.

    As for test_php.php:

    builder_scripts/builder_common.sh:      cp $BUILDER_SCRIPTS/scripts/test_php.php $PFSENSEBASEDIR/
    $ find . -name test_php.php
    ./builder_scripts/scripts/test_php.php
    

    As long as the vars are set right it should pick it up…



  • @jimp:

    Never use the ports version or pkg_add. Anything you need is in ports or pfPorts and must be built by the build system.

    # ls -l /usr/sbin/clog
    -r-xr-xr-x  1 root  wheel   8616 Jun 30 15:12 /usr/sbin/clog
    # cd /home/pfsense/tools/builder_scripts; ./build_pfPorts.sh clog
    # ls -l /usr/sbin/clog
    -r-xr-xr-x  1 root  wheel   8616 Aug  5 15:28 /usr/sbin/clog
    

    Looks like the pfPorts build process is building clog and installing it where it should.

    Thanks Jim for this useful pointer. Gradually learning the pfSense way of doing things. Your replies are very useful and helpful.

    @jimp:

    Your BASE_DIR and BUILDER_TOOLS vars have trailing slashes when they shouldn't, but that shouldn't be fatal.

    Removed trailing slashes and compiled, but … :-(

    @jimp:

    As for test_php.php:

    builder_scripts/builder_common.sh:      cp $BUILDER_SCRIPTS/scripts/test_php.php $PFSENSEBASEDIR/
    $ find . -name test_php.php
    ./builder_scripts/scripts/test_php.php
    

    As long as the vars are set right it should pick it up…

    Reset all build environment and then rebuilt after copying the test_php.php to the pfSense base directory (which is /home/pfsense because the build_common.sh already has a line 1087 associated with 'cp $BUILDER_SCRIPTS/scripts/test_php.php $PFSENSEBASEDIR/', yet the compiler spits out the same error. Attaching pruned log file (in txt format) for perusal.

    My builder_common.sh looks like:

    grep -Rn test_php.php builder_common.sh

    1087: cp $BUILDER_SCRIPTS/scripts/test_php.php $PFSENSEBASEDIR/
    1088: chmod a+rx $PFSENSEBASEDIR/test_php.php
    1089: HOSTNAME=env SHELL=/bin/sh chroot $PFSENSEBASEDIR /test_php.php
    1136: rm $PFSENSEBASEDIR/test_php.php

    test_php.php is already there:

    find . -name test_php.php

    ./builder_scripts/scripts/test_php.php

    Actually I need only NanoBSD images. Thanks again for your cooperation!

    kernel.pfSense_wrap.8.amd64.txt



  • Hi:

    This seems to be an unending compiling process with the same test_php.php chroot error:

    Testing PHP installation in /usr/local/pfsense-fs:chroot: /test_php.php: No such file or directory

    An error occured while testing the php installation in /usr/local/pfsense-fs

    I have tried to compile NanoBSD image (./build_nano.sh) for more than 50 times with several changes. The last thing I did was replacing $PFSENSEBASEDIR to $PFSENSEBASEDIR/ in line 1089 of builder_common.sh (1089: HOSTNAME=env SHELL=/bin/sh chroot $PFSENSEBASEDIR/ /test_php.php), didn't work. Changed from /test_php.php to ./test_php.php, no go. :-(

    My build environemnt is:

    uname -a

    FreeBSD nano.dev.lan 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Mon Apr  9 21:23:18 UTC 2012    root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

    Is there no way that one could compile pfSense images with some kernel tweaks? Or is this a bug? Thanks!


  • Rebel Alliance Developer Netgate

    Did you ever try to build an ISO?

    I just made a completely fresh builder from scratch today (and updated the instructions a little along the way) and it built OK and passed that PHP test with no problems.

    It's building NanoBSD now but I won't know until morning if it completed OK.



  • @jimp:

    Did you ever try to build an ISO?

    I just made a completely fresh builder from scratch today (and updated the instructions a little along the way) and it built OK and passed that PHP test with no problems.

    It's building NanoBSD now but I won't know until morning if it completed OK.

    Thansk, Jim for confirmation.

    I could build after some additional steps*** as mentioned below which is not stated in http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso. Kudos to idwer in the IRC channel.

    However, after booting the iso, I kept on getting errors writing configurations:

    1. On bootup:

    Fatal error: Call to undefined function pfSense_interface_listget() in /etc/inc/interfaces.inc on line 66

    2. Therefore, I just used 'dhclient <interface_name>' to get an IP and went well

    3. When I tried to 'Enable Secure Shell (sshd)' by selecting 14 in the console menu:

    Writing configurations…
    Fatal error: Call to undefined function pfSense_sync() in /etc/inc/config.lib.inc on line 468

    Thus, the image is built but seems something went wrong, but could not figure out what?

    ***This is for those who encounter the problems while building an image.

    Use all command line tools to have everything in your command, not menu.sh.

    1. since cvsup is deprecated in FreeBSD to pull sources, use svn:

    cd /usr/ports/devel/subversion && make install clean
    svn co svn://svn0.eu.FreeBSD.org//base/release/8.3.0/ /usr/src/

    1. then the first thing to do after pulling the git repos is:

    cd /home/pfsense/tools/builder_scripts/ && ./build_pfPorts.sh

    It takes adequate time.

    1. Execute:

    ./build_iso.sh

    even though you only need to build NanoBSD image.

    1. Then build NanoBSD image:

    ./build_nano.sh

    And if you want to create a specific sized NanoBSD image, execute:

    ./build_nano_resized.sh

    where <imgsize>could be 512mb, 1g, 2g or 4g.</imgsize></interface_name>


  • Rebel Alliance Developer Netgate

    My nano build worked fine when it finished.

    The pfSense_*() functions missing mean your pfSense PHP module did not build.

    You don't have to manually build pfPorts, that will happen automatically when doing build_iso.sh

    You don't need to use subversion yet, cvsup still works fine for RELENG_8_3 (it doesn't work for the most recent RELENG_9_* and won't work at all for RELENG_10* but older tags are still there)

    And doing build_iso.sh is always best to do first, I will need to remember to add a note for that



  • @jimp:

    My nano build worked fine when it finished.

    The pfSense_*() functions missing mean your pfSense PHP module did not build.

    In that case, do I rebuild the image again? Or there is a way to rebuilde pfSense PHP module separately?

    @jimp:

    You don't have to manually build pfPorts, that will happen automatically when doing build_iso.sh

    That is what missing in the documentation that first iso needs to be built before any other images.

    @jimp:

    You don't need to use subversion yet, cvsup still works fine for RELENG_8_3 (it doesn't work for the most recent RELENG_9_* and won't work at all for RELENG_10* but older tags are still there)

    Yes, I tried just to follow latest FreeBSD documenation.

    @jimp:

    And doing build_iso.sh is always best to do first, I will need to remember to add a note for that

    Yes, that would save hours and hours of whining like it happened to me.


  • Rebel Alliance Developer Netgate

    To fix the module, you can either rebuild all pfPorts, or just the module:

    ./build_pfPorts.sh php53-pfSense-module
    

    If I were you, I'd re-run the full pfPorts build again and make sure none of them fail.

    The documentation actually already stated to run build_iso.sh it just wasn't forceful about it. Apparently you also missed point #8 on the page

    8. Other images available for building
    Once you have a working ISO you can build pretty much any pfSense image listed in the menu system.

    "Once you have a working ISO…" implies you must first build an ISO before you can move on to other image types. I added some more clarification above in the step that actually says to build the ISO first.



  • @jimp:

    To fix the module, you can either rebuild all pfPorts, or just the module:

    ./build_pfPorts.sh php53-pfSense-module
    

    If I were you, I'd re-run the full pfPorts build again and make sure none of them fail.

    trying to build php53-pfSense-module gave me the following error:

    Building php53-pfSense-module(php53-pfSense-module-0.1)…Failed to build. Error log in /tmp/pfPort/buildlogs/php53-pfSense-module.
    Ports with failures:        2
    Building php53-pfSense-module(php53-pfSense-module-0.1)...ERROR!
    File not found php53-pfSense-module /usr/ports/devel/php53-pfSense-module /usr/local/lib/php/20090626/pfSense.so - /usr/local/lib/php/20090626/pfSense.so

    Building athstats...main.c: In function 'getfmt':
    main.c:73: warning: implicit declaration of function 'strcasecmp'
    Done!

    Building tools/crytpo...Done!
    Operation ./build_pfPorts.sh has ended at Fri Aug  9 17:48:20 CEST 2013

    @jimp:

    The documentation actually already stated to run build_iso.sh it just wasn't forceful about it. Apparently you also missed point #8 on the page

    8. Other images available for building
    Once you have a working ISO you can build pretty much any pfSense image listed in the menu system.

    "Once you have a working ISO…" implies you must first build an ISO before you can move on to other image types. I added some more clarification above in the step that actually says to build the ISO first.

    Yep, you are right, but it is partially my oversight and partially lack of explicit explanation.


  • Rebel Alliance Developer Netgate

    For the module, it lists the location of a log in the output. What is in that log?



  • Rebuilt pfPorts completely. Went without errors:

    Rebuilt iso image without a problem.

    But when I booted the LiveCD, the php5-related errors were gone, instead got kernel panic as seen in the attachment. :-(




  • I rebuilt everything (hostOS, git repo, src and ports), rebuilt two times once with #MAKE_JOBS_UNSAFE=YES in make.conf and without, but the booting halts at the same point like as shown in the previous screenshot.

    Is there any secret (that is yet to be divulged) to build a pfSense image with some modificatios in kernel? The problem is really whining me for more than a week!



  • Now what I did was:

    1. reinstalled FreeBSD-8.3
      2)```
      freebsd-update fetch install
    3)```
    echo "WITHOUT_X11=yo" >> /etc/make.conf
    echo "OPTIONS_UNSET=X11" >> /etc/make.conf
    echo "BATCH=yo" >> /etc/make.conf
    

    4)```
    mkdir -p /home/pfsense/pfSenseGITREPO /usr/pfSensesrc

    5)```
    portsnap fetch extract
    

    6)```
    cd /usr/ports/textproc/expat2 && make depends install
    cd /usr/ports/devel/git && make depends install
    cd /usr/ports/sysutils/fastest_cvsup/ && make depends install

    7)```
    rehash
    

    8)```
    cd /home/pfsense && git clone git://github.com/pfsense/pfsense-tools.git tools
    cd /home/pfsense && git clone git://github.com/pfsense/freesbie2.git freesbie2
    cd /home/pfsense/tools/builder_scripts && chmod a+rx *.sh

    9)```
    csup -h `fastest_cvsup -c tld -q` /usr/share/examples/cvsup/standard-supfile
    

    10)```
    ./build_pfPorts.sh

    It completes with a following line which has been reported also in this thread (http://forum.pfsense.org/index.php/topic,47878.15.html) which states that it is not critical so went to 12) below:
    
    > >>> Building svnup(svnup-1.0)…Done.
    > >>> Ports with failures:        0
    > >>> Building athstats...main.c: In function 'getfmt':
    > main.c:73: warning: implicit declaration of function 'strcasecmp'
    > Done!
    > >>> Building tools/crytpo…Done!
    > ==> End of pfPorts...
    > >>> Operation ./build_pfPorts.sh has ended at Sat Aug 10 15:05:27 CEST 2013
    
    12)```
    ./apply_kernel_patches.sh
    

    13)```
    ./build_iso.sh

    
    After hours, the iso gets built, but when booted it gives the db> prompt as shown in the screenshot attahced with 'Fatal trap 12: page fault while in kernel mode' (reported also in Freebsd Forum http://forums.freebsd.org/showthread.php?t=16575, but does not seem it applies to pfSense).
    
    Where in the world did I do wrong?  :o  Is there any secret recipe to build pfSense?  ???
    
    ![pfsense_2013-08-10--1376144883.png](/public/_imported_attachments_/1/pfsense_2013-08-10–1376144883.png)
    ![pfsense_2013-08-10--1376144883.png_thumb](/public/_imported_attachments_/1/pfsense_2013-08-10--1376144883.png_thumb)

  • Rebel Alliance Developer Netgate

    Can you try in a different virtual environment, such as Virtualbox, or on real hardware?

    There really isn't any secret to it, what you've done is the same as how we build images and they work fine from here.

    The error you're seeing is in the kernel and not from ports/packages.



  • @jimp:

    Can you try in a different virtual environment, such as Virtualbox, or on real hardware?

    There really isn't any secret to it, what you've done is the same as how we build images and they work fine from here.

    The error you're seeing is in the kernel and not from ports/packages.

    I tried first in the real hardware which I reported earlier, and now using a qemu virtual instance. I find virtualbox more buggy than qemu.

    I am just lost.  :o



  • @jimp:

    Can you try in a different virtual environment, such as Virtualbox, or on real hardware?

    There really isn't any secret to it, what you've done is the same as how we build images and they work fine from here.

    The error you're seeing is in the kernel and not from ports/packages.

    I tried to build it again in Virtualbox, and after many hours of compiling, the created iso finally landed into the same kernel panic (see attached screenshot) at boottime.:-(

    BTW, can you append two EXTRA_DEVICES (epair, if_bridge), two EXTRA_OPTIONS (VIMAGE,NULLFS) and an EXTRA_NOOPTIONS (SCTP) to your pfsense-build.conf and confirm whether it gets built in your machine? Thanks in advance.



  • Rebel Alliance Developer Netgate

    Try a completely stock build first. Only when you know that works should you try to customize it.

    It is most likely one of your extra options causing the panic. There is a reason we don't include some things, either because the code isn't compatible or because they don't work the way they should with our other changes.



  • @jimp:

    Try a completely stock build first. Only when you know that works should you try to customize it.

    It is most likely one of your extra options causing the panic. There is a reason we don't include some things, either because the code isn't compatible or because they don't work the way they should with our other changes.

    Is there a 'known issues' regarding the kernel modules that are not compatible, and therefore, not included with the pfSense 2.1? It would be nice to know rather than frustratingly trying to compile image without success.

    Thanks!


  • Rebel Alliance Developer Netgate

    We don't have a list that I'm aware of, unless there are other posts on the forum or mailing lists mentioning it. If they were truly beneficial and compatible, they're most likely already included though or at least available as modules.

    Beyond that, you're really on your own. Monkeying with any network-related options outside of drivers is bound to give you grief. Odds are they will not interact well with our patches/gui/etc.



  • [SOLVED] Seems like it is upstream problem. I tried once again after some 12 hours and [CMD]freebsd-update fetch[/CMD] worked this time.

    I tried to build once again with a fresh install. This time it stopped at a very first stage.

    While trying to update the OS with [CMD]freebsd-update fetch[/CMD] I kept on getting an error that reads:

    ...gunzip: (stdin): unexpected end of file
    metadata is corrupt
    

    To resolve, I removed [FILE]/var/db/freebsd-update/files[/FILE] and executed above command again, but it spits out the same error. hmmm… any clue?




  • UPDATE: Running './build_pfPorts.sh b43-fwcutter' solved the problem. However this ./build_iso.sh failed to build iso (just memstick images were built) and the memstick image has the same kernel panic as stated in my above posts. :-( )

    Now, I am getting the following error during ./build_pfPorts.sh

    File not found b43-fwcutter /usr/ports/sysutils/b43-fwcutter /usr/local/bin/b43-fwcutter - /usr/local/bin/b43-fwcutter
    Building athstats…main.c: In function 'getfmt':
    main.c:73: warning: implicit declaration of function 'strcasecmp'
    Done!

    Building tools/crytpo...Done!

    I checked with whether b43-fwcutter is available and it is there:

    dev# ls -lah /usr/ports/sysutils/b43-fwcutter/
    total 32
    drwxr-xr-x    3 root  wheel  512B Aug 22 19:01 .
    drwxr-xr-x  1078 root  wheel    21k Aug 22 19:04 ..
    -rw-r–r--    1 root  wheel  428B Aug 22 19:01 Makefile
    -rw-r--r--    1 root  wheel  141B Aug 22 19:01 distinfo
    drwxr-xr-x    2 root  wheel  512B Aug 22 19:01 files
    -rw-r--r--    1 root  wheel  113B Aug 22 19:01 pkg-descr

    Did anyone encouter something like this?


Log in to reply