Building pfSense - Something basic I'm doing wrong?



  • Ok, got signed up for the tools repo.

    Yesterday, I built a new FreeBSD 8.3 VM, and installed git from ports.  I was able to pull the tools repo down to /usr/local/pfsense-tools.

    Using the menu, I set the pfSense TAG to RELENG_2_1_2 and FreeBSD Branch to releng/8.3.

    When I try to build nanoBSD, I saw that there was an issue with the version of git that I had.  These scripts require a version that supports the "single-branch" option.  This morning, I've managed to get git 1.9.0 installed.  I no longer see the error about the single-branch option, but things still aren't building properly.  This is the output I get:

    >>> Using /usr/local/pfsense-tools/builder_scripts/conf/rmlist/remove.list.iso.8 ...
    >>> Operation ./build_nano.sh has started at Wed Apr 23 13:25:22 EDT 2014
    >>> Obtaining FreeBSD sources releng/8.3...Done!
    >>> Removing needed files listed in patches RELENG_2_1_2
    >>> Applying patches from /usr/local/pfsense-tools/builder_scripts/../builder_scripts/conf/patchlist/releng/8.3/patches please wait...No file to patch.  Skipping...
    1 out of 1 hunks ignored--saving rejects to sys/contrib/altq/altq/altq_cbq.h.rej
    No file to patch.  Skipping...
    1 out of 1 hunks ignored--saving rejects to sys/contrib/altq/altq/altq_hfsc.h.rej
    failed to apply altq_max_queues.diff
    No file to patch.  Skipping...
    11 out of 11 hunks ignored--saving rejects to parse.y.rej
    No file to patch.  Skipping...
    14 out of 14 hunks ignored--saving rejects to pfctl_altq.c.rej
    No file to patch.  Skipping...
    
    

    And it just goes on like that for pages…  No file to patch.

    The scripts did successfully pull down /usr/pfSensesrc, so that part looks good.

    I feel like I've done something very basic wrong, but I'm not seeing it.  Was /usr/local/pfsense-tools a bad place to put my copy of the tools repo?  Did I do not install something that is required with my FreeBSD install?

    If there's a walk-through somewhere, just point me that way, but I think this method is probably so new that there isn't anything like that just yet...

    Any help would be greatly appreciated.

    Thanks,
    Paul



  • Yeah you should use tools folder or tell the builder where the new folder is.



  • I get error too.
    My apply patches is not working. Is failing on obtain sources from freebsd. Maybe is not authenticating in git server ?. csup is gone for 8.3…



  • I blew away my /usr/pfsense-tools/ folder and ran this command:

    git clone ssh://git@git.pfsense.org/git/pfsense-tools.git tools

    That cloned it into /usr/tools/

    Using menu, I set version to RELENG_2_1_2 / releng/8.3.

    When I try to build, I still get all the "No file to patch" messages.

    So, the name of the directory didn't seem to matter.



  • Hello,

    I'm facing to a similar issue (impossible to buid pfSense 2.1.X anymore), I may be wrong but I suspect the problem comes from the following commit :

    Point repos URLs to internal git
    rbgarga authored 19 days ago
    b7f866d7340ae723841b40e2c8ccbd5ab18fa9a1

    -export FREEBSD_REPO_BASE=${FREEBSD_REPO_BASE:-"https://github.com/freebsd/freebsd.git"}
    +export FREEBSD_REPO_BASE=${FREEBSD_REPO_BASE:-"git@git.pfmechanics.com:outsidemirrors/freebsd.git"}
    etc, etc …

    Git refuses to connect to that host (git.pfmechanics.com) despite the fact my SSH access is working fine (git.pfsense.org), it looks like a private host being only reachable from ESF employees (which has been configured everywhere for git repos).

    In order to see the commits, I had to push the new pfsense-tools repo to my own github account (private repo of course) ... is it another easier way to see the commits as before ?

    Thanks in advance for any help,

    Best regards,

    Cyriles



  • I was just looking for info on git.pfmechanics.com trying to further troubleshoot this, as I ran across this error message:

    >>> Updating BSDInstaller collection...Cloning into 'installer'...
    ssh: connect to host git.pfmechanics.com port 22: No route to host
    fatal: Could not read from remote repository.
    
    

    It looks like several things reference the pfmechanics.com git repo…



  • I think I've changed all the references to the internal git repo…  I've attached a zip file to this email (named .zip.txt, just rename it when you download it and unzip).  It's structured by directory, so just unzip it into your copy of the tools repo.  Two files in builder_scripts/ will be replaced, as will two more files under builder_scripts/scripts/

    I can't say 100% whether this works yet, but my build sure seems to be getting a lot further.

    pfsense-tools.changes.zip.txt



  • What worked for me is telling the tools to use subversion for the checkout instead of git:



  • I know hand editing the scripts isn't the most elegant solution, but since ESF just re-opened this late last week, they probably have more work to do to get everything working for people just starting to build their own.

    I know they've had dev VM images before, but I'm sure whatever they come up with will be good (even if it's just a walk through on how to build your own system).



  • There is no need to edit scripts people, think smart.
    Just use pfsense-build.conf created by set_version.sh to override things.



  • I only see a single item to customize in the conf file:

    export FREEBSD_REPO_BASE=https://github.com/freebsd/freebsd.git
    
    

    Perhaps there are other items you can add?  I had to change multiple REPOs in the files I uploaded.

    So far, my builds have not completed 100% successfully either…

    Part of the issue is the varying skill levels of everyone involved.  There are some rare individuals that are talented with FreeBSD, networking, and programming.  There are many more that are probably even better at individual aspects I mentioned, but have trouble with others.

    In my case, I do a lot of PHP programming that's CLI based, mostly for managing (auditing, deploying, etc) the configuration a few thousand switches and almost 900 routers and UTMs.  Back about 6 years ago, I contributed several features to the Monowall project, the most visible one was administrative groups that allow users to have granular access to the GUI, which was brought over to pfSense soon after.

    In my opinion, the lower the barrier to entry to be able to contribute to pfSense, the better it is for the project.

    I feel certain that ESF isn't done with their plans for the development community.  I just wish they would share more information with us about what they are planning and give some sort of guidance as to when to expect their plans to come to fruition.

    At any rate, I'm going to try to wait for ESF before trying much more to build my own image.

    Edited:  Just noticed that I now have "-1 Karma".  Thanks, whoever apparently doesn't like what I have said.



  • #export FREEBSD_REPO_BASE=git@git.pfmechanics.com:outsidemirrors/freebsd.git
    export USE_SVN=YES
    export FREEBSD_REPO_BASE=https://svn0.us-west.freebsd.org/base
    export GIT_REPO_PFSENSE=${GIT_REPO_PFSENSE:-"git://github.com/pfsense/pfsense.git"}



  • Well setup has awlays been easy.

    Checkout latest sources and run build.sh –configure it is a GUI that has been always there.



  • Well, it's still impossible to build anything with the "pfsense-tools" repo which is available through the SSH access (2.1.X).

    I even tried to backport the ports (through SVN) and pfsense-tools commits from the 10/04/2014 (official 2.1.2 release date) but with even more build ports errors (23 in stand of 4 while trying to build with up to date ports).

    I'm pretty sure the pfsense-tools repo which is available through the SSH access is not up to date (no commits since the famous day it has been made available again) … it looks like just a clone made to shut up "annoying" people asking for the sources ... I think the simple fact the git repos are linking to a private and local EFS host can only mean there was no initial plans to make it public in the future, am I wrong ? If so then what was the point doing this (commit b7f866d7340ae723841b40e2c8ccbd5ab18fa9a1 for example) ?

    The only thing which could make sense, is that the new pfsense-tools repo can only be used to build 2.2 releases (not yet tried but I bet I'll face to similar build errors).

    Let's hope things will soon start moving in the right direction because right now we're all wasting our time, open source is no more meaning anything here (I have the feeling all is done to stop people building by themselves).

    PS1: I don't like felating dogs or anything else (just in case ... you'll surely understand why I say that; if not then just give a look to the recent threads from the "development" forum section).
    PS2: Even if you don't have any reasons to trust me, I'm not making any money with pfSense BUT I need to customise my personal installations as most people (in order to save precious time) ... is it now such a crime ?



  • Well the tools repo has never been for the general people anyway.
    Its trying to get better.

    The internal repos committed there are because of speed on having local repos rather than going to github.
    Please stop this 'drama' and ask real questions rather than accusations that do not go nowhere and bring nothing of value.



  • @ermal :

    It's not a personal complain, I respect and appreciate your work.

    The problem is that I (and I'm sure I'm not the only one) can't buid my own customised pfSense release anymore.

    If I well understood the problem, it's a matter of money (ESF) … I'm ready to subscribe to what is necessary but I think the minimum is to communicate, and here there are no communications at all (the injuries are all but a kind of communication, providing an useless pfsense-repo doesn't help anyone).

    If nothing changes, then the only solution for me (and most people) will be to move to another real Open Source solution ... is it the point ?

    If so then just tell it and the "non founded accusations" will be over (sorry for my English, I'm French and learnt English by myself).

    The Real question is : If you were me (without dept programming skills), what would you do ? Is it fair providing an Open Source firewall solution which can't be customized ?

    Please take the time to think about what I wrote (the human way, not the capitalist way), else I won't waste more time answering.

    Best regards,

    Cyriles



  • @cyrilles: perhaps you should just ask how to use the repo (and resolve your errors). Instead you let your imagination run wild, with cospiracy theories.

    i'm sure someone will one day write up a nice how-to-build-your-trademark-infrigment-free-software-based-on-pf-source-wiki

    until then, i propose people to ask answereable questions ;
    in other words, show errors, screenshots, useful info | then perhaps someone who uses the tools daily, can provide an answer.

    but thats just my 2 cents



  • @cyriles:

    @ermal :

    It's not a personal complain, I respect and appreciate your work.

    The problem is that I (and I'm sure I'm not the only one) can't buid my own customised pfSense release anymore.

    If I well understood the problem, it's a matter of money (ESF) … I'm ready to subscribe to what is necessary but I think the minimum is to communicate, and here there are no communications at all (the injuries are all but a kind of communication, providing an useless pfsense-repo doesn't help anyone).

    If nothing changes, then the only solution for me (and most people) will be to move to another real Open Source solution ... is it the point ?

    If so then just tell it and the "non founded accusations" will be over (sorry for my English, I'm French and learnt English by myself).

    The Real question is : If you were me (without dept programming skills), what would you do ? Is it fair providing an Open Source firewall solution which can't be customized ?

    Please take the time to think about what I wrote (the human way, not the capitalist way), else I won't waste more time answering.

    Best regards,

    Cyriles

    I do not take this personal.
    Its just you people complain about things not being open source and than complain that they not work.
    If you want open source to fix it yourself have fun its there.

    If you want help on thing stop accusing, imagining things and get real.
    What you post is you need help and nothing in that is a direct question to what is wrong, you just complain about 'nonclear' things in your mind.

    In human way, there have been times when the tools repo has been in very bad shape and no one complained, now that it is in way better position to be understood by general people complains come for policies and not the repository itself.

    By the way, I will not answer anything that is not a real question!



  • It looks like git.pfmechanics.com is either down can not handle the traffic it is getting. You can override the pfsense repo URL by setting GIT_REPO_PFSENSE to git@github.com:pfsense/pfsense.git but unfortunately the URL for the BSD installer repo is hard coded to git@git.pfmechanics.com:pfsense/bsdinstaller.git.



  • @kpa:

    It looks like git.pfmechanics.com is either down can not handle the traffic it is getting. You can override the pfsense repo URL by setting GIT_REPO_PFSENSE to git@github.com:pfsense/pfsense.git but unfortunately the URL for the BSD installer repo is hard coded to git@git.pfmechanics.com:pfsense/bsdinstaller.git.

    I'm not seeing any load issues, and it's been up for 4.5 days:

    jim@git:~ % uptime
    11:12AM  up 4 days, 10:58, 2 users, load averages: 0.39, 0.40, 0.22
    jim@git:~ % date
    Tue Apr 29 11:12:27 CDT 2014



  • It is not reachable for me. It looks like there's only an IPv6 address for it in DNS, is that right?

    
    drill git.pfmechanics.com ANY
    ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 36656
    ;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 3 
    ;; QUESTION SECTION:
    ;; git.pfmechanics.com. IN      ANY
    
    ;; ANSWER SECTION:
    git.pfmechanics.com.    3470    IN      AAAA    2610:160:11:10::20
    
    ;; AUTHORITY SECTION:
    pfmechanics.com.        3470    IN      NS      ns1.pfmechanics.com.
    pfmechanics.com.        3470    IN      NS      ns2.pfmechanics.com.
    ...
    
    


  • @ermal:

    @cyriles:

    @ermal :

    It's not a personal complain, I respect and appreciate your work.

    The problem is that I (and I'm sure I'm not the only one) can't buid my own customised pfSense release anymore.

    If I well understood the problem, it's a matter of money (ESF) … I'm ready to subscribe to what is necessary but I think the minimum is to communicate, and here there are no communications at all (the injuries are all but a kind of communication, providing an useless pfsense-repo doesn't help anyone).

    If nothing changes, then the only solution for me (and most people) will be to move to another real Open Source solution ... is it the point ?

    If so then just tell it and the "non founded accusations" will be over (sorry for my English, I'm French and learnt English by myself).

    The Real question is : If you were me (without dept programming skills), what would you do ? Is it fair providing an Open Source firewall solution which can't be customized ?

    Please take the time to think about what I wrote (the human way, not the capitalist way), else I won't waste more time answering.

    Best regards,

    Cyriles

    I do not take this personal.
    Its just you people complain about things not being open source and than complain that they not work.
    If you want open source to fix it yourself have fun its there.

    If you want help on thing stop accusing, imagining things and get real.
    What you post is you need help and nothing in that is a direct question to what is wrong, you just complain about 'nonclear' things in your mind.

    In human way, there have been times when the tools repo has been in very bad shape and no one complained, now that it is in way better position to be understood by general people complains come for policies and not the repository itself.

    By the way, I will not answer anything that is not a real question!

    Points:

    • building a customized release of pfSense software (and then distributing it) is only allowed if you do NOT call the result pfSense, and you must otherwise fully comply with the licenses of pfSense.  This, in part, requires you to state on all marketing materials that your product is derived from pfSense software

    • if you need help building pfSense, commercial support is available

    • the pfSense repos are not "useless".  We use the same repos that you're attempting to use.

    • Ermal has invested a lot of work recently in the '-tools' repo to cut the build time from over 4 hours to under 30 minutes (on our infrastructure, YMMV).



  • @kpa:

    It is not reachable for me. It looks like there's only an IPv6 address for it in DNS, is that right?

    
    drill git.pfmechanics.com ANY
    ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 36656
    ;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 5, ADDITIONAL: 3 
    ;; QUESTION SECTION:
    ;; git.pfmechanics.com. IN      ANY
    
    ;; ANSWER SECTION:
    git.pfmechanics.com.    3470    IN      AAAA    2610:160:11:10::20
    
    ;; AUTHORITY SECTION:
    pfmechanics.com.        3470    IN      NS      ns1.pfmechanics.com.
    pfmechanics.com.        3470    IN      NS      ns2.pfmechanics.com.
    ...
    
    

    Maybe.  It's difficult for me to tell from here.

    Posting to the forum isn't the ideal way to inform.  In this case, I'll forward it on internally, but a more direct communication about these will serve you well.
    (We do not constantly monitor the forums.)



  • you're using the wrong host

    git.pfsense.org has address 208.123.73.74

    https://forum.pfsense.org/index.php?topic=76132.0



  • Just FYI, I've added the repos to "git.pfsense.org".  The machine "git.pfmechanics.com" is not publicly accessible.

    You should be able to clone from the following repo's:
    git@git.pfsense.org:bsdinstaller.git
    git@git.pfsense.org:pfsense-tools.git
    git@git.pfsense.org:xmlrpc-server.git
    git@git.pfsense.org:pfsense-packages.git

    You will need to still setup a FreeBSD site to pull from.

    If there are other bugs in the scripts, best way is to create a bug report in Redmine, and we'll get it addressed.  We're working on ways to speed up builds as gonzopancho noted, part if this is locally mirroring commonly used sources.



  • @kpa:

    It looks like git.pfmechanics.com is either down can not handle the traffic it is getting. You can override the pfsense repo URL by setting GIT_REPO_PFSENSE to git@github.com:pfsense/pfsense.git but unfortunately the URL for the BSD installer repo is hard coded to git@git.pfmechanics.com:pfsense/bsdinstaller.git.

    Fixed, can be overwritten now.



  • Thanks, I'll give it a go soon.



  • Hello,

    First of all, my apologies if I did wrong assumptions and/or used bad words (I remind you English is not my native language), however the latest posts from that thread are slowly but surely confirming it :

    It is still impossible to build anything with the current pfsense-tools repo (FreeBSD 8.3 and 10.0 -> with the most recent ports, that's obvious).

    Facts :

    1. I'm using pfSense for my own personal usage so why are you still speaking about subscription and copyright ? If it's the only thing which matters for you then let's stop discussing as it's just a waste of time, can't you really guess there are honest people not making profit of pfSense ?! Is it really impossible to get usefull discussions without being subcribed ? (questions have been asked, there are questions for who is open minded).

    2. that repo has just been updated today (so I was right while saying it was just a non-updated cloned repo as we can see commits made every days since it has been made public again). Moreover it is now confirmed by developpers that the hardcoded repos are not reachable from non-ESF members (which confirms what I said in my previous posts).

    3. the fact you're able to compile with your own machine doens't mean anything, it just means your developpers correctly played with the ports revisions but who could guess which revision has to be downloaded for each port ? That's just impossible … the only solution would be to make public a VMWare image (ready for the builds) or something similar ...

    4. you think I'm an idiot, fine ... then just try to install a clean and virgin FreeBSD (8.3 or 10.0, the issue being the same) ... then fetch the most recent ports (I even tried with older ones, just in case the developpers worked on older ones), then clone the pfsense-tools repo you recently made available again and try to build the pfPorts ... as you'll clearly notice there are several important ports which can't be compiled ... I fixed more than half of these failures by doing manual backports (most of the times caused by the NO_STAGE setting), however some can't easily be fixed because it needs PHP 5.5 (php-suhosin for example).

    Any developper can easily guess the time I wasted trying to compile and fix things again and again ... that's the main reason of that post.

    5. I could continue like that for hours but that's not my point and that would be disrespectfull for the developpers who make a great job and have no links at all with these recent issues.

    I'm not here to make war, I'll better choose to stop posting than to continue like that (you make me feel like the bad guy, I'm all but that guy) ... my only point is to be able to compile my own pfSense release (no commercial reasons, I repeat it as I'm sure it will be said again) without having to ask for help (you should respect that, at least I always try to do things by myself without bothering people … it's very rare I make such posts).

    I'm sure in other circumstances we would enjoy having a beer together, I'm really sad seeing how things are going as I love the pfSense project (it could become THE reference in the next few years if things are done correctly).

    Let's stop this, I'm pretty sure it will generate "hatefull" answers but I had to post my thoughts.

    Best regards,

    Cyriles



    • The only discussion around subscription is for services in pfSense Gold.  Copyright still holds, but we allow anyone to download the images (thereby making a copy) and run them on their machine.  What we don't allow is that these images are changed (in any way) and then further redistributed.

    • i am unsure of the point you're trying to make here.

    • i am unsure of the point you're trying to make here.

    • I don't think you're an idiot, nor have I asserted that you are an idiot.  Where did this come from?  (Yes, we are moving to PHP 5.5 in pfSense 2.2.)

    • i am unsure of the point you're trying to make here.

    In terms of being able to compile your own version of pfSense, the rules are as I stated.  You can compile your own, and we want to make sure that is possible.  What we can not allow is for you to compile your own and then distribute it with our marks intact.  I've posted the reasons for this elsewhere.

    Thank you for helping find the issues with the repos.  I think people here were very responsive once the issue was identified.



  • I got the RELENG_2_2 build working to a point where it can fetch the repositories and apply patches but it failed later on building one of the pfPorts (I think that where it failed but lack of proper status messages left me unsure). Expect a better report and bug reports when I have more time to dig deeper. The build system definitely needs some improvements to be usable for outsiders, it's quite difficult to understand even for me who happens to have quite a bit of experience with FreeBSD's build systems and building everything from source in general.



  • Ok I'm back again trying to build RELENG_2_2 for i386. The build started fine but errors in sysutils/squashfs-tools port. The log shows this:

    
    cc -O2 -pipe -fno-strict-aliasing -std=gnu89  -I. -I/usr/local/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DFNM_EXTMATCH
    =0 -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT  i386 -c -o mksquashfs.o mksquashfs.c
    cc -O2 -pipe -fno-strict-aliasing -std=gnu89  -I. -I/usr/local/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DFNM_EXTMATCH
    =0 -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT  i386 -c -o read_fs.o read_fs.c
    cc: cc -O2 -pipe -fno-strict-aliasing -std=gnu89  -I. -I/usr/local/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DFNM_EXTM
    ATCH=0 -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT  i386 -c -o sort.o sort.c
    error: no such file or directory: 'i386'
    cc: error: no such file or directory: 'i386'
    cc -O2 -pipe -fno-strict-aliasing -std=gnu89  -I. -I/usr/local/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -DFNM_EXTMATCH
    =0 -D_GNU_SOURCE -DCOMP_DEFAULT=\"gzip\" -Wall -DGZIP_SUPPORT -DXZ_SUPPORT -DLZO_SUPPORT  i386 -c -o swap.o swap.c
    gmake[2]: *** [mksquashfs.o] Error 1
    gmake[2]: *** Waiting for unfinished jobs....
    gmake[2]: *** [read_fs.o] Error 1
    cc: error: no such file or directory: 'i386'
    cc: error: no such file or directory: 'i386'
    gmake[2]: *** [sort.o] Error 1
    gmake[2]: *** [swap.o] Error 1
    gmake[2]: Leaving directory `/usr/ports/sysutils/squashfs-tools/work/squashfs4.2/squashfs-tools'
    ===> Compilation failed unexpectedly.
    Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
    the maintainer.
    *** Error code 1
    
    Stop.
    make[1]: stopped in /usr/ports/sysutils/squashfs-tools
    *** Error code 1
    
    Stop.
    make: stopped in /usr/ports/sysutils/pcbsd-utils
    Done!
    
    

    The build continued though after the error with the buildworld part so I didn't stop it yet.

    I'm using the most upto date ports tree from the official FreeBSD SVN repository, do I need to use a specific version of the ports tree?

    My build host is:

    
    FreeBSD freebsd10.rdnzl.info 10.0-STABLE FreeBSD 10.0-STABLE #2 r266005: Wed May 14 16:42:13 EEST 2014     kimmo@freebsd10.rdnzl.info:/usr/obj/usr/src/sys/VT  i386
    
    

    (The VT kernel is just GENERIC kernel with the newcons console included so ignore that)



  • I was able to work around the error in my previous post by installing sysutils/pcbsd-utils manually before firing up the build.sh again. However this time the buildworld stopped with an error:

    
    --- sbin.depend__D ---
    /usr/pfSensesrc/src/sbin/pfctl/pfctl_qstats.c:41:10: fatal error: 'altq/altq_fairq.h' file not found
    #include <altq altq_fairq.h="">^
    1 error generated.</altq> 
    

    This is bad, very bad. The buildworld that is done with /usr/pfSensesrc sources is not finding its own include files (altq/altq_fairq.h is a pfSense addition and not present on vanilla FreeBSD 10). I am definitely not going to do a manual copy of the missing includes to /usr/include on the build host because that kind of manual step shouldn't be needed when the whole intention of the scripts is to automate the build process.

    Here's a suggestion to all of you who are working on the tools repo. Every once in a while do a complete wipe of the build host and reinstall it from scratch using the latest FreeBSD stable/10 snapshot. Re-fetch all the repositories and redo all the configurations. This would reveal any problems that are caused by extra files that exist on the build host and are not present in a clean state when an outsider tries to repeat the build process starting from scratch.

    Edit: Redmine ticket opened, https://redmine.pfsense.org/issues/3668.



  • Replied on redmine.



  • Fair enough. I'll summarize quickly what I have learned so far:

    There is no documentation available for using the tools repo so one has trough trial and error and asking around gather the necessary information on how to use it. In my case the missing piece of information was that pfPorts needs to built first before anything else. Building of pfPorts is not done automatically if you just fire up the build script as```
    build.sh iso

    
    You need an ultra-clean FreeBSD installation on the build host because the tools want to build everything its way and anything non-standard in terms of configuration trips it very easily.
    
    Building pfSense will "contaminate" your build host with pfSense speficic modifications. This is something I absolutely don't want and I'm now moving to using a jail as the build host.
    
    More to follow later.


  • Hail,

    jporter helped me to get the sources, but I can't find a post I had before that would walk through the steps to compile pfsense. I found the kernel dir (my main project), but can't figure out the what is the definitive script to config and build.

    I tried the build.sh –configure get me issues:

    %./build.sh --configure
    
    You must first run ./set_version.sh !
    See http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso for more information.
    
    You can also run ./menu.sh which will assist with the available options
    
    

    is there a new version of http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso ?

    thanks,

    none

    ps: if I should open new thread, please say. As this is on the subject, I figured here as a right place.



  • @kpa:

    Fair enough. I'll summarize quickly what I have learned so far:

    There is no documentation available for using the tools repo so one has trough trial and error and asking around gather the necessary information on how to use it. In my case the missing piece of information was that pfPorts needs to built first before anything else. Building of pfPorts is not done automatically if you just fire up the build script as```
    build.sh iso

    
    You need an ultra-clean FreeBSD installation on the build host because the tools want to build everything its way and anything non-standard in terms of configuration trips it very easily.
    
    Building pfSense will "contaminate" your build host with pfSense speficic modifications. This is something I absolutely don't want and I'm now moving to using a jail as the build host.
    
    More to follow later.
    

    Yes..  We maintain dedicated builders for this reason and more.

    There is 10 years of history to deal with in changing things.



  • I am trying to build on 8.3. when i run cd /usr/ports/textproc/expat2 && make depends install; i get error as shown in attchment. i have tried 3 different installations for i386 as well as amd64. Plz help on what im doing wrong




  • @bhawk6901:

    I am trying to build on 8.3. when i run cd /usr/ports/textproc/expat2 && make depends install; i get error as shown in attchment. i have tried 3 different installations for i386 as well as amd64. Plz help on what im doing wrong

    You have hit the problem with the FreeBSD 8.3 release not being compatible with the newer FreeBSD ports tree.  Changing out the make utility as someone described in a previous post may help.  You can also try this tip I was given and which worked for me:

    Ports tree doesn’t support FreeBSD 8.3 anymore, you need to use the stable branch of ports tree called “2014Q2” instead of head. You can get it using:

    svn co http://svn.freebsd.org/ports/branches/2014Q2 /usr/ports

    This branch won’t have latest version of all ports, it only gets security updates, but the core files (/usr/ports/Mk/*) are still complaint with 8.3.

    So to do the above on your builder, first delete the entire existing ports tree with –

    
    rm -rf /usr/ports
    
    

    – then run this command:

    
    svn co http://svn.freebsd.org/ports/branches/2014Q2 /usr/ports
    
    

    DO NOT perform a normal ports update in the future or you will break the builder again.  My suggestion would be to abandon building FreeBSD 8.3-based pfSense and use 2.2 instead since it uses the current FreeBSD 10-STABLE.

    Bill



  • @bmeeks:

    @bhawk6901:

    I am trying to build on 8.3. when i run cd /usr/ports/textproc/expat2 && make depends install; i get error as shown in attchment. i have tried 3 different installations for i386 as well as amd64. Plz help on what im doing wrong

    You have hit the problem with the FreeBSD 8.3 release not being compatible with the newer FreeBSD ports tree.  Changing out the make utility as someone described in a previous post may help.  You can also try this tip I was given and which worked for me:

    Ports tree doesn’t support FreeBSD 8.3 anymore, you need to use the stable branch of ports tree called “2014Q2” instead of head. You can get it using:

    svn co http://svn.freebsd.org/ports/branches/2014Q2 /usr/ports

    This branch won’t have latest version of all ports, it only gets security updates, but the core files (/usr/ports/Mk/*) are still complaint with 8.3.

    So to do the above on your builder, first delete the entire existing ports tree with –

    
    rm -rf /usr/ports
    
    

    – then run this command:

    
    svn co http://svn.freebsd.org/ports/branches/2014Q2 /usr/ports
    
    

    DO NOT perform a normal ports update in the future or you will break the builder again.  My suggestion would be to abandon building FreeBSD 8.3-based pfSense and use 2.2 instead since it uses the current FreeBSD 10-STABLE.

    Bill

    Thanks for your reply. I shifted to freebsd10 before you replied. When i apply patches, it fails saying /usr/pfSensesrc/src does not exist.
    Kindly guide me further



  • @bhawk6901:

    Thanks for your reply. I shifted to freebsd10 before you replied. When i apply patches, it fails saying /usr/pfSensesrc/src does not exist.
    Kindly guide me further

    Well, it's been a long time since I set up my builder virtual machine, but I seem to remember having to manually create several directories along the way as I hit various bumps.  As you have seen, the documentation is either missing and/or not updated in some cases.  Try manually creating the directory /usr/pfSensesrc using:

    
    mkdir -p /usr/pfSensesrc
    
    

    Then try the apply patches step again.  There may well be several directories you will need to manually create as the scripts seem to expect some to exist already, while others it will create if they are missing.

    Bill