Creating pfSense 2.0 Developers ISO on FreeBSD Beta 8.1

trijames · Jun 22, 2010, 5:52 PM

Hi All,

apologies if this is posted anywhere else…

I'm trying to build a pfsense developers ISO...

I've tried 2 things:

1.) tried to follow the instructions DevelopersBootStrapAndDevIso on http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso

with FreeBSD Beta 8.1 RC1. But, with no success. I am getting an error during the ISO creation stage:

">>> Checking BSDInstaller health... dfuife_curses missing [ERROR!]

####################################
Something went wrong, check errors!
####################################
"

If anybody has some insight on where to start looking - you would be a tremendous help!

2.) And I've also tried to to grab one of the pre-made developer ISO's from:
http://devwiki.pfsense.org/UsingProjectEvilOnpfSense
It doesn't look like any 2.0 are available (obviously since this is still beta)…
Is this page still maintained? Or is no 2.0 there because it is still beta?

On a general note: What can I do at this stage of flux-development? I really would like to contribute - but can't seem to get an environment setup where I could be of any help.

thanks
James

Is this page still up

jimp · Jun 23, 2010, 3:00 PM

We don't use the Dev ISO any more, to be honest. If you want to build something, a standard FreeBSD 8.1-RC box would be fine.

That said, I just made a builder VM the other day and didn't get that error, but I didn't use the menu.

I did something like this:

Install FreeBSD 8.0
Update source with csup and then build and install world and kernel
Update ports tree (I use csup for ports, not portsnap as on the instructions)

From http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso
Do steps 1, 2, 3
Step 4 was already done when I updated the base system above

For Step 5, I ran ./set_version.sh and set the branch and such I wanted

Run ./apply_kernel_patches.sh

cd /home/pfsense/tools/pfPorts/php5/; make all install clean
cd /home/pfsense/tools/pfPorts/php5-pfSense-module/; make all install clean
cd /usr/ports/devel/readline/; make all install clean
(That last one may not be needed, but I had a vague recollection of running it the last time I made a VM for this)

Then:
cd /home/pfsense/tools/builder_scripts
./build_iso.sh

trijames · Jun 25, 2010, 4:16 PM

Hi Jimp,

thanks for the heads up. Just curious - you mentioned you can build with standard freebsd 8.1 rc1 - but in your instructions you referenced 8.0. Does that mean that I should definitely only try this with 8.0? I Just want to make sure I'm not over-thinking this.

regards
trijames

jimp · Jun 25, 2010, 4:22 PM

If you start with an ISO for 8.1-RC then it should be OK - but you still must install the source and such anyhow, and IIRC you need the files that a buildworld/buildkernel makes anyhow, so it really doesn't matter where you start as long as you end up with a recent RELENG_8_1 world, kernel, sources, etc.

trijames · Jun 28, 2010, 1:33 PM

HI Jimp,

thanks again for the updated procedures - that seems to be getting me further.
Unfortunately, I'm still having an issue:
when I try to apply kernel patches - I get the following error:
WARNING! Rejected patches found! Please fix before building!

/usr/pfSensesrc/src/contrib/pf/pfctl/parse.y.rej
/usr/pfSensesrc/src/sbin/ipfw/ipfw2.c.rej
/usr/pfSensesrc/src/sys/contrib/pf/net/pf.c.rej
…. (omitted for e-z reading)
/usr/pfSensesrc/src/usr.bin/netstat/unix.c.rej
####################################
Something went wrong, check errors!
####################################

Did I screw up somewhere? or is this a known problem.
Sorry If I'm not too familiar with this build process - I'm doing it for the first time and trying to do it as systematic and logical as possible.
How do I go about fixing this error?

regards
Trijames

jimp · Jun 28, 2010, 1:41 PM

That doesn't look familiar, but rejected patches are usually a sign that your source tree doesn't match up with what is expected, or that something was updated upstream (in FreeBSD) and the patches on pfSense may need adjusted.

What settings did you use when you set the version to build?

The contents of /home/pfsense/tools/builder-scripts/pfsense-build.conf would help.

trijames · Jun 28, 2010, 9:40 PM

Hi Jimp,

thanks.
just fyi… I ended up using freebsd 8.1 RC1 as my base.
Then i run through the steps as outlined.
The version I set is RELENG_8_1...
But it seems that my FREEBSD_BRANCH is 8_0 as shown below...
Is there something wrong with that?

Here's my pfsense-build.conf below.

export PFSENSE_VERSION=2.0-BETA3
export FREEBSD_VERSION=8
export FREEBSD_BRANCH=RELENG_8_0
export PFSENSETAG=HEAD
export PFSPATCHFILE=/usr/home/pfsense/tools/builder_scripts/../builder_scripts/patches.RELENG_8_1
export PFSPATCHDIR=/usr/home/pfsense/tools/builder_scripts/../patches/RELENG_8_1
export SUPFILE=/usr/home/pfsense/tools/builder_scripts/../builder_scripts/RELENG_8-supfile
export CUSTOM_COPY_LIST=/usr/home/pfsense/tools/builder_scripts/../builder_scripts/copy.list.RELENG_8_0
export OVERRIDE_FREEBSD_CVSUP_HOST=cvsup4.us.freebsd.org
export BASE_DIR=/usr/home/pfsense/tools/builder_scripts/../..
export BUILDER_TOOLS=/usr/home/pfsense/tools/builder_scripts/..
export BUILDER_SCRIPTS=/usr/home/pfsense/tools/builder_scripts
export PFSPORTSFILE=buildports.RELENG_2_0
freebsd81rc1386#

jimp · Jun 28, 2010, 9:47 PM

No that all looks right. Some things are named 8_0 and haven't been renamed to 8_1 yet, but they work for any 8, not just 8_0.

trijames · Jun 28, 2010, 10:07 PM

hi Jimp,

interesting… I wonder why it's not working...
here's the full text of the error I'm getting during apply-kernel-patches.sh
If you have any ideas - I'm more than happy to try it out...

Setting CVSUp host to cvsup4.us.freebsd.org
Removing needed files listed in patches.RELENG_8_1 HEAD
Obtaining FreeBSD sources RELENG_8-supfile...Done!
Removing old patch rejects...
Applying patches, please wait...Hunk #1 failed at 1637.
Hunk #2 failed at 1669.
2 out of 2 hunks failed--saving rejects to sys/net/if.c.rej
Hunk #1 failed at 875.
1 out of 1 hunks failed--saving rejects to sys/net/if_var.h.rej
Hunk #1 failed at 519.
Hunk #2 failed at 686.
Hunk #3 failed at 797.
3 out of 3 hunks failed--saving rejects to sys/net/route.c.rej
Hunk #1 failed at 55.
Hunk #2 failed at 674.
2 out of 2 hunks failed--saving rejects to sys/net/rtsock.c.rej
Hunk #1 failed at 1379.
1 out of 1 hunks failed--saving rejects to sys/netinet/in.c.rej
Hunk #1 failed at 590.
Hunk #2 failed at 707.
2 out of 2 hunks failed--saving rejects to sys/netinet/in_pcb.c.rej
Hunk #1 failed at 230.
1 out of 1 hunks failed--saving rejects to sys/netinet/ip_options.c.rej
Hunk #1 failed at 247.
1 out of 1 hunks failed--saving rejects to sys/netinet/ip_output.c.rej
Done!
Finding patch rejects...

WARNING! Rejected patches found! Please fix before building!

/usr/pfSensesrc/src/sys/net/if.c.rej
/usr/pfSensesrc/src/sys/net/if_var.h.rej
/usr/pfSensesrc/src/sys/net/route.c.rej
/usr/pfSensesrc/src/sys/net/rtsock.c.rej
/usr/pfSensesrc/src/sys/netinet/in.c.rej
/usr/pfSensesrc/src/sys/netinet/in_pcb.c.rej
/usr/pfSensesrc/src/sys/netinet/ip_options.c.rej
/usr/pfSensesrc/src/sys/netinet/ip_output.c.rej

####################################
Something went wrong, check errors!
####################################

trijames · Jun 28, 2010, 10:20 PM

Hi Jimp,

in case it helps - here's exactly the steps I'm doing:

1.) install freebsd 8.1 RC1

2.) update the kernel source with csup
2a.) copy the supfile to the /etc

cp /usr/share/examples/cvsup/standard-supfile /etc/supfile

2b.) edit the supfile

vi /etc/supfile

2c.) add the csup server
host=CHANGE_THIS.FreeBSD.org
Change to:
host=cvsup4.us.FreeBSD.org
2d.) run csup
#> csup /etc/supfile

3.) build the kernel

cd /usr/src

make buildworld

make buildkernel

make installkernel

shutdown -r now

4.) boot into single user mode

boot -s from the boot loader prompt

adjkerntz -i

mount -a -t ufs

mergemaster -p

cd /usr/src

make installworld

mergemaster

reboot

5.) update the ports directory using csup

rm -fr /usr/ports/*

csup -L 2 /etc/supfile

6.) configure make (step 1 from devwiki.pfsense/org/DevelopersBootStrapAndevIso)

echo "WITHOUT_X11=yo" >> /etc/make.conf

mkdir -p /home/pfsense/pfSenseGITREPO /home/pfsense/installer /usr/pfSensesrc

7.) grab ports and build GIT

portsnap fetch extract

cd /usr/ports/textproc/expat2 && make depends install

cd /usr/ports/devel/git && make depends install

(take the standard default settings from the menu's)

cd /usr/ports/sysutils/fastest_cvsup/ && make depends install

(take the standard default settings from the menu's)

rehash

8.) continue on with the GIT repo checkouts

cd /home/pfsense && git clone http://gitweb.pfsense.org/pfsense-tools/mainline.git tools

cd /home/pfsense && git clone http://gitweb.pfsense.org/freesbie2/mainline.git freesbie2

cd /home/pfsense/tools/builder_scripts && chmod a+rx *.sh

9.) now do the pfsense stuff

cd /home/pfsense/tools/builder_scripts

./set_version.sh RELENG_8_1 cvsup4.us.freebsd.org

./apply_kernel_patches.sh

(here's where I get the reject errors…)

(the rest I don't get to because of the above errors)

cd /home/pfsense/tools/pfPorts/php5; make all install clean

cd /home/pfsense/tools/pfPorts/php5-pfSense-module/; make all install clean

cd /usr/ports/devel/readline/; make all install clean

10 now create the ISO

cd/ home/pfsense/tools/builder_scripts

./build_iso.sh

trijames · Jun 28, 2010, 10:23 PM

Hi Jimp,

one more thing - I'm building this as a VM… could that be the cause?

regards
TJ

jimp · Jun 29, 2010, 2:59 PM

VM is fine, nothing wrong with that.

Just checked on our main snapshot builder and the patches are failing there, too, so it must be an upstream update that happened.

EDIT: I restarted the build and it worked. Try updating the sources, git repos, etc, and then make another attempt.

trijames · Jun 29, 2010, 3:19 PM

Hi Jimp,

ok I guess that's good news - and will get fixed soon.
What I notice is that the errors are all related to the tcp stack
ip.c
route.c
rtsock.c
etc…

So, would seem to me that something has changed there.
What does it mean when these are "rejects"? Is it basically a 'diff' that found between the freebsd baseline and pfrsense baseline?

regards
TJ

trijames · Jun 29, 2010, 3:24 PM

Oh - I just saw your updated EDIT…
Ok - I'll try to update and see if that works...
Which source should I update? kernel? or pfsense? or both?

regards
TJ

jimp · Jun 29, 2010, 3:35 PM

Try this:

# cd/ home/pfsense/tools/builder_scripts
# ./update_git_repos.sh
# ./build_iso.sh

trijames · Jun 29, 2010, 3:50 PM

Jimp,

not sure if I understand…
Just in case - I want to check my thinking:
1.) should I not do the following:
# ./apply_kernel_patches.sh
# cd /home/pfsense/tools/pfPorts/php5; make all install clean
# cd /home/pfsense/tools/pfPorts/php5-pfSense-module/; make all install clean
# cd /usr/ports/devel/readline/; make all install clean

So even though apply_kernel_patches.sh failed - I should instead do update_git_repos.sh?
And also skip the rest? Is that right?

Or should I still do the above steps and then also do the update_git_repos.sh?

regards
TJ

jimp · Jun 29, 2010, 3:52 PM

Whatever you do, do update_git_repos.sh first. I'd just try to build the iso first and see what happens. It should do some of that automatically, but if it fails, then fall back to doing those steps (apply patches, build those ports, etc)

Though you can compile those ports whether or not the kernel patches apply

trijames · Jun 29, 2010, 4:35 PM

ah ok - excellent. I'll give that a try.

jimp · Jun 29, 2010, 9:15 PM

That didn't work. I had to adjust the supfile to have a date, and go back a few days. If you update_git_repos.sh again and then redo the build it should succeed.

trijames · Jun 30, 2010, 1:06 AM

Jimp,

does it matter which which shell I use to run these?
I just saw a warning about ksh as below in the build_iso.sh

"the O_NONBLOCK symbol with a syntax error. This is apparently a
sh error. Rerunning Configure with ksh apparently fixes the
error, then please read the section "64 bit data in perl 5.6.0 and 5.6.1"

(I'm building the i386 version)

regards
TJ