Building pfSense 2.2 / FreeBSD 10.1 - WORKING STEPS

irah

Hi PiBa,

I am trying to build the latest pfsense using your steps and I believe the same steps might apply now too.
I don't find any of the aforementioned scripts (rebuild_bsdinstaller.sh and get_bsdinstaller.sh) in the repository, for rebuilding the bsdinstaller (step# 6).
I mean for building pfsense 2.2, neither master nor RELENG_2_2 branch has those scripts. Only RELENG_2_1 has those scripts, but then it's on top of freebsd 8.3.
Am I wrong in the steps or the repository for building it? Can you please help me out?

P.S. These were the most recent steps that I could find for building the latest pfsense.

Thanks.

irah

Thanks PiBa, your suggestion helped! The steps mentioned earlier in this thread are applicable for the latest build too, only change being in step# 6, where bsdinstaller has been integrated into step# 7 as a static port.

How to build: – pfSense 2.2.5 FreeBSD 10.1-PRERELEASE --

0 - Gain access to the pfsense-tools

Use PuTTYgen to generate a "SSH-2 RSA" key and Export the OpenSSH key to a file locally for usage later.
also save the 'ssh-rsa ..... key-YYYYMMDD' public key to fill in the online registration form.

Information about how the signup works also here: https://forum.pfsense.org/index.php?topic=76132.msg415051#msg415051
-Sign the ICLA or CCLA agreement online https://www.pfsense.org/about-pfsense/#cla
-then also sign the LA, and configure your public ssh key, that will gaining you access to the pfsense-tools repository after a few minutes.

my VirtualBox build machine configuration

disk : 20GB
vcpu : 4 << more could be faster..
mem : 3 GB is enough to use 'memory disks' while compiling which speeds up the build process, if you have plenty more probably wont hurt ;)

download and install into the virtual machine : http://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-PRERELEASE-amd64-2014????-r27????-disc1.iso
Then also update it to the recent version:
freebsd-update fetch
freebsd-update install
-reboot after that

After this "uname -r" shows me: 10.1-RELEASE-p16

-install: lib & src
-enable: ssh
-configure user: admin

allow root to login over ssh / for easy access with winscp and putty (NEVER DO THIS on a production machine, only for testing/easy access)

echo PermitRootLogin yes >> /etc/ssh/sshd_config
service sshd restart

Login with WinSCP to be able to easely transfer files from/to the FreeBSD build machine.

copy your earlier saved ssh private key to the system using WinSCP to: /home/admin/private_putty_key_ssh.ppk

1. Begin pfSense bootstrap by running these shell commands:

echo "WITHOUT_X11=yo" > /etc/make.conf
echo "OPTIONS_UNSET=X11" >> /etc/make.conf
echo "BATCH=yo" >> /etc/make.conf
mkdir -p /home/pfsense/pfSenseGITREPO /usr/pfSensesrc

portsnap fetch extract
cd /usr/ports/devel/git && make depends install
rehash

2. Load SSH key and checkout sources for pfSense-tools from RELENG_2_2 branch

eval ssh-agent -c
chmod 600 /home/admin/private_putty_key_ssh.ppk
ssh-add /home/admin/private_putty_key_ssh.ppk

cd /home/pfsense && git clone -b RELENG_2_2 git@git.pfsense.org:pfsense-tools tools
-Are you sure you want to continue connecting (yes/no)? yes << required to type all 3 letters of y e s

3. Set correct publicly available repositories

'pfmechanics' is used by ESF internally for fast local mirrors of several repositories to speed up their build process it is not available from outside.

p.s. dont try to use git@ for github, it wil fail later because it will try to use the ssh key loaded above..

sed -i "" "s,git@git.pfmechanics.com:pfsense/pfsense-tools.git,git@git.pfsense.org:pfsense-tools," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:outsidemirrors/freebsd.git,https://github.com/freebsd/freebsd.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:pfsense/pfsense.git,https://github.com/pfsense/pfsense.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:pfsense/bsdinstaller.git,https://github.com/pfsense/bsdinstaller.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh

to check if the above sed commands went ok, the command below should show nothing:

cat /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh | grep git.pfmechanics.com

4. Checkout Freesbie2

cd /home/pfsense && git clone git://github.com/pfsense/freesbie2.git freesbie2

5. set version to build

cd /home/pfsense/tools/builder_scripts
./set_version.sh RELENG_2_2 https://github.com/freebsd/freebsd.git

6. Rebuild the bsdinstaller.

This step can be skipped, creation of the bsdinstaller has been integrated into the script called at step7. It's now converted into a static port and is available under pfPorts.

7. Build ports

Now it will also be checking out FreeBSD sources, this will take quite some time with no visible progress, in the background the 'git' process will take a little cpu usage..

tmp file in /usr/pfSensesrc/src/.git/objects/pack will slowly grow over 800MB..

cd /home/pfsense/tools/builder_scripts
./build.sh --build-pfPorts --no-cleanrepos

to check progress see what ports will be build in: /usr/home/pfsense/tools/builder_scripts/conf/pfPorts/buildports.RELENG_2_2

It should show all ports build without failures..

>>> Ports with failures: 0
>>> Building tools/crytpo...Done!
==> End of pfPorts...
>>> Operation ./build.sh has ended at Wed Nov 5 22:16:57 CET 2014

7. Build pfSense iso

cd /home/pfsense/tools/builder_scripts
./build.sh iso --no-cleanrepos

ISO created: Wed Nov 5 22:53:02 CET 2014
NOTE: waiting for jobs: to finish...
/tmp/pfSense/ now contains:
total 239208
drwxr-xr-x 2 root wheel 512B Nov 5 22:53 .
drwxrwxrwt 10 root wheel 512B Nov 5 22:52 ..
-rw-r--r-- 1 root wheel 234M Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso
-rw------- 1 root wheel 0B Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso.gz
Operation ./build.sh has ended at Wed Nov 5 22:53:02 CET 2014

8. YOUR DONE (hopefully)

The resulting iso should show
ls -l /tmp/pfSense/

-rw-r--r-- 1 root wheel 90629317 Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso.gz

p.s.

Im using --no-cleanrepos flag above to speedup second and further build attempts(the same day?) by skipping the slow checkout of FreeBSD sources.

LOGS

Kernel buildworld log:
/usr/home/pfsense/tools/logs/buildworld.amd64
ports building (failure) logs (empty is good, during building there can be temporary logfiles here):
/tmp/pfPort/buildlogs/

To apply a custom patch written against FreeBSD code this is one way to apply it

add patch file divert-reply10.1.patch to : /usr/home/pfsense/tools/patches/releng/10.1
add the patch to the list of patches, depending on the format of the patch git/diff you might need to specify it a little different.:
echo "-p1~~divert-reply10.1.patch~" >> /usr/home/pfsense/tools/builder_scripts/conf/patchlist/releng/10.1/patches

sreekupmgmail.com

Hii,

I am a new comer, Please assist me how to build own pfSense

PiBa

Hi sreekupm,

Step 1: READ the posts above yours !
Step 2: read some more forum posts, try to do the steps, then read some more.
Step 3: fail while trying those steps, then try to understand the error's show, gather logfiles, look in those logs for clues what might have gone wrong.
Step 4: search the forum and use google to find possible solutions to your problems..
Step 5: try those possible solutions.
Step 6: repeat above steps 3 times (or more..)

Step 7: then, and only then, clearly explain what you have done, tried and failed at, what errors you have seen, what step you could not follow..

Now please follow these instructions and you might get a more useful reply next time..

Some computer knowledge and is expected from a developer so we are not going to tell every mouse click you need to do..
What would you say when i report you the problem that the program/website you created is not working properly? Wouldn't you like to know what button&page i was talking about and what behavior i expected but didn't get?

I'm sorry if i sound a bit harsh, but please do understand that your question is useless like it is currently written..

Regards,
PiBa-NL

S. Kirschner

Hi,

I couldnt clone the repo's.

Following steps I performed
1. accepted Individual CLA
2. accepted the License Agreement (also insert my pubkey)
3. imported the openssh private key in my freebsd machine (eval ssh-agent -c , ssh-add /path/to/private/openssh_key
4. waited 1 hour
5. tried to clone (git clone -b RELENG_2_2 git@git.pfsense.org:pfsense-tools tools)
And here I´m asked for an Password (Not the password from my private key)

To verifiy if I made an mistake with my pub key , i uploaded the pub key to github.com and tested the verification (ssh -T git@github.com), these worked as it should and returned.
"Hi skirschner! You've successfully authenticated, but GitHub does not provide shell access."

My Pfsense profil contain my github username and the pubkey , also the subscriptions would displayed as active.

Have someone an idea ?

[root@freebsd-2836] /home/pfsense# git clone -b RELENG_2_2 git@git.pfsense.org:pfsense-tools tools
Cloning into 'tools'...
Password:

PiBa

Im not sure what is currently the best repository to clone from.. But you could try cloning from github instead. Now that you have signed the ICLA and LA you have also been added to the 'pfsense-tools team'..
what happens if you try? The following:

git clone -b RELENG_2_2 git@github.com:pfsense/pfsense-tools.git tools

S. Kirschner

Thank you , it worked for me.

Now i have to find out how to build a specific appilcation in a pbi.

doktornotor

@S.:

Now i have to find out how to build a specific appilcation in a pbi.

Not really sure why'd anyone do that… Use pkg instead. https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages#pfSense_2.2

S. Kirschner

@doktornotor:

@S.:

Now i have to find out how to build a specific appilcation in a pbi.

Not really sure why'd anyone do that… Use pkg instead. https://doc.pfsense.org/index.php/Installing_FreeBSD_Packages#pfSense_2.2

Yes allready tried it , but I would like to use some options that isnt available in "stock".

Thats the reason why i try to build my own pbi for the pfsense machine.

PiBa

Still using pkg instead of pbi makes sense.

Building a single PBI takes hours on my system.. While using a 'pkg create YourPackage' on a FreeBSD system where the package was compiled and then 'pkg add YourPackage-1.2.3.txz' on pfSense the whole process can be done in a matter of a few minutes.

doktornotor

^^^ There. Plus, worse yet, the end result with PBI is usually a completely broken thing needing tons of hacks to work.

S. Kirschner

Ok thats are some points why the pkg method gets a + :)

But I try it with a pbi , for following reason:
1.keep the usability of the webinterface for that software
2.keep the possability to update the software later via the webinterface without "tons" of old useless files
3.publish my expierence in the forum, that other got an better idea how it works and maybe faster an update for that software could be published to all via the offical repo.

The software i try to build is the latest version of squid 3 (which is actual the version 3.5.8 and the latest (not fully offical) available version for pfsense is 3.5.3 at the moment).
And i hope if its get working that the offical version of squid 3 got updated for pfsense that all other could use the new features.

At the moment I stuck a bit with the default prefix path.

doktornotor

That's just completely wrong reason. The usability with the web GUI will be much better if you avoid PBI completely. With huge things like this, probably some 1/3 of the code could be dropped altogether if there was no PBI in the first place. Huge part of the rest could be simplified and made much less buggy.

Squid 3.5.x is already compiled as PBI (https://files.pfsense.org/packages/10/All/). Upgrade never happened mostly likely because

it'd require another round of insane hacking around the PBI screwed paths in the XML/PHP package (the webGUI)
the whole binary thing doesn't work properly anyway

If you want to work on Squid 3.5.x GUI integration

install Squid via pkg
take the current package code - https://github.com/pfsense/pfsense-packages/tree/master/config/squid3/34
fix the package code so that it points to the sane paths under the /usr/local prefix, ditching all the PBI and pfSense-version specific shit/generally useless garbage (if,else,blah…)

S. Kirschner

@doktornotor:

Squid 3.5.x is already compiled as PBI (https://files.pfsense.org/packages/10/All/). Upgrade never happened mostly likely because

it'd require another round of insane hacking around the PBI screwed paths in the XML/PHP package (the webGUI)

the whole binary thing doesn't work properly anyway

What I dont get is , I think between 3.5.3 and 3.5.9 didnt changed so much in the file structure,
why the person who build 3.5.3 couldnt fetch the new files and build 3.5.9 ?

doktornotor

Hmm? Why'd you waste hours of time building something that ain't gonna be used anywhere?

sreekupmgmail.com

@PiBa:

Hi sreekupm,

Step 1: READ the posts above yours !
Step 2: read some more forum posts, try to do the steps, then read some more.
Step 3: fail while trying those steps, then try to understand the error's show, gather logfiles, look in those logs for clues what might have gone wrong.
Step 4: search the forum and use google to find possible solutions to your problems..
Step 5: try those possible solutions.
Step 6: repeat above steps 3 times (or more..)

Step 7: then, and only then, clearly explain what you have done, tried and failed at, what errors you have seen, what step you could not follow..

Now please follow these instructions and you might get a more useful reply next time..

Some computer knowledge and is expected from a developer so we are not going to tell every mouse click you need to do..
What would you say when i report you the problem that the program/website you created is not working properly? Wouldn't you like to know what button&page i was talking about and what behavior i expected but didn't get?

I'm sorry if i sound a bit harsh, but please do understand that your question is useless like it is currently written..

Regards,
PiBa-NL

I had done above steps but stuck up here what I can do?

[2.2.4-RELEASE][admin@pfSense.localdomain]/home/pfsense: git clone -b RELENG_2_2 git@github.com:pfsense/pfsense-tools.git tools
Cloning into 'tools'…
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

PiBa

Surely you did not do "step7: clearly explain what you have done, tried and failed at".
Anyway..

So, did you sign the ICLA or CCLA and the LA for the pfSense portal ? Did you recieve a email that you added to the pfsense-tools team? Did you configure your public ssh key to github?

sreekupmgmail.com

Hi PiBa

I already sign Individual CLA - Lifetime

I am stuck up
root@develop:/usr/ports/devel/git # cd /home/pfsense && git clone git@git.pfsense.org:pfsense-tools tools
Cloning into 'tools'…
The authenticity of host 'git.pfsense.org (208.123.73.74)' can't be established.
ECDSA key fingerprint is 52:7b:22:f9:3b:7e:dd:85:85:1a:a5:98:b8:8f:7f:62.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'git.pfsense.org' (ECDSA) to the list of known hosts.
Password: Which Password??
Password for git@git.pfsense.org: Which Password??
Password:
Password for git@git.pfsense.org:
Password:
Password for git@git.pfsense.org:
Permission denied (publickey,keyboard-interactive).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

phil.davis

pfsense-tools moved onto GitHub as a private repo a while ago. After completed the CLA you should have it as a repo in GitHub. That GitHub repo has the RELENG_2_2 branch from which you should be able to clone and build 2.2.* …

I guess the CLA signup code is no longer automagically adding access to the git.pfsense.org:pfsense-tools repo?

@cmb @jimp ?

PiBa

@sreekupm@gmail.com:

I had done above steps but stuck up here what I can do?
[2.2.4-RELEASE][admin@pfSense.localdomain]/home/pfsense: git clone -b RELENG_2_2 git@github.com:pfsense/pfsense-tools.git tools
Cloning into 'tools'…
Permission denied (publickey).
fatal: Could not read from remote repository.

Some questions where asked after you asked that.
@PiBa:

So, did you sign the ICLA or CCLA and the LA for the pfSense portal ?

Did you recieve a email that you added to the pfsense-tools team?

Did you configure your public ssh key to github?

Which you did not answer..

@sreekupm@gmail.com:

I am stuck up
root@develop:/usr/ports/devel/git # cd /home/pfsense && git clone git@git.pfsense.org:pfsense-tools tools
Cloning into 'tools'…

Indeed you are.. And probably will be for a while longer.. Why now try and clone a different repository? Others have confirmed the github repository to be working fine? (when ICLA/CCLA LA are signed, and ssh key has been configured)

If i may repeat myself.. Please read the other posts in this thread, and don't blindly run commands until something breaks, instead try to understand what your doing, pretty much all information required is there already.

Could you please answer the 3 questions above before continuing with other attempts?