Building pfSense 2.2 / FreeBSD 10.1 - WORKING STEPS

Hobby-Student

Hi Hobby-Student,

Seems like i made a mistake writing it up there, should have been "SSH-2 RSA", fixed above.

Thanks.
PiBa-NL

This also doesn't work for me. Every key generated with PuTTYgen is making problems. Perhaps you could add an alternative "0b" with ssh-keygen.

This Tools-Repo was driving me crazy… :o

PiBa

Hi Hobby-Student,

Seems i have just lost my ability to edit my own post..
Would have added your "ssh-keygen -b 1024" as an alternative otherwise.
I've made ssh keys with PuTTYgen twice, and they both worked.. I dont know why yours didn't.

Greetz PiBa-NL

tamldt

Hi PiBa!

I have created iso file of pfsense. But when I install from this iso file on Vmware WorkStation, CPU load very high, 100% if i creat virtual machine 1 CPU, 50% if 2 CPU, …

I see process check_reload_status used 100% CPU

You can help me fix it!

Thanks!

PiBa

Hi tamldt,

I dont build verry frequently.. so could be a bug was introduced somewhere recently..

The nic in the vm is 'bridged' to a real nic?

Another option is that the vm is not having a proper wan connection? And causes gateway checks to keep running and restarting services.. Does it keep that high cpu usage even after 5 or more minutes? Does it have proper dns / wan gateway settings, enough memory assigned?
Anything in the systemlogs that indicates a flapping nic or something?

I've seen check_reload_status being 'bussy' to for official builds to for a while after booting a vm..

Perhaps try and rebuild the whole iso again with a 'clean' environment, if you did some experimentation to get it to build in the first place..?

Other than those things i don't know why it would happen, sorry.

Greets PiBa-NL

tamldt

Hi PiBa!

i will try!

Thank PiBa

tamldt

Hi PiBa…!

i try and i success CPU load low :D, thank you so much...!

and i have an idea, make a file script to automatic compile pfsense ! do you know how to implement it ? if you can please help me, i'm just beginning to learn about pfsense :D

thank you so much...!

PiBa

Hi tamldt,

Nice to hear cpu load is good now.

As for scripting the +-8 steps required into maybe 4? (you still have to generate a ssh key, and upload it on pfSense portal, and install FreeBSD, and have to checkout pfsense-tools, before you can run a buildscripts in there..), i really don't see why another script would add to make it 'better'.. Choosing a version, and building an iso would be 'hardcoded' in the script, though now you can easily see where you might want to choose to build memstick images instead of an iso, or building pfSense 2.1 instead of 2.2, it would remove understanding of whats going on. That its actually possible to get a working .iso in just 8 steps is already nice i think..
As for how to script stuff, just look at the scripts that you've ran while getting it to compile the .iso , if you understand scripting it shouldn't be hard to build another script that calls them in order, and checks if they output errors. Sorry but i'm not going to invest time into making such a script..

As for learning pfSense, i really wouldn't think the first step would be to compile it yourself.. You would just install it from the released downloadable .iso files.. wouldn't you?

If you want to build it yourself, and presumably change 'things' in it, you will need to be knowledgeable about scripting/c++/php/stuff anyway, and likely will want to skip as many steps in the compilation process as possible, so putting it all in 1 script would only hinder development/testing of changes you make..

Hope you succeed with what you wanted to do, and try to push any improvements you make back upstream.

Greets PiBa-NL

tamldt

Hi PiBa!

Oke! I am very grateful for advice from your!

Thanks

yakar

Hi PiBa,

How can add mysql56-server to buildports.RELENG_2_2 file or how can i build iso with mysql56-server?

Thank you for your sharing..

PiBa

We seem to be wondering off of the actual topic here (or how i intended it to be anyway.). Which would be how to build pfSense the same as how it is build by the official buildservers of the pfSense team.

Any modifications you want to do to it like adding other packages / binaries, and how to change those are beyond the scope of this topic. So perhaps you could start a new topic with the question you have and a appropriate subject.

Anyway I don't know about adding mysql and whats needed for that, ive done verry little development on the pfSense binaries/iso, a little more with the php side of things, but that doesn't help for these questions.. I can only advice you to look at existing buildscripts and try to understand what go's where from those files.. Search for an existing included items/binaries, then try to add mysql in a similar manner and run the buildscripts again, and again, try a little different, and try again..(you get the picture ;) ).

yakar

Thank you very much for your answer.. :)

sheriftk

Hello…

I was trying to build 2.2.1 on FreeBSD 10.1 x64 following the instructions above and it stops while trying to build the ISO.

Builder is running the command: env __MAKE_CONF= BOOT_COMCONSOLE_SPEED=115200 BOOT_BOOT0_COMCONSOLE_SPEED=0 WITHOUT_CROSS_COMPILER=yes script -aq /home/pfsense/tools/logs/buildworld.amd64 make -j4 __MAKE_CONF= BOOT_COMCONSOLE_SPEED=115200 BOOT_BOOT0_COMCONSOLE_SPEED=0 WITHOUT_CROSS_COMPILER=yes SRCCONF=/home/pfsense/tools/builder_scripts/conf/src/src.conf.10 TARGET=amd64 TARGET_ARCH=amd64 LOCAL_ITOOLS="uuencode uudecode ex" buildworld
make: make: don't know how to make uudecode. Stop

I have tried with various options including starting the build process in a clean state but no joy. The ports build completes successfully. Thanks in Advance.

bhawk6901

builsding on freebsd10.1-release
getting error on build of mpd5
error is as follows
******pppoe.c:334:26: warning: 'memset' call operates on objects of type 'struct ngpppoe_init_data' while the size is based on a different type 'struct ngpppoe_init_data *const' [-Wsizeof-pointer-memaccess]
memset(idata, 0, sizeof(idata));
~~~~~ ^~~~~
pppoe.c:334:26: note: did you mean to dereference the argument to 'sizeof' (and multiply it by the number of elements)?
memset(idata, 0, sizeof(idata));
^~~~~
pppoe.c:718:11: warning: cast from 'char ' to 'const struct typelist ' increases required alignment from 1 to 4 [-Wcast-align]
tlist = (const struct typelist) resp->data;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pppoe.c:740:6: error: use of undeclared identifier 'NGM_ETHER_ATTACH'; did you mean 'NGM_ETHER_DETACH'?
NGM_ETHER_ATTACH, &name, sizeof(name)) < 0) {
^~~~~~~~~~~~~~~~
NGM_ETHER_DETACH
/usr/include/netgraph/ng_ether.h:70:2: note: 'NGM_ETHER_DETACH' declared here
NGM_ETHER_DETACH, / our way to be shut down */
^
pppoe.c:770:10: warning: cast from 'char *' to 'const struct hooklist *' increases required alignment from 1 to 4 [-Wcast-align]
hlist = (const struct hooklist *)resp->data;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pppoe.c:899:8: warning: cast from 'const struct pppoe_tag *' to 'const uint32_t *' (aka 'const unsigned int *') increases required alignment from 1 to 4 [-Wcast-align]
(const uint32_t)(pt + 1) == idx)
^~~~~~~~~~~~~~~~~~~~~~~~~
4 warnings and 1 error generated.
*** [pppoe.o] Error code 1

make[2]: stopped in /usr/ports/net/mpd5/work/mpd-5.7/src
1 error

make[2]: stopped in /usr/ports/net/mpd5/work/mpd-5.7/src
*** [_sub.all] Error code 2

make[1]: stopped in /usr/ports/net/mpd5/work/mpd-5.7
1 error

make[1]: stopped in /usr/ports/net/mpd5/work/mpd-5.7
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1******
Please help

bhawk6901

edit: built fine when i tried to build pfsense 2.2-development instead of 2.2.1 :/

dmitryluhtionov

Hi.
Can I get patched sources of mpd5 ?

I am planned to out mpd-5.8 in nearest time.

madscientist159

The –build-pfPorts option to ./build.sh no longer exists. What replaced it?

Thanks!

rbgarga

@madscientist159:

The –build-pfPorts option to ./build.sh no longer exists. What replaced it?

Thanks!

Are you using branch RELENG_2_2? This is the branch to build 2.2 images, and it still contains –build-pfPorts

madscientist159

@Renato:

Are you using branch RELENG_2_2? This is the branch to build 2.2 images, and it still contains –build-pfPorts

That did the trick. Thanks for the pointer in the right direction! :D

irah

Hi PiBa,

I am trying to build the latest pfsense using your steps and I believe the same steps might apply now too.
I don't find any of the aforementioned scripts (rebuild_bsdinstaller.sh and get_bsdinstaller.sh) in the repository, for rebuilding the bsdinstaller (step# 6).
I mean for building pfsense 2.2, neither master nor RELENG_2_2 branch has those scripts. Only RELENG_2_1 has those scripts, but then it's on top of freebsd 8.3.
Am I wrong in the steps or the repository for building it? Can you please help me out?

P.S. These were the most recent steps that I could find for building the latest pfsense.

Thanks.

irah

Thanks PiBa, your suggestion helped! The steps mentioned earlier in this thread are applicable for the latest build too, only change being in step# 6, where bsdinstaller has been integrated into step# 7 as a static port.

How to build: – pfSense 2.2.5 FreeBSD 10.1-PRERELEASE --

0 - Gain access to the pfsense-tools

Use PuTTYgen to generate a "SSH-2 RSA" key and Export the OpenSSH key to a file locally for usage later.
also save the 'ssh-rsa ..... key-YYYYMMDD' public key to fill in the online registration form.

Information about how the signup works also here: https://forum.pfsense.org/index.php?topic=76132.msg415051#msg415051
-Sign the ICLA or CCLA agreement online https://www.pfsense.org/about-pfsense/#cla
-then also sign the LA, and configure your public ssh key, that will gaining you access to the pfsense-tools repository after a few minutes.

my VirtualBox build machine configuration

disk : 20GB
vcpu : 4 << more could be faster..
mem : 3 GB is enough to use 'memory disks' while compiling which speeds up the build process, if you have plenty more probably wont hurt ;)

download and install into the virtual machine : http://ftp.freebsd.org/pub/FreeBSD/snapshots/amd64/amd64/ISO-IMAGES/10.1/FreeBSD-10.1-PRERELEASE-amd64-2014????-r27????-disc1.iso
Then also update it to the recent version:
freebsd-update fetch
freebsd-update install
-reboot after that

After this "uname -r" shows me: 10.1-RELEASE-p16

-install: lib & src
-enable: ssh
-configure user: admin

allow root to login over ssh / for easy access with winscp and putty (NEVER DO THIS on a production machine, only for testing/easy access)

echo PermitRootLogin yes >> /etc/ssh/sshd_config
service sshd restart

Login with WinSCP to be able to easely transfer files from/to the FreeBSD build machine.

copy your earlier saved ssh private key to the system using WinSCP to: /home/admin/private_putty_key_ssh.ppk

1. Begin pfSense bootstrap by running these shell commands:

echo "WITHOUT_X11=yo" > /etc/make.conf
echo "OPTIONS_UNSET=X11" >> /etc/make.conf
echo "BATCH=yo" >> /etc/make.conf
mkdir -p /home/pfsense/pfSenseGITREPO /usr/pfSensesrc

portsnap fetch extract
cd /usr/ports/devel/git && make depends install
rehash

2. Load SSH key and checkout sources for pfSense-tools from RELENG_2_2 branch

eval ssh-agent -c
chmod 600 /home/admin/private_putty_key_ssh.ppk
ssh-add /home/admin/private_putty_key_ssh.ppk

cd /home/pfsense && git clone -b RELENG_2_2 git@git.pfsense.org:pfsense-tools tools
-Are you sure you want to continue connecting (yes/no)? yes << required to type all 3 letters of y e s

3. Set correct publicly available repositories

'pfmechanics' is used by ESF internally for fast local mirrors of several repositories to speed up their build process it is not available from outside.

p.s. dont try to use git@ for github, it wil fail later because it will try to use the ssh key loaded above..

sed -i "" "s,git@git.pfmechanics.com:pfsense/pfsense-tools.git,git@git.pfsense.org:pfsense-tools," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:outsidemirrors/freebsd.git,https://github.com/freebsd/freebsd.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:pfsense/pfsense.git,https://github.com/pfsense/pfsense.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh
sed -i "" "s,git@git.pfmechanics.com:pfsense/bsdinstaller.git,https://github.com/pfsense/bsdinstaller.git," /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh

to check if the above sed commands went ok, the command below should show nothing:

cat /usr/home/pfsense/tools/builder_scripts/builder_defaults.sh | grep git.pfmechanics.com

4. Checkout Freesbie2

cd /home/pfsense && git clone git://github.com/pfsense/freesbie2.git freesbie2

5. set version to build

cd /home/pfsense/tools/builder_scripts
./set_version.sh RELENG_2_2 https://github.com/freebsd/freebsd.git

6. Rebuild the bsdinstaller.

This step can be skipped, creation of the bsdinstaller has been integrated into the script called at step7. It's now converted into a static port and is available under pfPorts.

7. Build ports

Now it will also be checking out FreeBSD sources, this will take quite some time with no visible progress, in the background the 'git' process will take a little cpu usage..

tmp file in /usr/pfSensesrc/src/.git/objects/pack will slowly grow over 800MB..

cd /home/pfsense/tools/builder_scripts
./build.sh --build-pfPorts --no-cleanrepos

to check progress see what ports will be build in: /usr/home/pfsense/tools/builder_scripts/conf/pfPorts/buildports.RELENG_2_2

It should show all ports build without failures..

>>> Ports with failures: 0
>>> Building tools/crytpo...Done!
==> End of pfPorts...
>>> Operation ./build.sh has ended at Wed Nov 5 22:16:57 CET 2014

7. Build pfSense iso

cd /home/pfsense/tools/builder_scripts
./build.sh iso --no-cleanrepos

ISO created: Wed Nov 5 22:53:02 CET 2014
NOTE: waiting for jobs: to finish...
/tmp/pfSense/ now contains:
total 239208
drwxr-xr-x 2 root wheel 512B Nov 5 22:53 .
drwxrwxrwt 10 root wheel 512B Nov 5 22:52 ..
-rw-r--r-- 1 root wheel 234M Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso
-rw------- 1 root wheel 0B Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso.gz
Operation ./build.sh has ended at Wed Nov 5 22:53:02 CET 2014

8. YOUR DONE (hopefully)

The resulting iso should show
ls -l /tmp/pfSense/

-rw-r--r-- 1 root wheel 90629317 Nov 5 22:53 pfSense-LiveCD-2.2-BETA-amd64-20141105-2218.iso.gz

p.s.

Im using --no-cleanrepos flag above to speedup second and further build attempts(the same day?) by skipping the slow checkout of FreeBSD sources.

LOGS

Kernel buildworld log:
/usr/home/pfsense/tools/logs/buildworld.amd64
ports building (failure) logs (empty is good, during building there can be temporary logfiles here):
/tmp/pfPort/buildlogs/

To apply a custom patch written against FreeBSD code this is one way to apply it

add patch file divert-reply10.1.patch to : /usr/home/pfsense/tools/patches/releng/10.1
add the patch to the list of patches, depending on the format of the patch git/diff you might need to specify it a little different.:
echo "-p1~~divert-reply10.1.patch~" >> /usr/home/pfsense/tools/builder_scripts/conf/patchlist/releng/10.1/patches