pfSense compile requirements for 3rd party software
-
@bmeeks said in pfSense compile requirements for 3rd party software:
I don't try to build a pfSense kernel. All I build is the packages repo, but even for that now I have to go in and manually remove the runtime dependencies from the pfSense-upgrade package so that it does not try and pull in the proprietary pfSense-repoc package
This is all I am really trying to do as well. I had the instructions I wrote from 2_6_0 to go on, which obviously won't work now based on everything you have pointed out that has changed.
Do I simply clone FreeBSD-ports and try to build again without the jail? What might be the path forward?
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
Do I simply clone FreeBSD-ports and try to build again without the jail? What might be the path forward?
You can build the jail. You just can't build a pfSense kernel itself.
Clone both the
FreeBSD-ports
tree and thepfSense
tree from the public GitHub repo.I edited the
pfsense/build.conf
file to point to the RELENG_2_7_2 branch.Then you need to comment out the lines pertaining to GNID in the
builder_common.sh
file so that the jail build does not try and pull down the proprietary Netgate ID source code files.Change into the
pfsense
git clone directory.Next, fetch the FreeBSD source files with:
./build.sh --update-sources
.You should now be able to create the poudriere jail with
./build.sh --setup-poudriere
.I then edited
tools/conf/pfPorts/poudriere.bulk
to comment out thepfSense
package so that a build of it will not be attempted (which would fail during the fetch phase due to the GitLab host location). Here is the edit topoudriere.bulk
:#security/%%PRODUCT_NAME%%
Now go into your Poudriere ports tree (
/usr/local/poudriere/ports/
) and find thesysutils/pfSense-upgrade
package. Edit its runtime depencies in the Makefile to comment out thepfSense-repoc
package.Packages should build now unless I've forgotten something off the top of my head.
But note the steps outlined above will NOT build a pfSense image. They only allow you to build the
FreeBSD-ports
packages repo tree. -
Thanks, that was helpful. I assume I now need the "-c" option for build.sh so that it doesn't try to build the kernel, but yet it still seems to need it see "installing kernel" in the logs below.
# ./build.sh none -a amd64.amd64 -c >>> Operation ./build.sh has started at Mon Feb 12 03:04:31 EST 2024 >>> Building image type(s): >>> Cleaning up previous build environment...Please wait! >>> Cleaning build directories: Done! >>> Cleaning previously built kernel stage area...Done! >>> Cleaning previously built images...Done! >>> Cleaning previous builder logs...Done! >>> Cleaning of builder environment has finished. >>> NO_BUILDWORLD and NO_BUILDKERNEL set, skipping update of freebsd sources >>> Last known commit Luiz Otavio O Souza - cf612ab9fc5711351fef5e0678d687aa3b88355d >>> LOGFILE set to /build/pfsense/logs/buildworld.amd64. >>> NO_BUILDWORLD set, skipping build >>> Building pfSense kernel. >>> NO_BUILDKERNEL set, skipping build >>> Staging pfSense kernel... >>> Installing kernel (pfSense) for amd64 architecture... ====>> ERROR: SRCCONF is pointing to a nonexistent file /build/pfsense/tmp/FreeBSD-src/release/conf/pfSense_src.conf #################################### Something went wrong, check errors! ####################################
A second issue I am having is that no matter what I try, the jail build always fails for me on 2.7.2 when it gets to this:
--- secure/lib/libcrypto__L --- make[4]: make[4]: don't know how to make aes-586.S. Stop make[4]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src/secure/lib/libcrypto make[3]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src
At a loss on how to solve that one. It works fine on 2.7.1 though.
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
Thanks, that was helpful. I assume I now need the "-c" option for build.sh so that it doesn't try to build the kernel, but yet it still seems to need it see "installing kernel" in the logs below.
# ./build.sh none -a amd64.amd64 -c >>> Operation ./build.sh has started at Mon Feb 12 03:04:31 EST 2024 >>> Building image type(s): >>> Cleaning up previous build environment...Please wait! >>> Cleaning build directories: Done! >>> Cleaning previously built kernel stage area...Done! >>> Cleaning previously built images...Done! >>> Cleaning previous builder logs...Done! >>> Cleaning of builder environment has finished. >>> NO_BUILDWORLD and NO_BUILDKERNEL set, skipping update of freebsd sources >>> Last known commit Luiz Otavio O Souza - cf612ab9fc5711351fef5e0678d687aa3b88355d >>> LOGFILE set to /build/pfsense/logs/buildworld.amd64. >>> NO_BUILDWORLD set, skipping build >>> Building pfSense kernel. >>> NO_BUILDKERNEL set, skipping build >>> Staging pfSense kernel... >>> Installing kernel (pfSense) for amd64 architecture... ====>> ERROR: SRCCONF is pointing to a nonexistent file /build/pfsense/tmp/FreeBSD-src/release/conf/pfSense_src.conf #################################### Something went wrong, check errors! ####################################
A second issue I am having is that no matter what I try, the jail build always fails for me on 2.7.2 when it gets to this:
--- secure/lib/libcrypto__L --- make[4]: make[4]: don't know how to make aes-586.S. Stop make[4]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src/secure/lib/libcrypto make[3]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src
At a loss on how to solve that one. It works fine on 2.7.1 though.
No, that's actually a problem with the copy of FreeBSD-src in the public repo. I had forgotten about that. There is a specific setting or optional argument to pass when building the jail. Let me consult my email history to see if I can find it.
Later Update: Here is the fix for that error. Add this to the file
/usr/local/etc/poudriere.d/src.conf
--WITHOUT_LIB32=y
-
Also, if you only want to build the poudriere ports tree, then run this command:
./build.sh --update-pkg-repo -a amd64.amd64
The changes I mentioned in posts above still are not likely to yield a successful kernel build. And if it builds, I'm not sure it will work correctly connecting to the official Netgate pfSense repositories for package installs and updates.
-
Getting there :)
The jail build now completes, although if I specify only the amd64 target, it goes much faster as it doesn't build other targets:
./build.sh --setup-poudriere -a amd64.amd64
Also, I needed to create this file, as it did not exist for me, but worked all the same:
/usr/local/etc/poudriere.d/src.conf
When I try and build the ports now I am right back to this error now, which probably means I have to set another variable or comment out something somewhere ;)
# ./build.sh --update-pkg-repo -c -a amd64.amd64 >>> Operation ./build.sh has started at Sat Feb 10 20:44:25 EST 2024 >>> Poudriere bulk started at 2024/02/10 20:44:25 for amd64.amd64 [00:00:00] Creating the reference jail... done [00:00:00] Mounting system devices for pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Warning: Using packages from previously failed, or uncommitted, build: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel/.building [00:00:00] Mounting ports from: /usr/local/poudriere/ports/pfSense_devel [00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Mounting distfiles from: /usr/ports/distfiles [00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/pfSense_devel-make.conf /etc/resolv.conf -> /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/etc/resolv.conf [00:00:00] Starting jail pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Will build as nobody:nobody (65534:65534) [00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s [00:00:01] Loading MOVED for /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/usr/ports [00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS [00:00:01] Inspecting ports tree for modifications to git checkout... yes [00:00:01] Ports top-level git hash: c3a0cffb7 (dirty) [00:00:01] Gathering ports metadata [00:00:01] Warning: MOVED: emulators/qemu-guest-agent renamed to emulators/qemu@guestagent [00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4 [00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4] [00:00:01] Error: Fatal errors encountered gathering initial ports metadata]** [pfSense_v2_7_2_amd64-pfSense_devel] [2024-02-10_20h44m25s] [crashed] Queued: 0 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 0 Time: 00:00:00 [00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s [00:00:01] Cleaning up [00:00:01] Unmounting file systems Exiting with status 1 >>> ERROR: Something went wrong...
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
Getting there :)
The jail build now completes, although if I specify only the amd64 target, it goes much faster as it doesn't build other targets:
./build.sh --setup-poudriere -a amd64.amd64
Also, I needed to create this file, as it did not exist for me, but worked all the same:
/usr/local/etc/poudriere.d/src.conf
When I try and build the ports now I am right back to this error now, which probably means I have to set another variable or comment out something somewhere ;)
# ./build.sh --update-pkg-repo -c -a amd64.amd64 >>> Operation ./build.sh has started at Sat Feb 10 20:44:25 EST 2024 >>> Poudriere bulk started at 2024/02/10 20:44:25 for amd64.amd64 [00:00:00] Creating the reference jail... done [00:00:00] Mounting system devices for pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Warning: Using packages from previously failed, or uncommitted, build: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel/.building [00:00:00] Mounting ports from: /usr/local/poudriere/ports/pfSense_devel [00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Mounting distfiles from: /usr/ports/distfiles [00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/pfSense_devel-make.conf /etc/resolv.conf -> /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/etc/resolv.conf [00:00:00] Starting jail pfSense_v2_7_2_amd64-pfSense_devel [00:00:00] Will build as nobody:nobody (65534:65534) [00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s [00:00:01] Loading MOVED for /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/usr/ports [00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS [00:00:01] Inspecting ports tree for modifications to git checkout... yes [00:00:01] Ports top-level git hash: c3a0cffb7 (dirty) [00:00:01] Gathering ports metadata [00:00:01] Warning: MOVED: emulators/qemu-guest-agent renamed to emulators/qemu@guestagent [00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4 [00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4] [00:00:01] Error: Fatal errors encountered gathering initial ports metadata]** [pfSense_v2_7_2_amd64-pfSense_devel] [2024-02-10_20h44m25s] [crashed] Queued: 0 Built: 0 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 Tobuild: 0 Time: 00:00:00 [00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s [00:00:01] Cleaning up [00:00:01] Unmounting file systems Exiting with status 1 >>> ERROR: Something went wrong...
Ah-- it's trying to build the DEVEL branch. That one I'm not sure works 100% yet. I am currently only building the RELENG_2_7_2 branch (which is the current CE Release branch).
You need to be sure your
build.conf
file is specifying the RELENG_2_7_2 branch. It defaults to the DEVEL branch. You need these lines in yourbuild.conf
file:# Define FreeBSD repository, branch and specific commit export FREEBSD_REPO_BASE=https://github.com/pfsense/freebsd-src.git export FREEBSD_BRANCH=RELENG_2_7_2 # Branch to replace pkg.conf template, defaults to $GIT_REPO_BRANCH_OR_TAG export PKG_REPO_BRANCH_DEVEL="RELENG_2_7_2" export PKG_REPO_BRANCH_RELEASE="RELENG_2_7_2"
There may also be some other changes required. I made a number of them the last time I reconstructed my package builder environment, and I forgot to write some of them down (witness that previous 32-bit library build command I posted).
I don't believe Netgate has actually tested a complete build from the available public repos. It fails in a number of ways without a lot of under-the-covers hacking to get it to work. And even then it only works partially, because at least two key packages are hosted on their private and proprietary GitLab repo and not on the public GitHub repo. Netgate has their own private build systems and environments for CE and Plus, and based on the current condition of the public GitHub available stuff, they must never attempt a build from that public stuff. Because it does not work out-of-the-box without a lot of tweaking and hacking. And I can somewhat understand why as they don't exactly want to make it super easy for anyone to build a pfSense clone (the whole trademark protection thing, which is justified).
-
@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.Any ideas what else I might add to get it to build against RELENG_2_7_2?
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.Any ideas what else I might add to get it to build against RELENG_2_7_2?
I seem to recall also having to temporarily change a line of code in the
builder_common.sh
script, but I don't recall exactly where. I may have actually temporarily hard-coded a varible with "RELENG_2_7_2". You will have to follow the function call tree of the "setup poudiere" command that starts in thebuild.sh
file. That file calls functions defined inbuilder_common.sh
. Another "gotcha" point is you need to wait until AFTER your Poudriere jail builds before setting the upstream repo in your FreeBSD-ports clone repo. Having the upstream pointing to the pfSense/FreeBSD-ports repo will cause a function that tries to auto-identify your local GitHub repo clone to fail.I spent many hours struggling through this when I had to rebuild my package builder system back in November and December of last year. And I could only get the RELENG_2_7_2 production builder to work. Never did get a DEVEL builder to work. In the past (from 2.7.0 and back) I was able to create both RELEASE and DEVEL builders.
This "broken" pfSense public builder system has proven to be very time consuming and frustrating for me as a volunteer package contributor/maintainer. I don't expect to- nor do I want to- spend hours and hours debugging a broken or partially functional builder creation ecosystem in order to be able to fully test deployment of my pfSense packages in a pfSense test machine that I then turn around and give back freely to Netgate and the pfSense community.
I want to be able to build my two IDS/IPS packages, copy them to a web server, and serve them via my private
pkg
repo so my pfSense test virtual machines can download and install packages from my repo. That way I can test the entire process exactly like it works for my users out in the real world. But starting with the introduction of thepfSense-repoc
setup, that no longer works. I can only install my packages now from the CLI, but for some reason usingpkg install my_package
does not trigger a subsequent "start" of the package as happens when installing from the Netgate repo. Thus, I can't know if my package will really auto-start for users upon an upgrade or not. I can go and start it manually after installation, but it won't auto-start at the end of the upgrade process anymore. And my packages absolutely will no longer install via the GUI as they used to. I can remove them via the GUI, but I can only install them now via the CLI usingpkg install xxxx
. That change I am attributing to the missingpfSense-repoc
package and the effect that has on thepkg
infrastructure within pfSense.I will say this -- I have not been able to get the build system on the pfSense GitHub repo to work out-of-the-box since probably all the way back to the 2.2 versions. Since then, it has always taken me a lot of debugging and rewriting of some of the scripts to make it successfully build a Poudriere jail that could build the pfSense FreeBSD-ports package tree. That debugging effort finally hit a brick wall with the release of 2.7.0 (I think it was, but maybe it was 2.7.1) and the
pfSense-repoc
package that I cannot build because the source is not posted on GitHub. -
@bmeeks
Zoiks. I feel your pain, or at least a small portion of it. I had no idea you were a volunteer. You deserve a raise! Seriously though, thanks for all your support.The platform seems to be going in a direction that isn't so much of a "community edition" anymore. Likely should just be rebranded as "free edition" to set expectations.
Given your experience, I don't think I am going to pursue this any further, other than to try and build the miniupnpd port with my changes directly in this folder which got created along the way (maybe during the jail build?):
cd /usr/local/poudriere/ports/pfSense_devel/net/miniupnpd make make package
Maybe that will work. I'll report back in a while once I've tested that approach. All of this was to fix those pesky IOCTL errors when compiling outside the jail.
-
Nope.
The package doesn't load on the firewall - missing a dependency on libpfctl. My gut (and past experience with 2.6.0) tells me that to build miniupnpd, you have to build the full pfSense enchilada.
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
Nope.
The package doesn't load on the firewall - missing a dependency on libpfctl. My gut (and past experience with 2.6.0) tells me that to build miniupnpd, you have to build the full pfSense enchilada.
You need to build it in an environment that at least duplicates fully the regular pfSense kernel and package builder system. Practically speaking that means you need a functioning build system created from the pfSense-supplied scripts. But as you and I have discovered, you can't create such a system from the scripts as they are default distributed. There are code issues in the various scripts that "break" the creation of the required builder environment.
The
libpfctl
dependency just recently came over from pfSense Plus into the CE tree. It is also going into upstream FreeBSD. It converts what was formerly a shared library distributed with the kernel into a separate FreeBSD ports package that can be updated outside of kernel updates. This makes it ultimately more flexible. -
@encrypt1d said in pfSense compile requirements for 3rd party software:
The platform seems to be going in a direction that isn't so much of a "community edition" anymore. Likely should just be rebranded as "free edition" to set expectations.
I don't disagree with you here. Building pfSense and/or the associated FreeBSD-ports packages tree is simply not possible with the currently posted open-source code on GitHub.
Some of the shell scripts used in the builder creation steps are faulty, and at least two critical packages required now in even the CE build are hidden behind the private Netgate GitLab account instead of being on the public GitHub repo. Those two packages are
security/pfSense
andsysutils/pfSense-repoc
. -
-
@encrypt1d, have you been able to compile miniupnpd under 2.7.2? I tried again this week, but no luck so far.
Using the default devel branch I can finish the poudriere jail, but can't compile packages due to missing
pfSense-pkg-zabbix-[agent4|proxy4]
pre-reqs. If I use the RELENG_2_7_2 branch, jail creation fails withmake[4]: don't know how to make aes-586.S. Stop
.I don't want to compile the kernel; just need to be able to compile a few packages.
This gave me a deep appreciation for every pfSense package maintainer out there. This is unbelievably painful.
-
@guiambros said in pfSense compile requirements for 3rd party software:
@encrypt1d, have you been able to compile miniupnpd under 2.7.2? I tried again this week, but no luck so far.
Using the default devel branch I can finish the poudriere jail, but can't compile packages due to missing
pfSense-pkg-zabbix-[agent4|proxy4]
pre-reqs. If I use the RELENG_2_7_2 branch, jail creation fails withmake[4]: don't know how to make aes-586.S. Stop
.I don't want to compile the kernel; just need to be able to compile a few packages.
This gave me a deep appreciation for every pfSense package maintainer out there. This is unbelievably painful.
Do this to work around that poudriere build error:
Edit
/usr/local/etc/poudriere.d/src.conf
in your builder machine and add the line "WITHOUT_LIB32=y". This will tell it not to try and build the 32-bit binaries (which it shouldn't be doing anyway because there is no longer a 32-bit build of pfSense CE). -
@bmeeks @guiambros
I threw in the towel and gave up after the revelation that the git repo is not up to date, nor are key pfSense dependencies being made available. The port I was trying to build (miniupnpd) has dependencies that are out of reach to the community. From my own perspective and needs, the CE version is dead.All I was after at this point was enhanced logging from miniupnpd so I tried asking the owners to improve it - but that hasn't gone anywhere yet.
https://github.com/miniupnp/miniupnp/issues/707Then I created a patch that can put miniupnpd in verbose mode, and one of the admins suggested that become permanent - so I opened a feature request for that:
https://redmine.pfsense.org/issues/15355
https://forum.netgate.com/post/1158289After that I had to rewrite the regex log decoders in my SIEM, so it's functional but not elegant.
-
@bmeeks said in pfSense compile requirements for 3rd party software:
Edit /usr/local/etc/poudriere.d/src.conf in your builder machine and add the line "WITHOUT_LIB32=y".
I don't have a src.conf, and
/usr/local/etc/poudriere.d
is virtually empty (just folder structure and two .sample files).I tried editing
/usr/local/etc/poudriere.conf
, but same error. Then I realized this conf file is being recreated every time bytools/builder_common.sh
, (line 1723). Tried adding the WITHOUT_LIB32 there, but also no success.Also tried adding an export WITHOUT_LIB32 to
build.conf
(in the hope that a child subprocess would inherit the variable), and also no luck.@bmeeks -- would appreciate if you have any other ideas, but I realize I already took a lot of your time in this seemingly pointless wild goose chase. I'm getting to the same conclusion as @encrypt1d: it seems Netgate made it (intentionally?) impossible to compile or do anything with CE.
-
Been casually watching this thread for months. Apologies if I'm misunderstanding, but I really was hoping for a different ending than Netgate made it impossible to compile or do anything with CE. Is that the end of the story here? I have some package ideas I've been working on, and this is discouraging me from continuing if at the end I'm going to face these impossible hurdles.
-
@guiambros @luckman212
I was able to build the jail using the WITHOUT_LIB32=Y option, but that is as far as I could go. It would be cool to see if you find a way forward. I don't want to discourage effort, but man, I just don't see how it would be possible without some rework on the git repo. I was an embedded systems dev for 15 years, so I know they CAN do it, they just haven't done it. -
@guiambros said in pfSense compile requirements for 3rd party software:
I don't have a src.conf, and /usr/local/etc/poudriere.d is virtually empty (just folder structure and two .sample files).
You must create that file if it is missing. Sorry I was not more clear on that point.