pfSense compile requirements for 3rd party software

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Getting there :)

The jail build now completes, although if I specify only the amd64 target, it goes much faster as it doesn't build other targets:

./build.sh --setup-poudriere -a amd64.amd64

Also, I needed to create this file, as it did not exist for me, but worked all the same:

/usr/local/etc/poudriere.d/src.conf

When I try and build the ports now I am right back to this error now, which probably means I have to set another variable or comment out something somewhere ;)

# ./build.sh --update-pkg-repo -c -a amd64.amd64
>>> Operation ./build.sh has started at Sat Feb 10 20:44:25 EST 2024
>>> Poudriere bulk started at 2024/02/10 20:44:25 for amd64.amd64
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Warning: Using packages from previously failed, or uncommitted, build: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel/.building
[00:00:00] Mounting ports from: /usr/local/poudriere/ports/pfSense_devel
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Mounting distfiles from: /usr/ports/distfiles
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/pfSense_devel-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/etc/resolv.conf
[00:00:00] Starting jail pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Will build as nobody:nobody (65534:65534)
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Loading MOVED for /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/usr/ports
[00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:01] Inspecting ports tree for modifications to git checkout... yes
[00:00:01] Ports top-level git hash: c3a0cffb7 (dirty)
[00:00:01] Gathering ports metadata
[00:00:01] Warning: MOVED: emulators/qemu-guest-agent renamed to emulators/qemu@guestagent
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4]
[00:00:01] Error: Fatal errors encountered gathering initial ports metadata]**
[pfSense_v2_7_2_amd64-pfSense_devel] [2024-02-10_20h44m25s] [crashed] Queued: 0  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 0  Tobuild: 0   Time: 00:00:00
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Cleaning up
[00:00:01] Unmounting file systems
Exiting with status 1
>>> ERROR: Something went wrong...

Ah-- it's trying to build the DEVEL branch. That one I'm not sure works 100% yet. I am currently only building the RELENG_2_7_2 branch (which is the current CE Release branch).

You need to be sure your build.conf file is specifying the RELENG_2_7_2 branch. It defaults to the DEVEL branch. You need these lines in your build.conf file:

# Define FreeBSD repository, branch and specific commit
export FREEBSD_REPO_BASE=https://github.com/pfsense/freebsd-src.git
export FREEBSD_BRANCH=RELENG_2_7_2

# Branch to replace pkg.conf template, defaults to $GIT_REPO_BRANCH_OR_TAG
export PKG_REPO_BRANCH_DEVEL="RELENG_2_7_2"
export PKG_REPO_BRANCH_RELEASE="RELENG_2_7_2"

There may also be some other changes required. I made a number of them the last time I reconstructed my package builder environment, and I forgot to write some of them down (witness that previous 32-bit library build command I posted).

I don't believe Netgate has actually tested a complete build from the available public repos. It fails in a number of ways without a lot of under-the-covers hacking to get it to work. And even then it only works partially, because at least two key packages are hosted on their private and proprietary GitLab repo and not on the public GitHub repo. Netgate has their own private build systems and environments for CE and Plus, and based on the current condition of the public GitHub available stuff, they must never attempt a build from that public stuff. Because it does not work out-of-the-box without a lot of tweaking and hacking. And I can somewhat understand why as they don't exactly want to make it super easy for anyone to build a pfSense clone (the whole trademark protection thing, which is justified).

encrypt1d

@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.

Any ideas what else I might add to get it to build against RELENG_2_7_2?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.

Any ideas what else I might add to get it to build against RELENG_2_7_2?

I seem to recall also having to temporarily change a line of code in the builder_common.sh script, but I don't recall exactly where. I may have actually temporarily hard-coded a varible with "RELENG_2_7_2". You will have to follow the function call tree of the "setup poudiere" command that starts in the build.sh file. That file calls functions defined in builder_common.sh. Another "gotcha" point is you need to wait until AFTER your Poudriere jail builds before setting the upstream repo in your FreeBSD-ports clone repo. Having the upstream pointing to the pfSense/FreeBSD-ports repo will cause a function that tries to auto-identify your local GitHub repo clone to fail.

I spent many hours struggling through this when I had to rebuild my package builder system back in November and December of last year. And I could only get the RELENG_2_7_2 production builder to work. Never did get a DEVEL builder to work. In the past (from 2.7.0 and back) I was able to create both RELEASE and DEVEL builders.

This "broken" pfSense public builder system has proven to be very time consuming and frustrating for me as a volunteer package contributor/maintainer. I don't expect to- nor do I want to- spend hours and hours debugging a broken or partially functional builder creation ecosystem in order to be able to fully test deployment of my pfSense packages in a pfSense test machine that I then turn around and give back freely to Netgate and the pfSense community.

I want to be able to build my two IDS/IPS packages, copy them to a web server, and serve them via my private pkg repo so my pfSense test virtual machines can download and install packages from my repo. That way I can test the entire process exactly like it works for my users out in the real world. But starting with the introduction of the pfSense-repoc setup, that no longer works. I can only install my packages now from the CLI, but for some reason using pkg install my_package does not trigger a subsequent "start" of the package as happens when installing from the Netgate repo. Thus, I can't know if my package will really auto-start for users upon an upgrade or not. I can go and start it manually after installation, but it won't auto-start at the end of the upgrade process anymore. And my packages absolutely will no longer install via the GUI as they used to. I can remove them via the GUI, but I can only install them now via the CLI using pkg install xxxx. That change I am attributing to the missing pfSense-repoc package and the effect that has on the pkg infrastructure within pfSense.

I will say this -- I have not been able to get the build system on the pfSense GitHub repo to work out-of-the-box since probably all the way back to the 2.2 versions. Since then, it has always taken me a lot of debugging and rewriting of some of the scripts to make it successfully build a Poudriere jail that could build the pfSense FreeBSD-ports package tree. That debugging effort finally hit a brick wall with the release of 2.7.0 (I think it was, but maybe it was 2.7.1) and the pfSense-repoc package that I cannot build because the source is not posted on GitHub.

encrypt1d

@bmeeks
Zoiks. I feel your pain, or at least a small portion of it. I had no idea you were a volunteer. You deserve a raise! Seriously though, thanks for all your support.

The platform seems to be going in a direction that isn't so much of a "community edition" anymore. Likely should just be rebranded as "free edition" to set expectations.

Given your experience, I don't think I am going to pursue this any further, other than to try and build the miniupnpd port with my changes directly in this folder which got created along the way (maybe during the jail build?):

cd /usr/local/poudriere/ports/pfSense_devel/net/miniupnpd
make
make package

Maybe that will work. I'll report back in a while once I've tested that approach. All of this was to fix those pesky IOCTL errors when compiling outside the jail.

encrypt1d

Nope.

The package doesn't load on the firewall - missing a dependency on libpfctl. My gut (and past experience with 2.6.0) tells me that to build miniupnpd, you have to build the full pfSense enchilada.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Nope.

The package doesn't load on the firewall - missing a dependency on libpfctl. My gut (and past experience with 2.6.0) tells me that to build miniupnpd, you have to build the full pfSense enchilada.

You need to build it in an environment that at least duplicates fully the regular pfSense kernel and package builder system. Practically speaking that means you need a functioning build system created from the pfSense-supplied scripts. But as you and I have discovered, you can't create such a system from the scripts as they are default distributed. There are code issues in the various scripts that "break" the creation of the required builder environment.

The libpfctl dependency just recently came over from pfSense Plus into the CE tree. It is also going into upstream FreeBSD. It converts what was formerly a shared library distributed with the kernel into a separate FreeBSD ports package that can be updated outside of kernel updates. This makes it ultimately more flexible.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

The platform seems to be going in a direction that isn't so much of a "community edition" anymore. Likely should just be rebranded as "free edition" to set expectations.

I don't disagree with you here. Building pfSense and/or the associated FreeBSD-ports packages tree is simply not possible with the currently posted open-source code on GitHub.

Some of the shell scripts used in the builder creation steps are faulty, and at least two critical packages required now in even the CE build are hidden behind the private Netgate GitLab account instead of being on the public GitHub repo. Those two packages are security/pfSense and sysutils/pfSense-repoc.

guiambros

@encrypt1d, have you been able to compile miniupnpd under 2.7.2? I tried again this week, but no luck so far.

Using the default devel branch I can finish the poudriere jail, but can't compile packages due to missing pfSense-pkg-zabbix-[agent4|proxy4] pre-reqs. If I use the RELENG_2_7_2 branch, jail creation fails with make[4]: don't know how to make aes-586.S. Stop.

I don't want to compile the kernel; just need to be able to compile a few packages.

This gave me a deep appreciation for every pfSense package maintainer out there. This is unbelievably painful.

bmeeks

@guiambros said in pfSense compile requirements for 3rd party software:

@encrypt1d, have you been able to compile miniupnpd under 2.7.2? I tried again this week, but no luck so far.

Using the default devel branch I can finish the poudriere jail, but can't compile packages due to missing pfSense-pkg-zabbix-[agent4|proxy4] pre-reqs. If I use the RELENG_2_7_2 branch, jail creation fails with make[4]: don't know how to make aes-586.S. Stop.

I don't want to compile the kernel; just need to be able to compile a few packages.

This gave me a deep appreciation for every pfSense package maintainer out there. This is unbelievably painful.

Do this to work around that poudriere build error:

Edit /usr/local/etc/poudriere.d/src.conf in your builder machine and add the line "WITHOUT_LIB32=y". This will tell it not to try and build the 32-bit binaries (which it shouldn't be doing anyway because there is no longer a 32-bit build of pfSense CE).

encrypt1d

@bmeeks @guiambros
I threw in the towel and gave up after the revelation that the git repo is not up to date, nor are key pfSense dependencies being made available. The port I was trying to build (miniupnpd) has dependencies that are out of reach to the community. From my own perspective and needs, the CE version is dead.

All I was after at this point was enhanced logging from miniupnpd so I tried asking the owners to improve it - but that hasn't gone anywhere yet.
https://github.com/miniupnp/miniupnp/issues/707

Then I created a patch that can put miniupnpd in verbose mode, and one of the admins suggested that become permanent - so I opened a feature request for that:

https://redmine.pfsense.org/issues/15355
https://forum.netgate.com/post/1158289

After that I had to rewrite the regex log decoders in my SIEM, so it's functional but not elegant.

guiambros

@bmeeks said in pfSense compile requirements for 3rd party software:

Edit /usr/local/etc/poudriere.d/src.conf in your builder machine and add the line "WITHOUT_LIB32=y".

I don't have a src.conf, and /usr/local/etc/poudriere.d is virtually empty (just folder structure and two .sample files).

I tried editing /usr/local/etc/poudriere.conf, but same error. Then I realized this conf file is being recreated every time by tools/builder_common.sh, (line 1723). Tried adding the WITHOUT_LIB32 there, but also no success.

Also tried adding an export WITHOUT_LIB32 to build.conf (in the hope that a child subprocess would inherit the variable), and also no luck.

@bmeeks -- would appreciate if you have any other ideas, but I realize I already took a lot of your time in this seemingly pointless wild goose chase. I'm getting to the same conclusion as @encrypt1d: it seems Netgate made it (intentionally?) impossible to compile or do anything with CE.

luckman212

Been casually watching this thread for months. Apologies if I'm misunderstanding, but I really was hoping for a different ending than Netgate made it impossible to compile or do anything with CE. Is that the end of the story here? I have some package ideas I've been working on, and this is discouraging me from continuing if at the end I'm going to face these impossible hurdles.

encrypt1d

@guiambros @luckman212
I was able to build the jail using the WITHOUT_LIB32=Y option, but that is as far as I could go. It would be cool to see if you find a way forward. I don't want to discourage effort, but man, I just don't see how it would be possible without some rework on the git repo. I was an embedded systems dev for 15 years, so I know they CAN do it, they just haven't done it.

bmeeks

@guiambros said in pfSense compile requirements for 3rd party software:

I don't have a src.conf, and /usr/local/etc/poudriere.d is virtually empty (just folder structure and two .sample files).

You must create that file if it is missing. Sorry I was not more clear on that point.

bmeeks

The state of the CE builder is "poorly maintained" to put it as kindly as I can .

I will not ascribe any ulterior motives as to why that is the case, but one possibility is that developer hours invested in CE don't pay the bills the same as developer hours invested in pfSense Plus. There are currently several issues that need correcting in the CE builder tree. These include package dependencies that reference proprietary GitLab repo locations for their source code, configuration problems within the RELENG_2_7_X builder tree (that 32-bit library problem being one glaring one), and little gotchas scattered around the scripting functions in builder_common.sh.

I wish the builder would be split out into two separate configurations: (1) one setup script for creating a kernel builder, which those of us in this thread- and most others- would never use; and (2) a separate setup script that creates a package builder ecosystem for those of us who maintain the third-party packages and only want to build the packages and copy them into a private testing repo so that we can test the full pfSense package installation process (rather than having to simply install the package as a local file from the CLI).

Right now it is impossible to run the builder setup script for the RELENG_2_7_2 CE branch and have it produce a functioning builder subsystem. Nor is it possible, after you futz around and finally hack the builder script so that it creates a poudriere jail package builder, to actually build the pfSense packages repo unless you manually go in and comment out some runtime dependencies for packages whose source code is behind the private GitLab location.

guiambros

Good news: I was finally able to compile all packages!

Only 1 package failed (pfSense-repoc, which the source is now proprietary), and 2 skipped (pfSense@php82, pfSense-upgrade); 547 compiled successfully.

After sorting out the issue with LIB32 (thanks again @bmeeks!), I kept bumping into an issue with pfSense-pkg-zabbix-[proxy|agent]4, that didn't exist in the Ports repo.

I tried moving to FreeBSD 15.0.dev, which solved the zabbix dependencies, but then the compiled packages didn't run properly in pfSense 2.7.2 (not surprisingly; the packages compiled in 15.0 linked to libraries that didn't exist in 14.0).

After enabling debug (set -x), I finally spotted the problem: the Poudriere jail kept defaulting to devel branch, so it was using zabbix packages that didn't exist in 14.0. Checking the Ports branch downloaded, I confirmed it was devel, and not 2.7.2

The solution was to add POUDRIERE_PORTS_GIT_BRANCH in build.conf. It then changed to the right Ports branch, and properly, and finally the packages as expected.

I was able to change, compile and move miniupnpd to 2.7.2 production, and has been running stable for the last 24h.

Here's my build.conf:

export PRODUCT_NAME="pfSense"
export PRODUCT_URL="https://pfsense.org/"
export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
export FREEBSD_BRANCH=RELENG_2_7_2
export DEFAULT_ARCH_LIST="amd64.amd64"
export PKG_REPO_BRANCH_DEVEL="RELENG_2_7_2"
export PKG_REPO_BRANCH_RELEASE="RELENG_2_7_2"
export POUDRIERE_PORTS_GIT_BRANCH="RELENG_2_7_2"

Also adjusted tools/builder_defaults.sh to:

PKG_REPO_BRANCH_DEVEL="v2_7_2"
PKG_REPO_BRANCH_RELEASE="v2_7_2"
PKG_REPO_BRANCH_PREVIOUS="v2_7_1"

All the other instructions above are still valid: e.g., comment out gnid in tools/builder_common.sh, create the file /usr/local/etc/poudriere.d/src.conf with WITHOUT_LIB32=y, etc.

@encrypt1d -- I remember you were also trying to compile miniupnpd. I've been running my modified version, and seems pretty stable so far. In my case I just wanted to fix the logging, removing some useless interface mismatch log spam, and flip some messages from LOG_INFO to LOG_WARNING (so I can audit them if necessary).

This time I took detailed instructions so I can reproduce the build, so lmk if you still have problems compiling it.

Thanks for all the help folks!

ps: of course, all this will be wasted effort once Netgate upgrades to 2.7.3+.