pfSense compile requirements for 3rd party software

Marc05

This is very handy to have, thank you very much!

guiambros

Just to say a heartfelt thank you to @encrypt1d @bmeeks @jimp for your efforts in documenting the process here. What an incredible arduous and cumbersome process, particularly for anyone who's not a core pfSense or package maintainer.

I too spent countless hours scratching my head trying to figure out why my manually-compiled miniupnpd wasn't working in prod (same IOCTL errors as @encrypt1d ), and eventually realized I'd have to build a pfSense dev environment in order to compile with the right libraries.

I finally found this thread, which helped me overcome the many twists and turns (including detours due to no ZFS, lack of disk space, recreating poudriere jails 4x, etc).

The thorniest issue for me (and in hindsight due to my own stupidity) was to ignore the "Jail is newer than host. (Jail: 1203500, Host: 1203000)". I thought it would be harmless -- c'mon, how different would 12.3.5 kernel be from 12.3.0? It turns out, they are indeed very different.

Once I upgraded the host to 12.4 (I didn't figure out how to upgrade to an intermediate 12.3.5 release), and cleaned / recreated the packages, everything finally started working.

I just got a fully functional miniupnpd binary running on 2.6.0. Still need to make the changes I want to make to source code, diff, etc, but much easier now that I have a working env.

note: I also got lucky, because half of my ports packages failed due to dependencies with devel/libffi-3.3, print/texinfo-6.8 and net/zabbix54-agent-5.4.7. But thankfully miniupnpd doesn't depend on any of them, so I decided to ignore and move on. YMMV.

THANK YOU!

guiambros

@guiambros said in pfSense compile requirements for 3rd party software:

I just got a fully functional miniupnpd binary running on 2.6.0. Still need to make the changes I want to make to source code, diff, etc, but much easier now that I have a working env.

Ugh, I spoke too soon. Its true that with FreeBSD 12.3.5 the resulting miniupnpd binary doesn't immediately throw ioctl() errors like before, but further tests indicated that new binary still rejects UPnP requests from clients.

@encrypt1d , I think your initial motivation was miniupnpd as well,no? Were you able to compile successfully? If yes, would you mind sharing a tarball of your resulting source files? Or at least config.h, so I can see which options you have enabled.

It's incredible how painful this is. I wonder if all package developers go through the same journey to set up their initial environment, or there's a docker/vm image somewhere that simplifies this.

encrypt1d

@guiambros Really glad to hear this helped.

This is actually a 2 part fix, depending on your ISP setup.

On release 2.6.0, you need to apply a patch first:
"Add UPnP NAT Anchors to fix outbound NAT for multiple consoles. (Redmine #7727, Forum Thread)".
You can do that from the "System" Menu under patches - but you might need to add the Patching package first from the Package Manager.

The Miniupnpd fix specifically allows for Double NAT scenarios to work (where your ISP gives you a private address like 10.x.x.x). If you aren't double NATed, just applying the patch should do it for you.

The folks that maintain the miniupnpd package have included the double NAT fix in 2.3.1. PFSense 2.7.0 is dropping very soon - I am hoping it will contain the right version of minupnpd. Maybe @jimp can confirm?

https://github.com/miniupnp/miniupnp/issues/598#issuecomment-1462959757

Now we get to do this all over again with 2.7.0 ;) I built custom logging into mine so I can get alerts when things punch holes in my firewall. UPNP is scary stuff. I will pretty much always run a custom build because of that.

guiambros

Thanks again!

@encrypt1d said in pfSense compile requirements for 3rd party software:

Now we get to do this all over again with 2.7.0 ;)

Ha, indeed, 2.7.0 just arrived a couple of days ago. The upgrade went smoothly, but I still want to customize my miniupnpd. In my case I don't have an issue with double NAT, but I want to fix an issue with log spam, and add some custom logging for security.

I built custom logging into mine so I can get alerts when things punch holes in my firewall. UPNP is scary stuff. I will pretty much always run a custom build because of that.

Any idea when RELENG_2._7_0 will be tagged on pfSense repo? I see recent commits in the master branch, but no indication of which commit was used for 2.7.0.

I guess we'll have to wait a few more days/weeks for github to catch up with Netgate's internal repo.

fabricioguzzy

@guiambros - I am also looking for the **"FreeBSD_ports RELENG_2_7_0 branch **" - Actually all the 2_7_0 branches are missing. (pfSense, FreeBSD and Ports)

guiambros

Quick update: I was finally able to successfully compile packages - and they work properly on 2.7.0! That's a first for me; all my previous attempts with 2.6.0 resulted in a binary that had all sorts of ioctl() errors.

The RELENG_2_7_0 branch does not yet exist on GitHub, but the master (pfSense) and devel-main (FreeBSD-src) branches seems to be close enough to what was just released, so most packages should be the same at this point. In my case I was only interested in miniupnpd package, so YMMV.

My step-by-step:

VM with FreeBSD 14.0-CURRENT
Follow the excellent steps above by @encrypt1d. Only pfSense source is needed; stay on the master branch
On build.conf, use export FREEBSD_BRANCH=devel-main
Create Poudriere jail, and build the ports
In my case I had 2 packages failed (missing original sources print/psutils and sysutils/pfSense-repoc) and 10 skipped due to dependencies.
Find the package directory (e.g. /usr/local/poudriere/ports/pfSense_devel/net/miniupnpd), use make extract to get the source code, modify the source as needed, and run make package to recompile. Package will be put on .../work/stage

After weeks of trial-and-error, this was the first time I was able to have a functional miniupnpd running in production.

I still want to grab the official RELENG_2_7_0 once released and recompile everything, but for now this is already a step in the right direction.

encrypt1d

@guiambros

I started down the path of trying to get a 2_7_2 build going recently. It seems that both labels RELENG2_7_1 and RELENG2_7_2 exist, but if I use RELENG2_7_2 the jail build fails with a compile failure on the AES-586 object, so I stuck with 2_7_1.

2_7_1 seems to create the jail ok, but then I run into build issues.

The variables in the build.conf file that control the repos used to be:

export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
export FREEBSD_BRANCH=RELENG_2_7_1
export PKG_REPO_SERVER_DEVEL="pkg+https://beta.pfsense.org/packages"
export PKG_REPO_SERVER_RELEASE="pkg+https://pkg.pfsense.org"
export PKG_REPO_SERVER_STAGING="pkg+https://pkg.pfsense.org"

These domains don't have IP address records anymore (or maybe never?).

What did you use for your build.conf?

When I ran my first build, it seems I am missing packages:

[00:00:11] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4
[00:00:11] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4

Trying to install these seems to use some default repos that don't exist:

# pkg update
Updating pfSense-core repository catalogue...
pkg: No SRV record found for the repo 'pfSense-core'
pkg: An error occured while fetching package
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-core/meta.txz -- pkg+:// implies SRV mirror type
repository pfSense-core has no meta file, using default settings
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-core/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-core/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: No SRV record found for the repo 'pfSense'
pkg: An error occured while fetching package
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-pfSense_%%VERSION%%/meta.txz -- pkg+:// implies SRV mirror type
repository pfSense has no meta file, using default settings
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-pfSense_%%VERSION%%/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg: packagesite URL error for pkg+http://release-staging.nyi.netgate.com/ce/packages/pfSense_%%OSVERSION%%_amd64-pfSense_%%VERSION%%/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository pfSense
Error updating repositories!

These repo domain names also don't exist. I think they are defined in builder_defaults.sh.

Would you mind sharing your build.conf file, or any changes to builder_defaults.sh?

Much appreciated.

encrypt1d

I thought I was on to something when I found these files in the pfsense repo that don't seem to be included when you do a clone on the RELENG2_7_1 branch:

pfSense-repo-devel.abi
pfSense-repo-devel.altabi
pfSense-repo-devel.conf
pfSense-repo-devel.descr
pfSense-repo-previous.abi
pfSense-repo-previous.altabi
pfSense-repo-previous.conf
pfSense-repo-previous.descr
pfSense-repo.abi
pfSense-repo.altabi
pfSense-repo.conf
pfSense-repo.descr

Putting those in ./tools/templates/pkg_repos didn't help though.

I watched my DNS queries on my old working 2_6_0 VM build, and they were going to pkg01-atx.netgate.com, so I tried this in my build.conf:

export PKG_REPO_SERVER_DEVEL="pkg+https://pkg01-atx.netgate.com/packages"
export PKG_REPO_SERVER_RELEASE="pkg+https://pkg01-atx.netgate.com"
export PKG_REPO_SERVER_STAGING="pkg+https://pkg01-atx.netgate.com"

No joy. On a fresh install, the pkg repos are ok, but after running ./build.sh --setup, the pkg repos are just broken.

Hoping @bmeeks or @guiambros know what works in terms of the repo list.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Hoping @bmeeks or @guiambros know what works in terms of the repo list.

I am reasonably sure you are just spinning wheels here on an impossible quest (that being building pfSense from the open-source repo on GitHub) -- at least since the last update to 2.7.2.

There is a new proprietary module called pfSense-repoc that handles all the repo-related stuff now on installation. It also handles the updates now in versions since at least 2.7.1. The source code for that package module is hosted on the private Netgate GitLab account, that's why I said it is proprietary because it is not available on the public GitHub. I strongly suspect that without that package in place, the repo stuff in your pfSense build is going to be broken.

There is also now another new pfSense package that needs to be built, and that package's source code is also hosted on the private Netgate GitLab and not on the public GitHub repo.

I don't try to build a pfSense kernel. All I build is the packages repo, but even for that now I have to go in and manually remove the runtime dependencies from the pfSense-upgrade package so that it does not try and pull in the proprietary pfSense-repoc package (which can't be built because its source code is in GitLab and not on GitHub). I also, starting with 2.7.2, have to comment out the build of the pfSense package in the poudriere.bulk file because that package's code is also on the private GitLab site.

encrypt1d

@bmeeks said in pfSense compile requirements for 3rd party software:

I don't try to build a pfSense kernel. All I build is the packages repo, but even for that now I have to go in and manually remove the runtime dependencies from the pfSense-upgrade package so that it does not try and pull in the proprietary pfSense-repoc package

This is all I am really trying to do as well. I had the instructions I wrote from 2_6_0 to go on, which obviously won't work now based on everything you have pointed out that has changed.

Do I simply clone FreeBSD-ports and try to build again without the jail? What might be the path forward?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Do I simply clone FreeBSD-ports and try to build again without the jail? What might be the path forward?

You can build the jail. You just can't build a pfSense kernel itself.

Clone both the FreeBSD-ports tree and the pfSense tree from the public GitHub repo.

I edited the pfsense/build.conf file to point to the RELENG_2_7_2 branch.

Then you need to comment out the lines pertaining to GNID in the builder_common.sh file so that the jail build does not try and pull down the proprietary Netgate ID source code files.

Change into the pfsense git clone directory.

Next, fetch the FreeBSD source files with: ./build.sh --update-sources.

You should now be able to create the poudriere jail with ./build.sh --setup-poudriere.

I then edited tools/conf/pfPorts/poudriere.bulk to comment out the pfSense package so that a build of it will not be attempted (which would fail during the fetch phase due to the GitLab host location). Here is the edit to poudriere.bulk:

#security/%%PRODUCT_NAME%%

Now go into your Poudriere ports tree (/usr/local/poudriere/ports/) and find the sysutils/pfSense-upgrade package. Edit its runtime depencies in the Makefile to comment out the pfSense-repoc package.

Packages should build now unless I've forgotten something off the top of my head.

But note the steps outlined above will NOT build a pfSense image. They only allow you to build the FreeBSD-ports packages repo tree.

encrypt1d

@bmeeks

Thanks, that was helpful. I assume I now need the "-c" option for build.sh so that it doesn't try to build the kernel, but yet it still seems to need it see "installing kernel" in the logs below.

# ./build.sh none -a amd64.amd64 -c
>>> Operation ./build.sh has started at Mon Feb 12 03:04:31 EST 2024
>>> Building image type(s):
>>> Cleaning up previous build environment...Please wait!
>>> Cleaning build directories: Done!
>>> Cleaning previously built kernel stage area...Done!
>>> Cleaning previously built images...Done!
>>> Cleaning previous builder logs...Done!
>>> Cleaning of builder environment has finished.
>>> NO_BUILDWORLD and NO_BUILDKERNEL set, skipping update of freebsd sources
>>> Last known commit Luiz Otavio O Souza - cf612ab9fc5711351fef5e0678d687aa3b88355d
>>> LOGFILE set to /build/pfsense/logs/buildworld.amd64.
>>> NO_BUILDWORLD set, skipping build
>>> Building pfSense kernel.
>>> NO_BUILDKERNEL set, skipping build
>>> Staging pfSense kernel...
>>> Installing kernel (pfSense) for amd64 architecture...
====>> ERROR: SRCCONF is pointing to a nonexistent file /build/pfsense/tmp/FreeBSD-src/release/conf/pfSense_src.conf

####################################
Something went wrong, check errors!
####################################

A second issue I am having is that no matter what I try, the jail build always fails for me on 2.7.2 when it gets to this:

--- secure/lib/libcrypto__L ---
make[4]: make[4]: don't know how to make aes-586.S. Stop
make[4]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src/secure/lib/libcrypto
make[3]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src

At a loss on how to solve that one. It works fine on 2.7.1 though.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Thanks, that was helpful. I assume I now need the "-c" option for build.sh so that it doesn't try to build the kernel, but yet it still seems to need it see "installing kernel" in the logs below.

# ./build.sh none -a amd64.amd64 -c
>>> Operation ./build.sh has started at Mon Feb 12 03:04:31 EST 2024
>>> Building image type(s):
>>> Cleaning up previous build environment...Please wait!
>>> Cleaning build directories: Done!
>>> Cleaning previously built kernel stage area...Done!
>>> Cleaning previously built images...Done!
>>> Cleaning previous builder logs...Done!
>>> Cleaning of builder environment has finished.
>>> NO_BUILDWORLD and NO_BUILDKERNEL set, skipping update of freebsd sources
>>> Last known commit Luiz Otavio O Souza - cf612ab9fc5711351fef5e0678d687aa3b88355d
>>> LOGFILE set to /build/pfsense/logs/buildworld.amd64.
>>> NO_BUILDWORLD set, skipping build
>>> Building pfSense kernel.
>>> NO_BUILDKERNEL set, skipping build
>>> Staging pfSense kernel...
>>> Installing kernel (pfSense) for amd64 architecture...
====>> ERROR: SRCCONF is pointing to a nonexistent file /build/pfsense/tmp/FreeBSD-src/release/conf/pfSense_src.conf

####################################
Something went wrong, check errors!
####################################

A second issue I am having is that no matter what I try, the jail build always fails for me on 2.7.2 when it gets to this:

--- secure/lib/libcrypto__L ---
make[4]: make[4]: don't know how to make aes-586.S. Stop
make[4]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src/secure/lib/libcrypto
make[3]: stopped in /usr/local/poudriere/jails/pfSense_v2_7_2_amd64/usr/src

At a loss on how to solve that one. It works fine on 2.7.1 though.

No, that's actually a problem with the copy of FreeBSD-src in the public repo. I had forgotten about that. There is a specific setting or optional argument to pass when building the jail. Let me consult my email history to see if I can find it.

Later Update: Here is the fix for that error. Add this to the file /usr/local/etc/poudriere.d/src.conf --

WITHOUT_LIB32=y

bmeeks

Also, if you only want to build the poudriere ports tree, then run this command:

./build.sh --update-pkg-repo -a amd64.amd64

The changes I mentioned in posts above still are not likely to yield a successful kernel build. And if it builds, I'm not sure it will work correctly connecting to the official Netgate pfSense repositories for package installs and updates.

encrypt1d

@bmeeks

Getting there :)

The jail build now completes, although if I specify only the amd64 target, it goes much faster as it doesn't build other targets:

./build.sh --setup-poudriere -a amd64.amd64

Also, I needed to create this file, as it did not exist for me, but worked all the same:

/usr/local/etc/poudriere.d/src.conf

When I try and build the ports now I am right back to this error now, which probably means I have to set another variable or comment out something somewhere ;)

# ./build.sh --update-pkg-repo -c -a amd64.amd64
>>> Operation ./build.sh has started at Sat Feb 10 20:44:25 EST 2024
>>> Poudriere bulk started at 2024/02/10 20:44:25 for amd64.amd64
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Warning: Using packages from previously failed, or uncommitted, build: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel/.building
[00:00:00] Mounting ports from: /usr/local/poudriere/ports/pfSense_devel
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Mounting distfiles from: /usr/ports/distfiles
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/pfSense_devel-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/etc/resolv.conf
[00:00:00] Starting jail pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Will build as nobody:nobody (65534:65534)
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Loading MOVED for /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/usr/ports
[00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:01] Inspecting ports tree for modifications to git checkout... yes
[00:00:01] Ports top-level git hash: c3a0cffb7 (dirty)
[00:00:01] Gathering ports metadata
[00:00:01] Warning: MOVED: emulators/qemu-guest-agent renamed to emulators/qemu@guestagent
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4]
[00:00:01] Error: Fatal errors encountered gathering initial ports metadata]**
[pfSense_v2_7_2_amd64-pfSense_devel] [2024-02-10_20h44m25s] [crashed] Queued: 0  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 0  Tobuild: 0   Time: 00:00:00
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Cleaning up
[00:00:01] Unmounting file systems
Exiting with status 1
>>> ERROR: Something went wrong...

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Getting there :)

The jail build now completes, although if I specify only the amd64 target, it goes much faster as it doesn't build other targets:

./build.sh --setup-poudriere -a amd64.amd64

Also, I needed to create this file, as it did not exist for me, but worked all the same:

/usr/local/etc/poudriere.d/src.conf

When I try and build the ports now I am right back to this error now, which probably means I have to set another variable or comment out something somewhere ;)

# ./build.sh --update-pkg-repo -c -a amd64.amd64
>>> Operation ./build.sh has started at Sat Feb 10 20:44:25 EST 2024
>>> Poudriere bulk started at 2024/02/10 20:44:25 for amd64.amd64
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Warning: Using packages from previously failed, or uncommitted, build: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel/.building
[00:00:00] Mounting ports from: /usr/local/poudriere/ports/pfSense_devel
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Mounting distfiles from: /usr/ports/distfiles
[00:00:00] Appending to make.conf: /usr/local/etc/poudriere.d/pfSense_devel-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/etc/resolv.conf
[00:00:00] Starting jail pfSense_v2_7_2_amd64-pfSense_devel
[00:00:00] Will build as nobody:nobody (65534:65534)
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Loading MOVED for /usr/local/poudriere/data/.m/pfSense_v2_7_2_amd64-pfSense_devel/ref/usr/ports
[00:00:01] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:01] Inspecting ports tree for modifications to git checkout... yes
[00:00:01] Ports top-level git hash: c3a0cffb7 (dirty)
[00:00:01] Gathering ports metadata
[00:00:01] Warning: MOVED: emulators/qemu-guest-agent renamed to emulators/qemu@guestagent
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-agent4
[00:00:01] Error: Nonexistent origin listed: net-mgmt/pfSense-pkg-zabbix-proxy4]
[00:00:01] Error: Fatal errors encountered gathering initial ports metadata]**
[pfSense_v2_7_2_amd64-pfSense_devel] [2024-02-10_20h44m25s] [crashed] Queued: 0  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 0  Tobuild: 0   Time: 00:00:00
[00:00:01] Logs: /usr/local/poudriere/data/logs/bulk/pfSense_v2_7_2_amd64-pfSense_devel/2024-02-10_20h44m25s
[00:00:01] Cleaning up
[00:00:01] Unmounting file systems
Exiting with status 1
>>> ERROR: Something went wrong...

Ah-- it's trying to build the DEVEL branch. That one I'm not sure works 100% yet. I am currently only building the RELENG_2_7_2 branch (which is the current CE Release branch).

You need to be sure your build.conf file is specifying the RELENG_2_7_2 branch. It defaults to the DEVEL branch. You need these lines in your build.conf file:

# Define FreeBSD repository, branch and specific commit
export FREEBSD_REPO_BASE=https://github.com/pfsense/freebsd-src.git
export FREEBSD_BRANCH=RELENG_2_7_2

# Branch to replace pkg.conf template, defaults to $GIT_REPO_BRANCH_OR_TAG
export PKG_REPO_BRANCH_DEVEL="RELENG_2_7_2"
export PKG_REPO_BRANCH_RELEASE="RELENG_2_7_2"

There may also be some other changes required. I made a number of them the last time I reconstructed my package builder environment, and I forgot to write some of them down (witness that previous 32-bit library build command I posted).

I don't believe Netgate has actually tested a complete build from the available public repos. It fails in a number of ways without a lot of under-the-covers hacking to get it to work. And even then it only works partially, because at least two key packages are hosted on their private and proprietary GitLab repo and not on the public GitHub repo. Netgate has their own private build systems and environments for CE and Plus, and based on the current condition of the public GitHub available stuff, they must never attempt a build from that public stuff. Because it does not work out-of-the-box without a lot of tweaking and hacking. And I can somewhat understand why as they don't exactly want to make it super easy for anyone to build a pfSense clone (the whole trademark protection thing, which is justified).

encrypt1d

@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.

Any ideas what else I might add to get it to build against RELENG_2_7_2?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
Unfortunately those extra export lines had no effect. I started from scratch to be sure. It still seems to want to be building the development branch. Thankfully the machine I am doing this on can build the jail in about 2 hours - sure beats the 11 hours it was taking me the last time I tried this.

Any ideas what else I might add to get it to build against RELENG_2_7_2?

I seem to recall also having to temporarily change a line of code in the builder_common.sh script, but I don't recall exactly where. I may have actually temporarily hard-coded a varible with "RELENG_2_7_2". You will have to follow the function call tree of the "setup poudiere" command that starts in the build.sh file. That file calls functions defined in builder_common.sh. Another "gotcha" point is you need to wait until AFTER your Poudriere jail builds before setting the upstream repo in your FreeBSD-ports clone repo. Having the upstream pointing to the pfSense/FreeBSD-ports repo will cause a function that tries to auto-identify your local GitHub repo clone to fail.

I spent many hours struggling through this when I had to rebuild my package builder system back in November and December of last year. And I could only get the RELENG_2_7_2 production builder to work. Never did get a DEVEL builder to work. In the past (from 2.7.0 and back) I was able to create both RELEASE and DEVEL builders.

This "broken" pfSense public builder system has proven to be very time consuming and frustrating for me as a volunteer package contributor/maintainer. I don't expect to- nor do I want to- spend hours and hours debugging a broken or partially functional builder creation ecosystem in order to be able to fully test deployment of my pfSense packages in a pfSense test machine that I then turn around and give back freely to Netgate and the pfSense community.

I want to be able to build my two IDS/IPS packages, copy them to a web server, and serve them via my private pkg repo so my pfSense test virtual machines can download and install packages from my repo. That way I can test the entire process exactly like it works for my users out in the real world. But starting with the introduction of the pfSense-repoc setup, that no longer works. I can only install my packages now from the CLI, but for some reason using pkg install my_package does not trigger a subsequent "start" of the package as happens when installing from the Netgate repo. Thus, I can't know if my package will really auto-start for users upon an upgrade or not. I can go and start it manually after installation, but it won't auto-start at the end of the upgrade process anymore. And my packages absolutely will no longer install via the GUI as they used to. I can remove them via the GUI, but I can only install them now via the CLI using pkg install xxxx. That change I am attributing to the missing pfSense-repoc package and the effect that has on the pkg infrastructure within pfSense.

I will say this -- I have not been able to get the build system on the pfSense GitHub repo to work out-of-the-box since probably all the way back to the 2.2 versions. Since then, it has always taken me a lot of debugging and rewriting of some of the scripts to make it successfully build a Poudriere jail that could build the pfSense FreeBSD-ports package tree. That debugging effort finally hit a brick wall with the release of 2.7.0 (I think it was, but maybe it was 2.7.1) and the pfSense-repoc package that I cannot build because the source is not posted on GitHub.

encrypt1d

@bmeeks
Zoiks. I feel your pain, or at least a small portion of it. I had no idea you were a volunteer. You deserve a raise! Seriously though, thanks for all your support.

The platform seems to be going in a direction that isn't so much of a "community edition" anymore. Likely should just be rebranded as "free edition" to set expectations.

Given your experience, I don't think I am going to pursue this any further, other than to try and build the miniupnpd port with my changes directly in this folder which got created along the way (maybe during the jail build?):

cd /usr/local/poudriere/ports/pfSense_devel/net/miniupnpd
make
make package

Maybe that will work. I'll report back in a while once I've tested that approach. All of this was to fix those pesky IOCTL errors when compiling outside the jail.