pfSense compile requirements for 3rd party software
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
I've narrowed the build down to just 4 errors now, almost there.
[00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch [00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch [00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch [00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetchThe logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?
In your specific use case, I don't think these failures matter. So just ignore them. So long as none of the impacted ports are dependencies of
miniupnpd, then their failure to build won't matter. I think you were interested in building justminiupnpd.Now at some point you will want to update your builder to 2.6.0 (RELENG_2_6_0 is the official branch name). But if the firewall you are testing on is at 2.5.2, then you want to keep your builder on the same version. The failures are likely the result of the older Ports tree which was based on FreeBSD-12.2.
FYI -- my RELEASE builder is just now finishing up the final packages for RELENG_2_6_0.
-
Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.
I think I can fix the other errors by placing the files it needs in /portdistfiles. The logs indicate it checks there after giving up on a direct download, but its good to know that doesn't matter.
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.
The packages are put in
/usr/local/poudriere/data/packages/{release}. So navigate to that directory and then in it will be symlinks. The symlink that contains the package files is named/All. -
@bmeeks said in pfSense compile requirements for 3rd party software:
files is named /All.
Thanks! I have a binary I can test now. :)
-
My favorite tool for connecting to my builders, browsing around there, and transferring files back and forth to my Windows PC is WinSCP.
-
My build worked, all those pesky ioctl errors are gone!
So now I am trying to change the source code, but not quite fully understanding this environment.
I figured I'd have to switch to this directory:
/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpdIn there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.
I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.
What's the trick here?
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
My build worked, all those pesky ioctl errors are gone!
So now I am trying to change the source code, but not quite fully understanding this environment.
I figured I'd have to switch to this directory:
/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpdIn there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.
I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.
What's the trick here?
Yea, I would not do the extraction in the build directory. That is a magic ZFS file system. I would instead use the native ports path of
/usr/ports/net/miniupnpd. Do all of your work there, and then produce the patch diff file. Copy that single diff file to the/filessubdirectory of the port on the builder.When changing the
Makefileversion, don't change the major or minor version. Instead, use the PORTREVISION tag. Here is an example from an old Suricata GUI package:PORTNAME= pfSense-pkg-suricata PORTVERSION= 6.0.3 PORTREVISION= 4If there is no PORTREVISION tag in your file, add it and start at 1 and increment by 1 for each build. That will produce a package file with an underscore on the end of the port name followed by the port revision. So this example
Makefileproduced a package named pfSense-pkg-suricata-6.0.3_4. -
@bmeeks
Thanks again so very much for all the help.
I think I am finally there. That last bit was the final hurdle. The build is fully clean of errors, and the app runs cleanly at runtime.I can see my own debug messages, and miniupnpd is working as well as it does with the package that ships with 2.5.2.
That was a process, but worth it. Thanks again to @jimp as well for helping out with the patching commands.
My progress on actually getting miniupnpd to work behind a double NAT will be over here:
https://forum.netgate.com/topic/169773/miniupnp-full-cone-double-natincorrectly-adding-rules/8?_=1644582288930
I'll make one additional post here soon, to capture the whole process, step by step. Just need to write/clean it up.
Cheers!
-
@bmeeks
What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE? -
@encrypt1d said in pfSense compile requirements for 3rd party software:
@bmeeks
What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE?I personally have never been too fond of IDEs. I did once work for a short time in one of the Microsoft tools when I was doing Windows-related development where I worked.
I started programming at the literal 1s and 0s of raw machine language, then graduated to assembly. Not much in the way of IDE for that
. I later moved on to C, C++, C#, a touch of Java, and then some PL/SQL while working with- and administering- some Oracle databases.These days I do only PHP for the GUI code in the Snort and Suricata packages, and then C for making customizations in the underlying binaries.
-
@encrypt1d said in pfSense compile requirements for 3rd party software:
What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE?
I use UltraEdit on Linux and Windows for most things. Not really as a full IDE, mostly as a code editor with lots of nice features. On systems where I don't have a license for that I use Kate, Notepad++, or TextMate.
-
Thanks.
Really just looking for something that speeds up GUI browsing on FreeBSD, not a full IDE. Tracing function calls is much easier that way.
I was a developer for the first 10 years of my career, and miss those tools (C on VXWorks with a proprietary IDE, then moved on to C++/Visual Studio, Java/Eclipse etc). They were multimillion line code repos, so grep just didn't cut it. I googled around bit, but it gets religious fast. -
One other thing I forgot to mention about the incremental building during development/troubleshooting is that I found it easier to do my initial work totally within the
/usr/ports/*tree and leave the Poudriere tree alone at first. This way I can run a quickmakein the/usr/ports/*directory to check for stupid C syntax errors or other coding oversights I may generate with my changes. You get immediate feedback of errors there via the console.When you run the
./build.shscript to fire off the build within the package builder, any compilation errors are buried in a corresponding log file down in the/usr/local/poudriere/tree. You can find the log, but it's a bit of effort.So I do my initial compile in the regular
/usr/ports/tree to make sure my code compiles successfully. Then, if it does, I will copy my patch diff over to the Poudriere tree and kick off the package builder so I get a package I can copy over to pfSense and actually execute. -
As promised, below is a full summary of everything I needed to do to get pfSense FreeBSD port builds working in an unofficial non-Netgate environment.
Why? My motivation was simple, and that was to develop a code fix to an issue within a package that pfSense uses, and has been broken since 2.4.5 for some configurations.
Note - only Netgate may produce official builds with the pfSense product ID. These steps are for debugging and exploratory purposes only.
Steps:
- Build a VM and install FreeBSD
- Install package dependencies
- Clone the git repos
- Edit the build.conf
- Edit the builder_common.sh
- Run the build setup
- Prepare the Poudriere environment
- Build ports
- Change code and build new package versions
Step 1) Build a VM
-
Use your favorite virtualization software to create a FreeBSD compatible VM with as many cores and as much RAM as you can throw at it. I used 4 CPUs, and 24 GB (eventually). My initial VM had 8GB which was insufficient.
-
Download the FreeBSD ISO corresponding to your pfSense revision, in this case 2.5.2 is build on FreeBSD 12.2 STABLE.
-
Boot the VM with the FreeBSD ISO you downloaded, and be sure to use ZFS for the filesystem! I selected to include the Ports Tree and the System Source Tree in my install. You cannot create a jail on a non-ZFS filesystem.
Step 2) Install package dependencies
You will need these packages to get started.
Running as root:pkg install git pkg install poudriere pkg install rsync pkg install screen pkg install nginxStep 3) Clone the git repos
Make a build folder, such as /build, change to it, and then start cloning.mkdir /build cd /build cd /build;git clone https://github.com/pfsense/pfsense.git cd /build/pfsense;git checkout RELENG_2_5_2 cd /build;git clone https://github.com/pfsense/FreeBSD-ports.git cd /build/FreeBSD-ports;git checkout RELENG_2_5_2You now have the following folders:
/build/pfsense /build/FreeBSD-ports4) Edit the build.conf
In the folder /build/pfsense:cp build.conf.sample build.confThen edit the build.conf file and ensure you use the following options set:
export PRODUCT_NAME="pfSense" export BUILD_AUTHORIZED_BY_NETGATE=yes export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git export FREEBSD_BRANCH="RELENG_2_5_2" export PKG_REPO_SERVER_DEVEL="pkg+https://beta.pfsense.org/packages" export PKG_REPO_SERVER_RELEASE="pkg+https://pkg.pfsense.org" export PKG_REPO_SERVER_STAGING="pkg+https://pkg.pfsense.org" export SKIP_FINAL_RSYNC=YES5) Edit the builder_common.sh script
This file is in:/build/pfsense/tools/builder_common.shYou want to make 2 changes, the first is to comment out the following:
# if [ "${PRODUCT_NAME}" = "pfSense" -a -n "${GNID_REPO_BASE}" ]; then # echo ">>> Obtaining gnid sources..." # ${BUILDER_SCRIPTS}/git_checkout.sh \ # -r ${GNID_REPO_BASE} \ # -d ${GNID_SRC_DIR} \ # -b ${GNID_BRANCH} # fiNext comment out this line:
#pkg install ${PRODUCT_NAME}-builder6) Run the build setup
cd /build/pfsense ./build.sh --setupIf anything fails, you'll have to determine the reason. Assuming the previous instructions have been followed everything should be fine.
7) Prepare the Poudriere environment
Now the really long command. This creates the Poudriere jail environment for building the ports. This took 11 hours on a core i7-11800H with an NVME SSD. Unfortunately it does not update the screen with any progress info.cd /build/pfsense ./build.sh --setup-poudriere8) Build ports
We're now ready to try building the port tree, which failed on the first go due to missing package dependencies for me.cd /build/pfsense ./build.sh --update-pkg-repo -a amd64.amd64I was missing these packages, you might be missing others:
sysutils/vmdktool emulators/qemu-user-static archivers/gtar textproc/xmlstarletThey were not available in my repo for a simple "pkg install" command, so I compiled and installed them from the ports tree we cloned earlier:
cd /build/FreeBSD-ports/sysutils/vmdktool/;make package pkg install /build/FreeBSD-ports/sysutils/vmdktool/work/pkg/vmdktool-1.4.pkg cd /build/FreeBSD-ports/emulators/qemu-user-static/;make package pkg install /build/FreeBSD-ports/emulators/qemu-user-static/work/pkg/qemu-user-static-3.1.0_12.pkg cd /build/FreeBSD-ports/archivers/gtar; make package pkg install /build/FreeBSD-ports/archivers/gtar/work/pkg/gtar-1.34.pkg cd /build/FreeBSD-ports/textproc/xmlstarlet; make package pkg install /build/FreeBSD-ports/textproc/xmlstarlet/work/pkg/xmlstarlet-1.6.1.pkgAfter that, the build ran but a number failed due to missing dist files it wasn't able to fetch. You can see exactly what it is trying to fetch from the repos in the build logs (one for each package) it points you to in the output. Mine was missing the following files:
sqlite-src-3350500.zip zabbix-5.2.6.tar.gz mysql-boost-5.7.34.tar.gz stunnel-5.59.tar.gzI found them on the internet and placed them in:
/usr/ports/distfilesAfter this, the build moved on. Then the "rust" package failed to build due to resource exhaustion (remember my original machine only had 8 GB RAM). I upped the VM RAM to 24 GB and then it passed. If you have less memory, keep trying - theoretically it should eventually finish.
Now the build ran to completion. It does fail on the signature step, but that's expected since we don't have the environment to sign the build (nor should we!).
9) Change code and build new package versions
Changing the code is a bit tricky, and one needs to be cautious on how it all works.
a) Extract the source
b) Copy it elsewhere and make your change
c) Copy it back
d) Make your patch
e) Copy your patch to the jail
f) Update your port revision to trigger a compile
g) Test your code on a pfSense non-production firewalla) Extract the Source
We will assume our package name is "foo"cd /build/FreeBSD-ports/foo make clean make extractThis places the source code in (the x's are a version number)
/build/FreeBSD-ports/foo/work/foo-x.x/b) Copy your code elsewhere
Find the source files you want to change and make 2 copies of each to somewhere outside of this folder (the "work" folder gets deleted each time you run "make clean" so you want to keep your changes safe and sound somewhere else).cp source1.c /tmp/source1.c cp source1.c /tmp/source1.c.orig cp source1.c /tmp/source1.h cp source1.c /tmp/source1.h.origThe .orig files are needed for patching later (do not change them), and the .c/.h files are where you make your changes. This step is only done once for each new file you change.
c) After editing, copy these files back
cp /tmp/source1.c /build/FreeBSD-ports/foo/work/foo-x.x/ cp /tmp/source1.c.orig /build/FreeBSD-ports/foo/work/foo-x.x/ cp /tmp/source1.h /build/FreeBSD-ports/foo/work/foo-x.x/ cp /tmp/source1.h.orig /build/FreeBSD-ports/foo/work/foo-x.x/d) Make the patch (or just do test compiles)
cd /build/FreeBSD-ports/foo/ make makepatch #Creates the diff file OR make package #test compileThe makepatch command creates patch (diff) files (with names like patch-source1.c) in
/build/FreeBSD-ports/foo/files/e) Copy the patches to the jail
Now in order to have the port build see the patch, copy them here. Your folder may be named differently depending on your release./usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpd/files/f) Edit the Makefile
Edit the Makefile for your port, for example:vi /usr/local/poudriere/ports/pfSense_v2_5_2/foo/MakefileChange the port revision, and increase it by 1, for example:
PORTREVISION=2g) Build your patched port
Run your build again, it should now create a new package for your patched codecd /build/pfsense ./build.sh --update-pkg-repo -a amd64.amd64If all went well, your package is in (or a similar folder):
/usr/local/poudriere/data/packages/pfSense_v2_5_2_amd64-pfSense_v2_5_2/All/Copy it to your test firewall (best not to test in production right?)
From an ssh shell on the firewall, you can replace the package with:pkg add -f foo.x.x.txzIf you need to revert to the original package from the distro:
pkg delete -f foo pkg install fooA note on building iterations of your package:
I found it was best to script the synchronization of the files I was changing, and the copying of the patch files, so I can start with a known baseline for every pass. The idea is this:make clean
make extract
copy my changed and .orig files over the freshly extracted code
remove old patch files from poudriere jail folder and the working port folder
make makepatch
copy new patch files to poudriere jail folder
Uprev the Port revision
BuildAnd that's it. Thanks again to @bmeeks and @jimp for helping me with this.
-
This is very handy to have, thank you very much!
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
B bmeeks referenced this topic on
-
Just to say a heartfelt thank you to @encrypt1d @bmeeks @jimp for your efforts in documenting the process here. What an incredible arduous and cumbersome process, particularly for anyone who's not a core pfSense or package maintainer.
I too spent countless hours scratching my head trying to figure out why my manually-compiled
miniupnpdwasn't working in prod (same IOCTL errors as @encrypt1d ), and eventually realized I'd have to build a pfSense dev environment in order to compile with the right libraries.I finally found this thread, which helped me overcome the many twists and turns (including detours due to no ZFS, lack of disk space, recreating poudriere jails 4x, etc).
The thorniest issue for me (and in hindsight due to my own stupidity) was to ignore the "Jail is newer than host. (Jail: 1203500, Host: 1203000)". I thought it would be harmless -- c'mon, how different would 12.3.5 kernel be from 12.3.0? It turns out, they are indeed very different.
Once I upgraded the host to 12.4 (I didn't figure out how to upgrade to an intermediate 12.3.5 release), and cleaned / recreated the packages, everything finally started working.
I just got a fully functional
miniupnpdbinary running on 2.6.0. Still need to make the changes I want to make to source code, diff, etc, but much easier now that I have a working env.note: I also got lucky, because half of my ports packages failed due to dependencies with devel/libffi-3.3, print/texinfo-6.8 and net/zabbix54-agent-5.4.7. But thankfully miniupnpd doesn't depend on any of them, so I decided to ignore and move on. YMMV.
THANK YOU!
-
@guiambros said in pfSense compile requirements for 3rd party software:
I just got a fully functional miniupnpd binary running on 2.6.0. Still need to make the changes I want to make to source code, diff, etc, but much easier now that I have a working env.
Ugh, I spoke too soon. Its true that with FreeBSD 12.3.5 the resulting
miniupnpdbinary doesn't immediately throw ioctl() errors like before, but further tests indicated that new binary still rejects UPnP requests from clients.@encrypt1d , I think your initial motivation was miniupnpd as well,no? Were you able to compile successfully? If yes, would you mind sharing a tarball of your resulting source files? Or at least
config.h, so I can see which options you have enabled.It's incredible how painful this is. I wonder if all package developers go through the same journey to set up their initial environment, or there's a docker/vm image somewhere that simplifies this.
-
@guiambros Really glad to hear this helped.
This is actually a 2 part fix, depending on your ISP setup.
On release 2.6.0, you need to apply a patch first:
"Add UPnP NAT Anchors to fix outbound NAT for multiple consoles. (Redmine #7727, Forum Thread)".
You can do that from the "System" Menu under patches - but you might need to add the Patching package first from the Package Manager.The Miniupnpd fix specifically allows for Double NAT scenarios to work (where your ISP gives you a private address like 10.x.x.x). If you aren't double NATed, just applying the patch should do it for you.
The folks that maintain the miniupnpd package have included the double NAT fix in 2.3.1. PFSense 2.7.0 is dropping very soon - I am hoping it will contain the right version of minupnpd. Maybe @jimp can confirm?
https://github.com/miniupnp/miniupnp/issues/598#issuecomment-1462959757
Now we get to do this all over again with 2.7.0 ;) I built custom logging into mine so I can get alerts when things punch holes in my firewall. UPNP is scary stuff. I will pretty much always run a custom build because of that.
-
Thanks again!
@encrypt1d said in pfSense compile requirements for 3rd party software:
Now we get to do this all over again with 2.7.0 ;)
Ha, indeed, 2.7.0 just arrived a couple of days ago. The upgrade went smoothly, but I still want to customize my miniupnpd. In my case I don't have an issue with double NAT, but I want to fix an issue with log spam, and add some custom logging for security.
I built custom logging into mine so I can get alerts when things punch holes in my firewall. UPNP is scary stuff. I will pretty much always run a custom build because of that.
Any idea when RELENG_2._7_0 will be tagged on pfSense repo? I see recent commits in the master branch, but no indication of which commit was used for 2.7.0.
I guess we'll have to wait a few more days/weeks for github to catch up with Netgate's internal repo.
-
@guiambros - I am also looking for the **"FreeBSD_ports RELENG_2_7_0 branch **" - Actually all the 2_7_0 branches are missing. (pfSense, FreeBSD and Ports)
