pfSense compile requirements for 3rd party software

encrypt1d

Looking good so far. Now I wait ;)

root@devbox:/build/pfsense # ./build.sh --setup
>>> Operation ./build.sh has started at Tue Feb 15 10:11:28 EST 2022
>>> Obtaining FreeBSD sources (RELENG_2_5_2)...
====>> Cloning git repository https://github.com/pfsense/FreeBSD-src.git (RELENG_2_5_2)
Skipping clone of https://github.com/pfsense/gnid.git
Skipping install of pfSense-builder
>>> Operation ./build.sh has ended at Tue Feb 15 10:25:52 EST 2022
root@devbox:/build/pfsense # ./build.sh --setup-poudriere
>>> Operation ./build.sh has started at Tue Feb 15 10:27:16 EST 2022
>>> Creating ZFS filesystem zroot/poudriere... Done!
>>> Creating poudriere.conf
>>> Creating jail pfSense_v2_5_2_amd64, it may take some time...

encrypt1d

@bmeeks
While that is running, what are the next steps beyond the jail setup in order to build a specific port package?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
While that is running, what are the next steps beyond the jail setup in order to build a specific port package?

I've never found the magic word for building a specific port. That's because the Netgate shell script is geared toward building the whole repository.

However, once you build the first round (which will take quite some time, especially the Rust port), then subsequent rounds will only build the packages that have been changed. Therefore, so long as you don't "refresh" your local Poudriere ports tree, none of the other packages will need to be rebuilt. You can just change your miniupnpd package and then when you run this:

./build.sh --update-pkg-repo -a amd64.amd64

it will only rebuild miniupnpd if that's the only package with a change. For your troubleshooting steps, assuming you will want to rebuild your package several times, you will need to delete the package from /usr/local/poudriere/data/packages/.. so Poudriere will see it missing and build it again. Either that, or in the Makefile for miniupnpd you can increment the version number to signal Poudriere to rebuild the port. In your case bumping the version number in the Makefile might be the fastest method.

Then when you have it working like you want, you can change the version number back to maybe just one past the current one in pfSense, delete the package from the path I mentioned above, then build it one last time.

encrypt1d

@bmeeks

I will still be able to use the patching technique @jimp suggested? I.e make extract, change files, make patch?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

I will still be able to use the patching technique @jimp suggested? I.e make extract, change files, make patch?

Yes, that technique should still work to create the patch file or files. One convention commonly used is to consolidate all the changes for a given bug fix or feature add into a single patch file. Then you name the patch file something that aligns with what the patch does. So if you are adding a new feature or fixing some specific bug, then put all the patched files into a single *.diff patch and name it something that reflects the new feature added or the bug fixed.

encrypt1d

@bmeeks
The jail setup completed successfully. Apparently your machine is faster than my virtual machine, ouch. 11 hrs.

root@devbox:/build/pfsense # ./build.sh --setup-poudriere
>>> Operation ./build.sh has started at Tue Feb 15 10:27:16 EST 2022
>>> Creating ZFS filesystem zroot/poudriere... Done!
>>> Creating poudriere.conf
>>> Creating jail pfSense_v2_5_2_amd64, it may take some time... 
Done!
>>> Creating poudriere ports tree, it may take some time... Done!
>>> Poudriere is now configured!
>>> Operation ./build.sh has ended at Tue Feb 15 21:16:43 EST 2022

Is my next step to run this?

./build.sh none

I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Is my next step to run this?
./build.sh none
I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

I run this command to build the package tree:

./build.sh --update-pkg-repo -a amd64.amd64

That will build all the packages for AMD64/Intel architectures. You can just execute the shell script with no arguments to see all the available options like this:

./build.sh

I just rebuilt the jail in my RELEASE builder since I had to update it to the latest FreeBSD 12.3 with the recent 2.6.0 pfSense release, and it took about as long as yours. Much longer than I remembered from the past.

encrypt1d

@bmeeks

Thanks again, it appears to be running.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Thanks again, it appears to be running.

Some of the packages will take a long time to build, particularly the Rust language one. Just be patient. On my builder VM, Rust takes over 4 hours by itself. And it needs lots of RAM (more than 8 GB). What I did was create a very large extra swap partition for it to use during that build. Makes it take longer, but my ESXi host only has 32 GB of RAM to share among my active VMs.

If Rust fails to build, it will probably be due to resource limits. If that happens, simply kick off the same build again (use the same command) and the process will pick up where it left off. It is smart enough not to rebuild everything from scratch every time.

So if you see any package fail during the process, just let it keep going until the job stops. Then you can run it again to rebuild any failures. Like I said, most times the failures are due to resource exhaustion in my minimal builder VM.

encrypt1d

@bmeeks

That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

Possibly, or at least a very long first day. After that, assuming you don't change any of the other ports' source code, each time you kick off a build it will only build your miniupnpd package (if you change the Makefile version) and a couple of small pfSense packages that are rebuilt on each run (these have timestamp info for the current build).

encrypt1d

@bmeeks

I've narrowed the build down to just 4 errors now, almost there.

[00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch
[00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch
[00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch
[00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetch

The logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

I've narrowed the build down to just 4 errors now, almost there.
[00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch
[00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch
[00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch
[00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetch
The logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?

In your specific use case, I don't think these failures matter. So just ignore them. So long as none of the impacted ports are dependencies of miniupnpd, then their failure to build won't matter. I think you were interested in building just miniupnpd.

Now at some point you will want to update your builder to 2.6.0 (RELENG_2_6_0 is the official branch name). But if the firewall you are testing on is at 2.5.2, then you want to keep your builder on the same version. The failures are likely the result of the older Ports tree which was based on FreeBSD-12.2.

FYI -- my RELEASE builder is just now finishing up the final packages for RELENG_2_6_0.

encrypt1d

@bmeeks

Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.

I think I can fix the other errors by placing the files it needs in /portdistfiles. The logs indicate it checks there after giving up on a direct download, but its good to know that doesn't matter.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.

The packages are put in /usr/local/poudriere/data/packages/{release}. So navigate to that directory and then in it will be symlinks. The symlink that contains the package files is named /All.

encrypt1d

@bmeeks said in pfSense compile requirements for 3rd party software:

files is named /All.

Thanks! I have a binary I can test now. :)

bmeeks

My favorite tool for connecting to my builders, browsing around there, and transferring files back and forth to my Windows PC is WinSCP.

encrypt1d

@bmeeks

My build worked, all those pesky ioctl errors are gone!

So now I am trying to change the source code, but not quite fully understanding this environment.

I figured I'd have to switch to this directory:

/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpd

In there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.

I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.

What's the trick here?

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

My build worked, all those pesky ioctl errors are gone!

So now I am trying to change the source code, but not quite fully understanding this environment.

I figured I'd have to switch to this directory:
/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpd
In there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.

I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.

What's the trick here?

Yea, I would not do the extraction in the build directory. That is a magic ZFS file system. I would instead use the native ports path of /usr/ports/net/miniupnpd. Do all of your work there, and then produce the patch diff file. Copy that single diff file to the /files subdirectory of the port on the builder.

When changing the Makefile version, don't change the major or minor version. Instead, use the PORTREVISION tag. Here is an example from an old Suricata GUI package:

PORTNAME=	pfSense-pkg-suricata
PORTVERSION=	6.0.3
PORTREVISION=	4

If there is no PORTREVISION tag in your file, add it and start at 1 and increment by 1 for each build. That will produce a package file with an underscore on the end of the port name followed by the port revision. So this example Makefile produced a package named pfSense-pkg-suricata-6.0.3_4.

encrypt1d

@bmeeks
Thanks again so very much for all the help.
I think I am finally there. That last bit was the final hurdle. The build is fully clean of errors, and the app runs cleanly at runtime.

I can see my own debug messages, and miniupnpd is working as well as it does with the package that ships with 2.5.2.

That was a process, but worth it. Thanks again to @jimp as well for helping out with the patching commands.

My progress on actually getting miniupnpd to work behind a double NAT will be over here:

https://forum.netgate.com/topic/169773/miniupnp-full-cone-double-natincorrectly-adding-rules/8?_=1644582288930

I'll make one additional post here soon, to capture the whole process, step by step. Just need to write/clean it up.

Cheers!