pfSense compile requirements for 3rd party software

[encrypt1d]

@bmeeks
The jail setup completed successfully. Apparently your machine is faster than my virtual machine, ouch. 11 hrs.

root@devbox:/build/pfsense # ./build.sh --setup-poudriere
>>> Operation ./build.sh has started at Tue Feb 15 10:27:16 EST 2022
>>> Creating ZFS filesystem zroot/poudriere... Done!
>>> Creating poudriere.conf
>>> Creating jail pfSense_v2_5_2_amd64, it may take some time... 
Done!
>>> Creating poudriere ports tree, it may take some time... Done!
>>> Poudriere is now configured!
>>> Operation ./build.sh has ended at Tue Feb 15 21:16:43 EST 2022

Is my next step to run this?

./build.sh none

I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

Is my next step to run this?

./build.sh none

I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

I run this command to build the package tree:

./build.sh --update-pkg-repo -a amd64.amd64

That will build all the packages for AMD64/Intel architectures. You can just execute the shell script with no arguments to see all the available options like this:

./build.sh

I just rebuilt the jail in my RELEASE builder since I had to update it to the latest FreeBSD 12.3 with the recent 2.6.0 pfSense release, and it took about as long as yours. Much longer than I remembered from the past.

[encrypt1d]

@bmeeks

Thanks again, it appears to be running.

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Thanks again, it appears to be running.

Some of the packages will take a long time to build, particularly the Rust language one. Just be patient. On my builder VM, Rust takes over 4 hours by itself. And it needs lots of RAM (more than 8 GB). What I did was create a very large extra swap partition for it to use during that build. Makes it take longer, but my ESXi host only has 32 GB of RAM to share among my active VMs.

If Rust fails to build, it will probably be due to resource limits. If that happens, simply kick off the same build again (use the same command) and the process will pick up where it left off. It is smart enough not to rebuild everything from scratch every time.

So if you see any package fail during the process, just let it keep going until the job stops. Then you can run it again to rebuild any failures. Like I said, most times the failures are due to resource exhaustion in my minimal builder VM.

[encrypt1d]

@bmeeks

That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

Possibly, or at least a very long first day. After that, assuming you don't change any of the other ports' source code, each time you kick off a build it will only build your miniupnpd package (if you change the Makefile version) and a couple of small pfSense packages that are rebuilt on each run (these have timestamp info for the current build).

[encrypt1d]

@bmeeks

I've narrowed the build down to just 4 errors now, almost there.

[00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch
[00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch
[00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch
[00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetch

The logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

I've narrowed the build down to just 4 errors now, almost there.

[00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch
[00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch
[00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch
[00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetch

The logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?

In your specific use case, I don't think these failures matter. So just ignore them. So long as none of the impacted ports are dependencies of miniupnpd, then their failure to build won't matter. I think you were interested in building just miniupnpd.

Now at some point you will want to update your builder to 2.6.0 (RELENG_2_6_0 is the official branch name). But if the firewall you are testing on is at 2.5.2, then you want to keep your builder on the same version. The failures are likely the result of the older Ports tree which was based on FreeBSD-12.2.

FYI -- my RELEASE builder is just now finishing up the final packages for RELENG_2_6_0.

[encrypt1d]

@bmeeks

Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.

I think I can fix the other errors by placing the files it needs in /portdistfiles. The logs indicate it checks there after giving up on a direct download, but its good to know that doesn't matter.

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

Where are the resulting packages being placed? I've scoured the whole hard drive, and cannot find them. They must not have a .pkg extension.

The packages are put in /usr/local/poudriere/data/packages/{release}. So navigate to that directory and then in it will be symlinks. The symlink that contains the package files is named /All.

[encrypt1d]

@bmeeks said in pfSense compile requirements for 3rd party software:

files is named /All.

Thanks! I have a binary I can test now. :)

[bmeeks]

My favorite tool for connecting to my builders, browsing around there, and transferring files back and forth to my Windows PC is WinSCP.

[encrypt1d]

@bmeeks

My build worked, all those pesky ioctl errors are gone!

So now I am trying to change the source code, but not quite fully understanding this environment.

I figured I'd have to switch to this directory:

/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpd

In there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.

I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.

What's the trick here?

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks

My build worked, all those pesky ioctl errors are gone!

So now I am trying to change the source code, but not quite fully understanding this environment.

I figured I'd have to switch to this directory:

/usr/local/poudriere/ports/pfSense_v2_5_2/net/miniupnpd

In there I can run "make extract", edit my code in the work folder, and then run the "make makepatch". Seems to work as desired.

I edited the version in the Makefile in that same directory, and when I ran the full build from the main build dir as before, it cleaned out the old miniupnpd I built earlier, but failed to build the new one. It fails because the first thing it tries to do is download the dist file with the new version, which doesn't exist.

What's the trick here?

Yea, I would not do the extraction in the build directory. That is a magic ZFS file system. I would instead use the native ports path of /usr/ports/net/miniupnpd. Do all of your work there, and then produce the patch diff file. Copy that single diff file to the /files subdirectory of the port on the builder.

When changing the Makefile version, don't change the major or minor version. Instead, use the PORTREVISION tag. Here is an example from an old Suricata GUI package:

PORTNAME=	pfSense-pkg-suricata
PORTVERSION=	6.0.3
PORTREVISION=	4

If there is no PORTREVISION tag in your file, add it and start at 1 and increment by 1 for each build. That will produce a package file with an underscore on the end of the port name followed by the port revision. So this example Makefile produced a package named pfSense-pkg-suricata-6.0.3_4.

[encrypt1d]

@bmeeks
Thanks again so very much for all the help.
I think I am finally there. That last bit was the final hurdle. The build is fully clean of errors, and the app runs cleanly at runtime.

I can see my own debug messages, and miniupnpd is working as well as it does with the package that ships with 2.5.2.

That was a process, but worth it. Thanks again to @jimp as well for helping out with the patching commands.

My progress on actually getting miniupnpd to work behind a double NAT will be over here:

https://forum.netgate.com/topic/169773/miniupnp-full-cone-double-natincorrectly-adding-rules/8?_=1644582288930

I'll make one additional post here soon, to capture the whole process, step by step. Just need to write/clean it up.

Cheers!

[encrypt1d]

@bmeeks
What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE?

[bmeeks]

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE?

I personally have never been too fond of IDEs. I did once work for a short time in one of the Microsoft tools when I was doing Windows-related development where I worked.

I started programming at the literal 1s and 0s of raw machine language, then graduated to assembly. Not much in the way of IDE for that . I later moved on to C, C++, C#, a touch of Java, and then some PL/SQL while working with- and administering- some Oracle databases.

These days I do only PHP for the GUI code in the Snort and Suricata packages, and then C for making customizations in the underlying binaries.

[jimp]

@encrypt1d said in pfSense compile requirements for 3rd party software:

What do you folks use for an IDE to make browsing through C code easier than grepping? Or perhaps, do you use an IDE?

I use UltraEdit on Linux and Windows for most things. Not really as a full IDE, mostly as a code editor with lots of nice features. On systems where I don't have a license for that I use Kate, Notepad++, or TextMate.

[encrypt1d]

@jimp @bmeeks

Thanks.
Really just looking for something that speeds up GUI browsing on FreeBSD, not a full IDE. Tracing function calls is much easier that way.
I was a developer for the first 10 years of my career, and miss those tools (C on VXWorks with a proprietary IDE, then moved on to C++/Visual Studio, Java/Eclipse etc). They were multimillion line code repos, so grep just didn't cut it. I googled around bit, but it gets religious fast.

[bmeeks]

One other thing I forgot to mention about the incremental building during development/troubleshooting is that I found it easier to do my initial work totally within the /usr/ports/* tree and leave the Poudriere tree alone at first. This way I can run a quick make in the /usr/ports/* directory to check for stupid C syntax errors or other coding oversights I may generate with my changes. You get immediate feedback of errors there via the console.

When you run the ./build.sh script to fire off the build within the package builder, any compilation errors are buried in a corresponding log file down in the /usr/local/poudriere/ tree. You can find the log, but it's a bit of effort.

So I do my initial compile in the regular /usr/ports/ tree to make sure my code compiles successfully. Then, if it does, I will copy my patch diff over to the Poudriere tree and kick off the package builder so I get a package I can copy over to pfSense and actually execute.