pfSense compile requirements for 3rd party software

encrypt1d

Hi folks,
The is my first time trying to compile software targeted for pfSense, so please be gentle :)

It is in follow up to this thread, where I am in a double NAT scenario and want to experiment with some fixes to make miniupnpd not care about private addresses.
https://forum.netgate.com/topic/154153/test-request-upnp-fix-for-multiple-consoles-playing-the-same-game-static-port-outbound-nat/122

Long story short, I tried compiling miniupnpd myself, and it goes cleanly. Running the binary on the firewall unfortunately results in a broken upnp server, and the only error I have to work with is:

Feb  9 12:54:07 firewall miniupnpd[5873]: ioctl(dev, DIOCGETRULES, ...): Operation not supported by device

Specifically, what is broken in my build is that miniupnpd isn't adding any pf rules.

Now to the questions:

Is that error due to some sort of compile setting I haven't changed, or changed incorrectly?
Does pfSense modify the code in order to work on this platform?
Does pfSense modify the makefile.bsd, config.h file or anything else during its builds?

Sure would appreciate any help that can be provided.
-Jeff

P.S.

I am building on a FreeBSD 12.2 RELEASE 64 bit OS
I am running pfSense 2.5.2
I used the master branch at miniupnpd, cloned via git

bmeeks

pfSense is built on FreeBSD 12.2 STABLE (at the moment, but this will be 12.3 STABLE as soon as 2.6.0 drops). So RELEASE is not the correct branch to build on.

But you may have other issues as well in terms of some of the system settings the compiler and build system defaults to. First off, take a look at the make.conf file pfSense uses with Poudriere when building packages. See if there are any specific settings for your port listed here: https://github.com/pfsense/pfsense/blob/master/tools/conf/pfPorts/make.conf.

Next, check the Makefile for the miniupnpd port here: https://github.com/pfsense/FreeBSD-ports/tree/devel/net/miniupnpd. This will show you the specific settings pfSense uses when building the package for users to install from the pfSense Packages repository. Don't forget to check for any custom patch files that might be needed on pfSense when building. Those will be shown in the files subdirectory of the port on GitHub.

encrypt1d

@bmeeks
Thanks for the reply. I am looking through these files now for clues.

Unfortunately, 12.2 STABLE has disappeared from the download sites, I have not been able to find it so far. That could be a problem.

encrypt1d

So far, not so good.

I extracted what I believe were the options that should work, as follows:

From: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/miniupnpd/Makefile

CFLAGS+=	-I${OPENSSLINC}
LDFLAGS+=	-L${OPENSSLLIB}

OPTIONS_DEFINE=	PF_FILTER_RULES CHECK_PORTINUSE IPV6 UPNP_IGDV2 UPNP_STRICT LEASEFILE

And from: https://github.com/pfsense/pfsense/blob/master/tools/conf/pfPorts/make.conf

net_miniupnpd_SET_FORCE = PF_FILTER_RULES IPV6 CHECK_PORTINUSE

It's worth noting that the source code actually uses PF_ENABLE_FILTER_RULES, not PF_FILTER_RULES, so I am puzzled as to how that actually works in the official build.

At any rate, when I add the CFLAGS option, the compile breaks complaining about undefined symbols which are actually contained within the miniupnpd files, for example:

$ make 
cc -I. -DMINIUPNPD_GIT_REF=\"miniupnpd_2_2_1\" -O2 -pipe -Wall -W -Wstrict-prototypes -fno-common -I -c -o miniupnpd.o ./miniupnpd.c
ld: error: undefined symbol: ext_stun_host
>>> referenced by miniupnpd.c
>>>               /tmp/miniupnpd-de5b16.o:(update_ext_ip_addr_from_stun)
>>> referenced by miniupnpd.c
>>>               /tmp/miniupnpd-de5b16.o:(update_ext_ip_addr_from_stun)
>>> referenced by miniupnpd.c
>>>               /tmp/miniupnpd-de5b16.o:(main)
>>> referenced by miniupnpd.c
>>>               /tmp/miniupnpd-de5b16.o:(main)
>>> referenced by miniupnpd.c
>>>               /tmp/miniupnpd-de5b16.o:(main)

Not sure where to go from here. Ideas?

bmeeks

How exactly are you launching the compilation? On FreeBSD the accepted method is to use the Makefile that exists in the Ports tree on the box.

So make sure you have a current snapshot of the FreeBSD 12.2/12.3 Ports tree on your FreeBSD box. Next, ensure you have all the compile and build options installed.

Try a test compile by changing to the directory containing the miniupnpd port. That should be /usr/ports/net/miniupnpd. Then simply type:

make

That should kick off a complete build including dependent libraries. It may take a while to complete the build. If all this works, then you can try building a binary package using this command issued from the same port directory:

make package

That will build a binary package and save it in /usr/ports/packages on the box. You can then try copying the built package over to pfSense and trying to install it.

If you are not building using the method above (or using an actual pfSense package builder environment), then you will have issues. When using the method outlined above, you will need to create patch files for your customizations and store them in the files subdirectory of the port (or create that directory and copy the patches there if it does not exist).

encrypt1d

@bmeeks
Much appreciated again.

Sounds like I was approaching this all wrong. I had simply cloned the miniupnpd tree from: https://github.com/miniupnp/miniupnp
I ran their ./configure & make, then swapped out the binary on the firewall. So much for the quick path :)

I now have a package built: miniupnpd-2.2.1_1,1.pkg
It did not end up in the folder you mentioned, but was in ./work/pkg, and it doesn't have a .txz extension - so I am assuming something went wrong? Neither the make or the make package produced any errors, they ran clean.

Is this as simple as making a compressed tarball from the .pkg file, and then ...

pkg add -f miniupnpd-2.2.1_1,1.txz

To back it out if it doesn't work?

pkg delete -fy miniupnpd
pkg install -y miniupnpd

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
Much appreciated again.

Sounds like I was approaching this all wrong. I had simply cloned the miniupnpd tree from: https://github.com/miniupnp/miniupnp
I ran their ./configure & make, then swapped out the binary on the firewall. So much for the quick path :)

I now have a package built: miniupnpd-2.2.1_1,1.pkg
It did not end up in the folder you mentioned, but was in ./work/pkg, and it doesn't have a .txz extension - so I am assuming something went wrong? Neither the make or the make package produced any errors, they ran clean.

Is this as simple as making a compressed tarball from the .pkg file, and then ...
pkg add -f miniupnpd-2.2.1_1,1.txz

To back it out if it doesn't work?

pkg delete -fy miniupnpd
pkg install -y miniupnpd

I don't think you will need the "-f -y" arguments with the delete and install commands -- at least typically you would not.

If you have a virtual environment of some sort, I would quickly install a pfSense VM and do my install testing there. That way you don't risk corrupting your production firewall if something goes wrong.

Also don't think zipping the package and adding the *.txz extension is mandatory.

bmeeks

You have to be very careful just cloning source from GitHub when using FreeBSD and/or pfSense. That's because FreeBSD has its own binary ports tree, and the packages available there are not always the latest and greatest version from GitHub. Additionally, many times there are "adjustments" needed to the configure scripts in order for the source to properly compile on FreeBSD and for the resulting binary to run. This includes things like tweaks for specific versions of shared libraries.

Because pfSense is a customized FreeBSD operating system, the same caveats apply when just grabbing code from GitHub for one of the add-on binary packages. You must use the source code info from the FreeBSD port. And more specifically, it must be from the FreeBSD-ports tree maintained by the pfSense team.

FreeBSD ports don't have the source stored with the port. Instead, the Makefile will contain the name (and version) of the source.tar.gz file to download and unpack. So when you run the make command in the port's directory, that kicks off a process of downloading the source tarball, unpacking it into a ./work/ subdirectory structure in the port's directory, applying any patches available in the files subdirectory of the port, and then actually running the configure, compile, and link processes to produce the final binary.

So if you are actually patching the C source code of the port, you will need to separately unpack your own copy of the correct source tarball for the port. You actually will want two identical copies of the unpacked source tarball in two separate subdirectories. Name one of them whatever_port.orig. That one is the master gold reference copy. Make your source code edits in the other subdirectory (the whatever_port directory). When finished editing you need to produce a patch file using diff that contains all of your changes. I run this command to do that after changing into the parent directory containing my two source code subdirectories:

diff -ruN ./whatever_port.orig/ ./whatever_port/ > ./my_patch.diff

When that is done, edit the resulting my_patch.diff file to fix up the relative directory paths. You will need to remove the leading subdirectory names that came from your local structure. I think that will make sense when you see it in the file.

encrypt1d

@bmeeks
It all makes much more sense now, thanks!

I have a diff file now, and placed it in the ./net/miniupnpd/files folder, then ran a make from ./net/miniupnpd

The output did not show the patch getting applied, should it?

===>  Applying FreeBSD patches for miniupnpd-2.2.1_1,1 from /usr/home/admin/git/FreeBSD-ports/net/miniupnpd/files
/usr/bin/sed -i.bak -e 's|\(-lssl -lcrypto\)|$(LDFLAGS) \1|g'  /usr/home/admin/git/FreeBSD-ports/net/miniupnpd/work/miniupnpd-2.2.1/Makefile.bsd

The one line of output isn't mine.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

@bmeeks
It all makes much more sense now, thanks!

I have a diff file now, and placed it in the ./net/miniupnpd/files folder, then ran a make from ./net/miniupnpd

The output did not show the patch getting applied, should it?
===>  Applying FreeBSD patches for miniupnpd-2.2.1_1,1 from /usr/home/admin/git/FreeBSD-ports/net/miniupnpd/files
/usr/bin/sed -i.bak -e 's|$-lssl -lcrypto$|$(LDFLAGS) \1|g'  /usr/home/admin/git/FreeBSD-ports/net/miniupnpd/work/miniupnpd-2.2.1/Makefile.bsd
The one line of output isn't mine.

If it compiled without error when running the make command, it should be fine. If it can't apply a patch due to an error, the console output will show that and compilation will halt if you are doing it interactively by running make.

So are you saying that line of screen output you posted is not from your run, or is it? That line shows it read the ./files/ subdirectory and applied all the patches it found there successfully.

If you want to actually see the patches, look in the ./work/ subdirectory that gets created in the port's directory when you build it. In the source code folder under work you will find all of the modified and original source code files. The original ones get a special extension but the same first name.

By the way, between build cycles it is customary to clean the build directory tree under ./work using this command:

make clean

That insures you are starting fresh each time and not using something from an earlier run. I like to always use clean between build cycles when I'm working on something new in code.

encrypt1d

@bmeeks
Hmm. I don't think it is picking up my patch - I don't see any evidence of it in ./work/miniupnpd-2.2.1. I was hoping that the output of make would show it picking up my file specifically (positive affirmation). To answer your question, that is the output of my make run, and the line that is there is coming from a post-patch action in the make file itself, and it should be there.

This is what my diff looks like. Perhaps it is a relative path issue. Is it using the work directory as the starting point?

diff -ruN ./miniupnpd_port.orig/miniupnpd.c ./miniupnpd_port/miniupnpd.c
--- ./miniupnpd.c	2020-10-30 17:38:06.000000000 -0400
+++ ./miniupnpd.c	2022-02-10 11:12:04.026100000 -0500
@@ -2144,6 +2144,7 @@
 	}
 
 	syslog(LOG_INFO, "version " MINIUPNPD_VERSION " starting%s%sext if %s BOOTID=%u",
+	syslog(LOG_INFO, "CUSTOM version " MINIUPNPD_VERSION " starting%s%sext if %s BOOTID=%u",
 #ifdef ENABLE_NATPMP
 #ifdef ENABLE_PCP
 	       GETFLAG(ENABLENATPMPMASK) ? " NAT-PMP/PCP " : " ",

I tried 2 versions, one with all of the leading paths remove as you suggested, and this one where they were still in the diff command itself.

My folder structure looks like this:

./FreeBSD-ports/net/miniupnpd (from port tree - copied patch into files subfolder within, and ran make from here)
./FreeBSD-ports/net/miniupnpd_port.orig (golden copy)
./FreeBSD-ports/net/miniupnpd_port (changed code)

bmeeks

I'm not sure I understand your directory structure. Or perhaps you misunderstood what I meant. I don't think your patch file has the correct directory paths.

You will need the actual source tarball. Where did you obtain the source tarball file? Since you have built the package once already on your FreeBSD box, go look in /usr/ports/distfiles for the miniupnpd source tarball. Copy that *.tar.gz file to /tmp to make things easier.

I do my modifications in a structure under /tmp, but you could do it under any directory you created.

Now change to the /tmp directory and run this sequence of commands to extract the content of the source tarball into two subdirectories (renaming one of them as an intermediate step):

tar -xvzf whatever_src.tar.gz
mv whatever_src what_ever_src.orig
tar -xvzf whatever_src.tar.gz

In the above, "whatever_src.tar.gz" is the name of the source tarball. When you extract it, it should create a subdirectory in /tmp and put all the C source files, configure files, and other stuff needed to compile the binary in there.

The second command in the sequence above renames the source subdirectory to the same name but appends ".orig" to show it is the reference copy of the port.

Now make your source code changes in the other directory. When finished, run that diff command I provided earlier. Don't forget to go in and fix the relative paths.

Now copy the resulting *.diff patch into the ./files subdirectory of miniupnpd in /usr/ports/net/miniupnpd/files.

Run make again in the port subdirectory /usr/ports/net/miniupnpd and see what happens. It should pick up and apply your patch.

To illustrate, I frequently build custom binaries for Suricata and Snort for use on pfSense as part of my package maintainer duties. To begin, I obtain the suricata-6.0.4.tar.gz source tarball (for the current binary as of this date). When I unzip that file in my /tmp directory it will create a subdirectory called suricata-6.0.4 in /tmp -- so the full path is /tmp/suricata-6.0.4/. And in that subdirectory it will create several other nested subdirectories containing all the files needed to build the binary. The C source code files will be in /tmp/suricata-6.0.4/src/. I rename the directory from the first extraction to suricata-6.0.4.orig, then I extract the tarball contents a second time. I then have two identical subdirectories as follows: /tmp/suricata-6.0.4.orig/ and /tmp/suricata-6.0.4/. I make my source code modifications in the /tmp/suricata-6.0.4/src directory where Suricata places the C source files. Your port may do something else with its C source files.

Once I have my new or modified C source files ready, I run this command to create a patch file in the top-level of /tmp:

diff -ruN ./suricata-6.0.4.orig/ ./suricata-6.0.4/ > /tmp/my_patch.diff

The file header line for each file modified by the patch will contain this path: /tmp/suricata-6.0.4/src/some_file. I modify that line to read ./src/some_file, because the Suricata tarball puts all the C source code files in a ./src subdirectory inside the main suricata-6.0.4 directory. I then copy my patch file to the ./files subdirectory of the Suricata port and run make.

Don't worry about the work directory structure you see left over after building the port. When the build process unpacks the source tarball, it will handle adding any necessary leading directory names. Your patch file just needs to reflect the paths stored within the source tarball. Or in other words, how it looks when you just unzip the tarball in some directory.

encrypt1d

@bmeeks
Just to keep things clean, I used your exact same directory structure.

admin@devbox:/tmp $ ls
miniupnpd-2.2.1		miniupnpd-2.2.1.orig
admin@devbox:/tmp $ diff -ruN ./miniupnpd-2.2.1.orig/ ./miniupnpd-2.2.1/ > /tmp/my_patch.diff
admin@devbox:/tmp $ cat /tmp/my_patch.diff 
diff -ruN ./miniupnpd-2.2.1.orig/miniupnpd.c ./miniupnpd-2.2.1/miniupnpd.c
--- ./miniupnpd-2.2.1.orig/miniupnpd.c	2020-10-30 17:38:06.000000000 -0400
+++ ./miniupnpd-2.2.1/miniupnpd.c	2022-02-11 07:46:28.667533000 -0500
@@ -1822,6 +1822,7 @@
 
 #ifdef TOMATO
 	syslog(LOG_NOTICE, "version " MINIUPNPD_VERSION " started");
+	syslog(LOG_NOTICE, "CUSTOM version " MINIUPNPD_VERSION " started");
 #endif /* TOMATO */
 
 	set_startup_time();

I then removed the leading paths (miniupnpd has no subfolders in its tarball):

diff -ruN ./miniupnpd.c ./miniupnpd.c
--- ./miniupnpd.c       2020-10-30 17:38:06.000000000 -0400
+++ ./miniupnpd.c       2022-02-11 07:46:28.667533000 -0500
@@ -1822,6 +1822,7 @@

 #ifdef TOMATO
        syslog(LOG_NOTICE, "version " MINIUPNPD_VERSION " started");
+       syslog(LOG_NOTICE, "CUSTOM version " MINIUPNPD_VERSION " started");
 #endif /* TOMATO */

        set_startup_time();

I then place the my_patch.diff file in file here, along with the ones miniupnpd ships with:

dmin@devbox:~/git/FreeBSD-ports/net/miniupnpd/files $ ls -l
total 20
-rw-r--r--  1 admin  admin   527 Feb 10 13:21 miniupnpd.c
-rw-r--r--  1 admin  admin  2037 Feb 10 11:02 miniupnpd.conf.sample.in
-rw-r--r--  1 admin  admin   558 Feb 10 11:02 miniupnpd.in
-rw-r--r--  1 admin  admin   400 Feb 11 07:52 my_patch.diff
-rw-r--r--  1 admin  admin   512 Feb 10 11:02 patch-Makefile.bsd

I ran make clean, and make.

admin@devbox:~/git/FreeBSD-ports/net/miniupnpd $ make clean
admin@devbox:~/git/FreeBSD-ports/net/miniupnpd $ make

The patch isn't being merged in. I verified that by testing in a VM, and by inspection of the work directory, the source files in there are the originals except for the compiled outputs (date agrees):

-rwxr-xr-x  1 admin  wheel  - 177016 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd*
-rw-r--r--  1 admin  admin  -   2735 Oct  5  2019 work/miniupnpd-2.2.1/miniupnpd.8
-rw-r--r--  1 admin  admin  -  80744 Oct 30  2020 work/miniupnpd-2.2.1/miniupnpd.c
-rw-r--r--  1 admin  admin  -   6490 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd.conf
-rw-r--r--  1 admin  admin  -  44680 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd.o

So its seems the build is ignoring my patch. I have to assume I have done something wrong, but I have no idea what it could be.

jimp

The easiest thing to do when hacking on ports like this is to use the ports tree features.

If you have a copy of the ports tree from the same branch as pfSense (e.g. clone the copy from the RELENG_2_6_0 branch of the copy of the FreeBSD ports repo on the pfSense github account) you can do something like this:

$ cd ~/devel/pfsense-ports/net/miniupnpd
$ make extract
$ cd work/miniupnpd<whatever version it is>
<copy original file to `<name>.orig`>
<edit file>
<repeat for each file>
$ make makepatch
$ make clean
$ make package

makepatch will automatically do the diff and such and create properly formatted patch files. No need to do any of that by hand.

Also by default the pfSense make.conf for ports sets PF_FILTER_RULES IPV6 CHECK_PORTINUSE for the options.

bmeeks

@jimp said in pfSense compile requirements for 3rd party software:

The easiest thing to do when hacking on ports like this is to use the ports tree features.

If you have a copy of the ports tree from the same branch as pfSense (e.g. clone the copy from the RELENG_2_6_0 branch of the copy of the FreeBSD ports repo on the pfSense github account) you can do something like this:
$ cd ~/devel/pfsense-ports/net/miniupnpd
$ make extract
$ cd work/miniupnpd<whatever version it is>
<edit files>
$ make makepatch
$ make clean
$ make package
makepatch will automatically do the diff and such and create properly formatted patch files. No need to do any of that by hand.

Also by default the pfSense make.conf for ports sets PF_FILTER_RULES IPV6 CHECK_PORTINUSE for the options.

Thanks @jimp! I did not know about those extra make commands. Where are these commands currently documented for 'make`? This will greatly simplify my work when updating the Suricata and Snort binaries.

jimp

@bmeeks said in pfSense compile requirements for 3rd party software:

Thanks @jimp! I did not know about those extra make commands. Where are these commands currently documented for 'make`? This will greatly simplify my work when updating the Suricata and Snort binaries.

I forgot a step in there but I added it in. Those are documented in the FreeBSD ports handbook IIRC or you can always look in Mk/bsd.port.mk in the ports repo (or any of the files in that part of the structure) to see what other useful make targets and features there are.

encrypt1d

@jimp @bmeeks
That did the trick! I can see my new log message when the service starts now. I was able to install and back out my patch in my VM successfully using the package manager.

Sadly I am still seeing this in my builds:

ioctl(dev, DIOCGETRULES, ...): Operation not supported by device

Note that I am unable to build against the exact same target as my firewall (12.2 stable) - they've removed that download from all of the internet as far as I can tell. I will try 12.3 Stable next, as that is the next closest.

encrypt1d

Thanks to you both @jimp and @bmeeks.

Switching to the 12.3 stable worked well, and I also I set my repo to the v2.5.2 label to be safe.

I can patch, compile, install and run custom code now without errors.

Now on to the actual work ... lol.

Much appreciated.

encrypt1d

I do have one final question - the first time I ran make, this nice little dialog box pops up in the shell:

Screenshot 2022-02-11 120805.png

If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again.

jimp

@encrypt1d said in pfSense compile requirements for 3rd party software:

If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again

make config