pfSense compile requirements for 3rd party software

bmeeks

I'm not sure I understand your directory structure. Or perhaps you misunderstood what I meant. I don't think your patch file has the correct directory paths.

You will need the actual source tarball. Where did you obtain the source tarball file? Since you have built the package once already on your FreeBSD box, go look in /usr/ports/distfiles for the miniupnpd source tarball. Copy that *.tar.gz file to /tmp to make things easier.

I do my modifications in a structure under /tmp, but you could do it under any directory you created.

Now change to the /tmp directory and run this sequence of commands to extract the content of the source tarball into two subdirectories (renaming one of them as an intermediate step):

tar -xvzf whatever_src.tar.gz
mv whatever_src what_ever_src.orig
tar -xvzf whatever_src.tar.gz

In the above, "whatever_src.tar.gz" is the name of the source tarball. When you extract it, it should create a subdirectory in /tmp and put all the C source files, configure files, and other stuff needed to compile the binary in there.

The second command in the sequence above renames the source subdirectory to the same name but appends ".orig" to show it is the reference copy of the port.

Now make your source code changes in the other directory. When finished, run that diff command I provided earlier. Don't forget to go in and fix the relative paths.

Now copy the resulting *.diff patch into the ./files subdirectory of miniupnpd in /usr/ports/net/miniupnpd/files.

Run make again in the port subdirectory /usr/ports/net/miniupnpd and see what happens. It should pick up and apply your patch.

To illustrate, I frequently build custom binaries for Suricata and Snort for use on pfSense as part of my package maintainer duties. To begin, I obtain the suricata-6.0.4.tar.gz source tarball (for the current binary as of this date). When I unzip that file in my /tmp directory it will create a subdirectory called suricata-6.0.4 in /tmp -- so the full path is /tmp/suricata-6.0.4/. And in that subdirectory it will create several other nested subdirectories containing all the files needed to build the binary. The C source code files will be in /tmp/suricata-6.0.4/src/. I rename the directory from the first extraction to suricata-6.0.4.orig, then I extract the tarball contents a second time. I then have two identical subdirectories as follows: /tmp/suricata-6.0.4.orig/ and /tmp/suricata-6.0.4/. I make my source code modifications in the /tmp/suricata-6.0.4/src directory where Suricata places the C source files. Your port may do something else with its C source files.

Once I have my new or modified C source files ready, I run this command to create a patch file in the top-level of /tmp:

diff -ruN ./suricata-6.0.4.orig/ ./suricata-6.0.4/ > /tmp/my_patch.diff

The file header line for each file modified by the patch will contain this path: /tmp/suricata-6.0.4/src/some_file. I modify that line to read ./src/some_file, because the Suricata tarball puts all the C source code files in a ./src subdirectory inside the main suricata-6.0.4 directory. I then copy my patch file to the ./files subdirectory of the Suricata port and run make.

Don't worry about the work directory structure you see left over after building the port. When the build process unpacks the source tarball, it will handle adding any necessary leading directory names. Your patch file just needs to reflect the paths stored within the source tarball. Or in other words, how it looks when you just unzip the tarball in some directory.

encrypt1d

@bmeeks
Just to keep things clean, I used your exact same directory structure.

admin@devbox:/tmp $ ls
miniupnpd-2.2.1		miniupnpd-2.2.1.orig
admin@devbox:/tmp $ diff -ruN ./miniupnpd-2.2.1.orig/ ./miniupnpd-2.2.1/ > /tmp/my_patch.diff
admin@devbox:/tmp $ cat /tmp/my_patch.diff 
diff -ruN ./miniupnpd-2.2.1.orig/miniupnpd.c ./miniupnpd-2.2.1/miniupnpd.c
--- ./miniupnpd-2.2.1.orig/miniupnpd.c	2020-10-30 17:38:06.000000000 -0400
+++ ./miniupnpd-2.2.1/miniupnpd.c	2022-02-11 07:46:28.667533000 -0500
@@ -1822,6 +1822,7 @@
 
 #ifdef TOMATO
 	syslog(LOG_NOTICE, "version " MINIUPNPD_VERSION " started");
+	syslog(LOG_NOTICE, "CUSTOM version " MINIUPNPD_VERSION " started");
 #endif /* TOMATO */
 
 	set_startup_time();

I then removed the leading paths (miniupnpd has no subfolders in its tarball):

diff -ruN ./miniupnpd.c ./miniupnpd.c
--- ./miniupnpd.c       2020-10-30 17:38:06.000000000 -0400
+++ ./miniupnpd.c       2022-02-11 07:46:28.667533000 -0500
@@ -1822,6 +1822,7 @@

 #ifdef TOMATO
        syslog(LOG_NOTICE, "version " MINIUPNPD_VERSION " started");
+       syslog(LOG_NOTICE, "CUSTOM version " MINIUPNPD_VERSION " started");
 #endif /* TOMATO */

        set_startup_time();

I then place the my_patch.diff file in file here, along with the ones miniupnpd ships with:

dmin@devbox:~/git/FreeBSD-ports/net/miniupnpd/files $ ls -l
total 20
-rw-r--r--  1 admin  admin   527 Feb 10 13:21 miniupnpd.c
-rw-r--r--  1 admin  admin  2037 Feb 10 11:02 miniupnpd.conf.sample.in
-rw-r--r--  1 admin  admin   558 Feb 10 11:02 miniupnpd.in
-rw-r--r--  1 admin  admin   400 Feb 11 07:52 my_patch.diff
-rw-r--r--  1 admin  admin   512 Feb 10 11:02 patch-Makefile.bsd

I ran make clean, and make.

admin@devbox:~/git/FreeBSD-ports/net/miniupnpd $ make clean
admin@devbox:~/git/FreeBSD-ports/net/miniupnpd $ make

The patch isn't being merged in. I verified that by testing in a VM, and by inspection of the work directory, the source files in there are the originals except for the compiled outputs (date agrees):

-rwxr-xr-x  1 admin  wheel  - 177016 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd*
-rw-r--r--  1 admin  admin  -   2735 Oct  5  2019 work/miniupnpd-2.2.1/miniupnpd.8
-rw-r--r--  1 admin  admin  -  80744 Oct 30  2020 work/miniupnpd-2.2.1/miniupnpd.c
-rw-r--r--  1 admin  admin  -   6490 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd.conf
-rw-r--r--  1 admin  admin  -  44680 Feb 11 07:54 work/miniupnpd-2.2.1/miniupnpd.o

So its seems the build is ignoring my patch. I have to assume I have done something wrong, but I have no idea what it could be.

jimp

The easiest thing to do when hacking on ports like this is to use the ports tree features.

If you have a copy of the ports tree from the same branch as pfSense (e.g. clone the copy from the RELENG_2_6_0 branch of the copy of the FreeBSD ports repo on the pfSense github account) you can do something like this:

$ cd ~/devel/pfsense-ports/net/miniupnpd
$ make extract
$ cd work/miniupnpd<whatever version it is>
<copy original file to `<name>.orig`>
<edit file>
<repeat for each file>
$ make makepatch
$ make clean
$ make package

makepatch will automatically do the diff and such and create properly formatted patch files. No need to do any of that by hand.

Also by default the pfSense make.conf for ports sets PF_FILTER_RULES IPV6 CHECK_PORTINUSE for the options.

bmeeks

@jimp said in pfSense compile requirements for 3rd party software:

The easiest thing to do when hacking on ports like this is to use the ports tree features.

If you have a copy of the ports tree from the same branch as pfSense (e.g. clone the copy from the RELENG_2_6_0 branch of the copy of the FreeBSD ports repo on the pfSense github account) you can do something like this:
$ cd ~/devel/pfsense-ports/net/miniupnpd
$ make extract
$ cd work/miniupnpd<whatever version it is>
<edit files>
$ make makepatch
$ make clean
$ make package
makepatch will automatically do the diff and such and create properly formatted patch files. No need to do any of that by hand.

Also by default the pfSense make.conf for ports sets PF_FILTER_RULES IPV6 CHECK_PORTINUSE for the options.

Thanks @jimp! I did not know about those extra make commands. Where are these commands currently documented for 'make`? This will greatly simplify my work when updating the Suricata and Snort binaries.

jimp

@bmeeks said in pfSense compile requirements for 3rd party software:

Thanks @jimp! I did not know about those extra make commands. Where are these commands currently documented for 'make`? This will greatly simplify my work when updating the Suricata and Snort binaries.

I forgot a step in there but I added it in. Those are documented in the FreeBSD ports handbook IIRC or you can always look in Mk/bsd.port.mk in the ports repo (or any of the files in that part of the structure) to see what other useful make targets and features there are.

encrypt1d

@jimp @bmeeks
That did the trick! I can see my new log message when the service starts now. I was able to install and back out my patch in my VM successfully using the package manager.

Sadly I am still seeing this in my builds:

ioctl(dev, DIOCGETRULES, ...): Operation not supported by device

Note that I am unable to build against the exact same target as my firewall (12.2 stable) - they've removed that download from all of the internet as far as I can tell. I will try 12.3 Stable next, as that is the next closest.

encrypt1d

Thanks to you both @jimp and @bmeeks.

Switching to the 12.3 stable worked well, and I also I set my repo to the v2.5.2 label to be safe.

I can patch, compile, install and run custom code now without errors.

Now on to the actual work ... lol.

Much appreciated.

encrypt1d

I do have one final question - the first time I ran make, this nice little dialog box pops up in the shell:

Screenshot 2022-02-11 120805.png

If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again.

jimp

@encrypt1d said in pfSense compile requirements for 3rd party software:

If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again

make config

encrypt1d

@jimp
I like the easy answers :)

encrypt1d

Well it seems I spoke too soon.

Any version I build still seems to have the same issue, I thought I had it fixed, but I was wrong.

Even just building what is natively in the port tree (no patches) does not work for me. Still the same errors from IOCTL.

No idea how to fix this, and open to ideas. its slightly inconsistent as well, some builds have far more of these errors than others, with really no changes.

ioctl(dev, DIOCGETRULES, ...): Operation not supported by device
ioctl(dev, DIOCCHANGERULE, ...) PF_CHANGE_GET_TICKET: Operation not supported by device

Seems like @jimp worked on something like this a LONG time ago here: https://redmine.pfsense.org/issues/2527

jimp

There are some recent changes in pf ioctls but it should be in our tree. Usually that kind of thing would happen if the pf sources in your src tree don't match the pf sources in the tree used to build pfSense.

You may need to make sure you have a copy of the FreeBSD src repo from the pfSense github in /usr/src and make sure it's on the RELENG_2_6_0 branch.

encrypt1d

@jimp
Since I am testing on 2.5.2 on my firewall, would I not want RELENG_2_5_2?

jimp

Then you'd want to be using the RELENG_2_5_2 branch of the src and ports trees.

encrypt1d

@jimp

I started fresh, (Build VM is FreeBSD 12.3 Stable Feb 10) and cloned the FreeBSD source as follows:

git clone -b RELENG_2_5_2 --single-branch https://github.com/pfsense/FreeBSD-src

I moved the contents of the ./FreeBSD-src/ that command downloaded into /usr/src, so that the FreeBSD-src folder isn't in the path anymore.

I then cloned the Ports as follows in my home dir.

git clone -b RELENG_2_5_2 --single-branch https://github.com/pfsense/FreeBSD-ports

The target test firewall is pfSense 2.5.2 RELEASE. Build is clean.

The IOCTL errors still happen, and the application is partially functional, but my game clients don't even try to talk to it. For now, I am not changing or patching anything, just building what comes right out of the port tree for this branch. Is there a checklist somewhere for setting up an environment? I must have missed something.

Could it be the kernel version difference?

Installing miniupnpd-2.2.1_1,1...
Newer FreeBSD version for package miniupnpd:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1203505
- running kernel: 1202504
Ignore the mismatch and continue? [y/N]: y

P.S. I am hoping this is a consequence of the errors, but the really weird stuff is that the version of miniupnpd that shipped with 2.5.2 looks like this from Windows (It has a settings option from the general tab, which I can manually program ports if I want to).
f4a4c32c-2481-45d0-b777-44ce671a5022-Native Server-1.png

The version I built straight from the port tree has a different icon, and is missing the general tab, and does not have a way to add ports. Game clients also don't even try. I'll worry about this after I finally fix the IOCTL erros if it is still happening.

encrypt1d

I continued to work on this over the weekend and found an image of FreeBSD which has a closer kernel version - now at 1202505 which is only 1 off from what pkg reports for pfSense 2.5.2 (1202504). That's the Jan 28 2021 snapshot of 12.2 stable. Still unable to find the ISO for the same build pfSense used. It would be good if you guys hosted that somewhere.

Same issue - ioctl runtime errors. It's also worth noting the binary I get is bigger than the one that comes out of the official pfSense repo:

original binary:
-rwxr-xr-x  1 root  wheel  155368 Nov 15 11:52 miniupnpd
built binary after pkg add:
-rwxr-xr-x  1 root  wheel  177000 Feb 13 10:23 miniupnpd

That suggests the compile is using different options I suppose.

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

I continued to work on this over the weekend and found an image of FreeBSD which has a closer kernel version - now at 1202505 which is only 1 off from what pkg reports for pfSense 2.5.2 (1202504). That's the Jan 28 2021 snapshot of 12.2 stable. Still unable to find the ISO for the same build pfSense used. It would be good if you guys hosted that somewhere.

Same issue - ioctl runtime errors. It's also worth noting the binary I get is bigger than the one that comes out of the official pfSense repo:
original binary:
-rwxr-xr-x  1 root  wheel  155368 Nov 15 11:52 miniupnpd
built binary after pkg add:
-rwxr-xr-x  1 root  wheel  177000 Feb 13 10:23 miniupnpd
That suggests the compile is using different options I suppose.

You need to compare the content of the pf header files in your source tree on the machine where you are executing the build to the same header files listed here: https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_5_2.

Whether or not you have the exact same kernel is not as critical as having the exact same header files in your build environment. My bet is some of the pf-related header files are different in your build environment as compared to the pfSense 2.5.2 build environment.

You should be able to compile the "stock" miniupndpd package from the pfSense ports repository and install and run it without incident on a pfSense 2.5.2 machine. If that is not working, then it most likely is header files that are your problem. You are not compiling a kernel. You are simply compiling a binary executable and perhas a few dependent libraries.

I assume you have actually installed miniupnpd from the SYSTEM > PACKAGE MANAGER screen in pfSense and it works from there. If so, then you should be able to compile the exact same package and have it work. Once you get past that, you can start modifying code. But if you can't get the stock 2.5.2 package to compile and install, then you most likely have header file mismatches.

Don't forget to switch to the proper branch in Git when pulling down files to sync in your local repo.

encrypt1d

@bmeeks

Don't forget to switch to the proper branch in Git when pulling down files to sync in your local repo.

I did a fresh clone, and yep I have done that for both repos, and they both showed this message:

Branch 'RELENG_2_5_2' set up to track remote branch 'RELENG_2_5_2' from 'origin'.

I assume you have actually installed miniupnpd from the SYSTEM > PACKAGE MANAGER screen in pfSense and it works from there

That is correct. No IOTCL errors from the official package.

But if you can't get the stock 2.5.2 package to compile and install, then you most likely have header file mismatches.

I have no issues with compile and install. The service actually does start, but it throws IOCTL errors in the log, and fails to function correctly.

Just be be certain,
I have the FreeBSD-src repo in:

/git/FreeBSD-src

I have the Free-BSD ports repo in:

/git/FreeBSD-ports

Earlier it was mentioned that the src should be in /usr/src, so I created a symbolic link in /usr as follows:

lrwxr-xr-x   1 root  wheel  -    17 Feb 13 15:55 src@ -> /git/FreeBSD-src/

Yet none of my build will function correctly upon starting the service on the firewall. This should just work right?

(pulls out hair)

bmeeks

@encrypt1d said in pfSense compile requirements for 3rd party software:

Earlier it was mentioned that the src should be in /usr/src, so I created a symbolic link in /usr as follows:
lrwxr-xr-x   1 root  wheel  -    17 Feb 13 15:55 src@ -> /git/FreeBSD-src/
Yet none of my build will function correctly upon starting the service on the firewall. This should just work right?

(pulls out hair)

I think your problem is that the include files on your build system do not match those used by pfSense. Let me explain.

The way the pfSense team builds their image is on a custom builder machine. That machine has their customized FreeBSD kernel along with its include files. The packages repository is built within a Poudriere jail on this builder machine. The Poudriere jail installs the pfSense FreeBSD source tree including its header files. Everything happens in that jail. But the key thing here is that pfSense itself (the operating system as it were for pfSense) is built from the FreeBSD source tree. Packages, on the other hand, are built from the source code referenced in the Makefile and the header include files in /usr/include within the Poudriere jail.

But you are building outside of the jail, so the files that actually get used are the local headers (include files, really) installed on YOUR machine. These are in /usr/include. Those are the files I'm betting do not match up with the ones in the pfSense build tree. Don't get too hung up on the /usr/src tree since your build is not really using that. The files in there are for building the kernel, but you are not building the kernel. You are building a package. Packages need to have the correct include files in the /usr/include tree on the box where the packages are being built.

encrypt1d

@bmeeks

Packages need to have the correct include files in the /usr/include tree on the box where the packages are being built.

I tried the following two things based on that advice:

Outright replacement of /usr/include with only the contents of the pfSense
/git/FreeBSD-src/include. This resulting in missing headers, and compile failure.
I set /usr/include set back to original content, then copied contents of the pfSense repo /git/FreeBSD-src/include into /usr/include (effectively a merge over top of the original). This compiles, links, and runs on the firewall, but with the same ioctl errors.
Copied all of the pfSense git repo contents (FreeBSD-src) into /usr. Build works, same ioctl issues.

Am I chasing my tail trying to do this outside a poudriere jail?