Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense compile requirements for 3rd party software

    Scheduled Pinned Locked Moved Development
    102 Posts 8 Posters 27.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      encrypt1d @encrypt1d
      last edited by

      Thanks to you both @jimp and @bmeeks.

      Switching to the 12.3 stable worked well, and I also I set my repo to the v2.5.2 label to be safe.

      I can patch, compile, install and run custom code now without errors.

      Now on to the actual work ... lol.

      Much appreciated.

      E 1 Reply Last reply Reply Quote 1
      • E
        encrypt1d @encrypt1d
        last edited by

        I do have one final question - the first time I ran make, this nice little dialog box pops up in the shell:

        Screenshot 2022-02-11 120805.png

        If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again.

        jimpJ 1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate @encrypt1d
          last edited by

          @encrypt1d said in pfSense compile requirements for 3rd party software:

          If I needed to - how can I re-invoke it? It only comes up on the first make, and then never again

          make config

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          E 1 Reply Last reply Reply Quote 2
          • E
            encrypt1d @jimp
            last edited by

            @jimp
            I like the easy answers :)

            E 1 Reply Last reply Reply Quote 0
            • E
              encrypt1d @encrypt1d
              last edited by

              Well it seems I spoke too soon.

              Any version I build still seems to have the same issue, I thought I had it fixed, but I was wrong.

              Even just building what is natively in the port tree (no patches) does not work for me. Still the same errors from IOCTL.

              No idea how to fix this, and open to ideas. its slightly inconsistent as well, some builds have far more of these errors than others, with really no changes.

              ioctl(dev, DIOCGETRULES, ...): Operation not supported by device
              ioctl(dev, DIOCCHANGERULE, ...) PF_CHANGE_GET_TICKET: Operation not supported by device
              

              Seems like @jimp worked on something like this a LONG time ago here: https://redmine.pfsense.org/issues/2527

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                There are some recent changes in pf ioctls but it should be in our tree. Usually that kind of thing would happen if the pf sources in your src tree don't match the pf sources in the tree used to build pfSense.

                You may need to make sure you have a copy of the FreeBSD src repo from the pfSense github in /usr/src and make sure it's on the RELENG_2_6_0 branch.

                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                E 1 Reply Last reply Reply Quote 1
                • E
                  encrypt1d @jimp
                  last edited by

                  @jimp
                  Since I am testing on 2.5.2 on my firewall, would I not want RELENG_2_5_2?

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    Then you'd want to be using the RELENG_2_5_2 branch of the src and ports trees.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    E 1 Reply Last reply Reply Quote 1
                    • E
                      encrypt1d @jimp
                      last edited by

                      @jimp

                      I started fresh, (Build VM is FreeBSD 12.3 Stable Feb 10) and cloned the FreeBSD source as follows:

                      git clone -b RELENG_2_5_2 --single-branch https://github.com/pfsense/FreeBSD-src
                      

                      I moved the contents of the ./FreeBSD-src/ that command downloaded into /usr/src, so that the FreeBSD-src folder isn't in the path anymore.

                      I then cloned the Ports as follows in my home dir.

                      git clone -b RELENG_2_5_2 --single-branch https://github.com/pfsense/FreeBSD-ports
                      

                      The target test firewall is pfSense 2.5.2 RELEASE. Build is clean.

                      The IOCTL errors still happen, and the application is partially functional, but my game clients don't even try to talk to it. For now, I am not changing or patching anything, just building what comes right out of the port tree for this branch. Is there a checklist somewhere for setting up an environment? I must have missed something.

                      Could it be the kernel version difference?

                      Installing miniupnpd-2.2.1_1,1...
                      Newer FreeBSD version for package miniupnpd:
                      To ignore this error set IGNORE_OSVERSION=yes
                      - package: 1203505
                      - running kernel: 1202504
                      Ignore the mismatch and continue? [y/N]: y
                      

                      P.S. I am hoping this is a consequence of the errors, but the really weird stuff is that the version of miniupnpd that shipped with 2.5.2 looks like this from Windows (It has a settings option from the general tab, which I can manually program ports if I want to).
                      f4a4c32c-2481-45d0-b777-44ce671a5022-Native Server-1.png

                      The version I built straight from the port tree has a different icon, and is missing the general tab, and does not have a way to add ports. Game clients also don't even try. I'll worry about this after I finally fix the IOCTL erros if it is still happening.

                      5f22dd02-edd4-4759-a6eb-5fde3336c0ff-image.png

                      E 1 Reply Last reply Reply Quote 0
                      • E
                        encrypt1d @encrypt1d
                        last edited by

                        I continued to work on this over the weekend and found an image of FreeBSD which has a closer kernel version - now at 1202505 which is only 1 off from what pkg reports for pfSense 2.5.2 (1202504). That's the Jan 28 2021 snapshot of 12.2 stable. Still unable to find the ISO for the same build pfSense used. It would be good if you guys hosted that somewhere.

                        Same issue - ioctl runtime errors. It's also worth noting the binary I get is bigger than the one that comes out of the official pfSense repo:

                        original binary:
                        -rwxr-xr-x  1 root  wheel  155368 Nov 15 11:52 miniupnpd
                        built binary after pkg add:
                        -rwxr-xr-x  1 root  wheel  177000 Feb 13 10:23 miniupnpd
                        

                        That suggests the compile is using different options I suppose.

                        bmeeksB 1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks @encrypt1d
                          last edited by bmeeks

                          @encrypt1d said in pfSense compile requirements for 3rd party software:

                          I continued to work on this over the weekend and found an image of FreeBSD which has a closer kernel version - now at 1202505 which is only 1 off from what pkg reports for pfSense 2.5.2 (1202504). That's the Jan 28 2021 snapshot of 12.2 stable. Still unable to find the ISO for the same build pfSense used. It would be good if you guys hosted that somewhere.

                          Same issue - ioctl runtime errors. It's also worth noting the binary I get is bigger than the one that comes out of the official pfSense repo:

                          original binary:
                          -rwxr-xr-x  1 root  wheel  155368 Nov 15 11:52 miniupnpd
                          built binary after pkg add:
                          -rwxr-xr-x  1 root  wheel  177000 Feb 13 10:23 miniupnpd
                          

                          That suggests the compile is using different options I suppose.

                          You need to compare the content of the pf header files in your source tree on the machine where you are executing the build to the same header files listed here: https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_5_2.

                          Whether or not you have the exact same kernel is not as critical as having the exact same header files in your build environment. My bet is some of the pf-related header files are different in your build environment as compared to the pfSense 2.5.2 build environment.

                          You should be able to compile the "stock" miniupndpd package from the pfSense ports repository and install and run it without incident on a pfSense 2.5.2 machine. If that is not working, then it most likely is header files that are your problem. You are not compiling a kernel. You are simply compiling a binary executable and perhas a few dependent libraries.

                          I assume you have actually installed miniupnpd from the SYSTEM > PACKAGE MANAGER screen in pfSense and it works from there. If so, then you should be able to compile the exact same package and have it work. Once you get past that, you can start modifying code. But if you can't get the stock 2.5.2 package to compile and install, then you most likely have header file mismatches.

                          Don't forget to switch to the proper branch in Git when pulling down files to sync in your local repo.

                          E 1 Reply Last reply Reply Quote 1
                          • E
                            encrypt1d @bmeeks
                            last edited by

                            @bmeeks

                            Don't forget to switch to the proper branch in Git when pulling down files to sync in your local repo.

                            I did a fresh clone, and yep I have done that for both repos, and they both showed this message:

                            Branch 'RELENG_2_5_2' set up to track remote branch 'RELENG_2_5_2' from 'origin'.
                            

                            I assume you have actually installed miniupnpd from the SYSTEM > PACKAGE MANAGER screen in pfSense and it works from there

                            That is correct. No IOTCL errors from the official package.

                            But if you can't get the stock 2.5.2 package to compile and install, then you most likely have header file mismatches.

                            I have no issues with compile and install. The service actually does start, but it throws IOCTL errors in the log, and fails to function correctly.

                            Just be be certain,
                            I have the FreeBSD-src repo in:

                            /git/FreeBSD-src
                            

                            I have the Free-BSD ports repo in:

                            /git/FreeBSD-ports
                            

                            Earlier it was mentioned that the src should be in /usr/src, so I created a symbolic link in /usr as follows:

                            lrwxr-xr-x   1 root  wheel  -    17 Feb 13 15:55 src@ -> /git/FreeBSD-src/
                            

                            Yet none of my build will function correctly upon starting the service on the firewall. This should just work right?

                            (pulls out hair)

                            bmeeksB 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @encrypt1d
                              last edited by bmeeks

                              @encrypt1d said in pfSense compile requirements for 3rd party software:

                              Earlier it was mentioned that the src should be in /usr/src, so I created a symbolic link in /usr as follows:

                              lrwxr-xr-x   1 root  wheel  -    17 Feb 13 15:55 src@ -> /git/FreeBSD-src/
                              

                              Yet none of my build will function correctly upon starting the service on the firewall. This should just work right?

                              (pulls out hair)

                              I think your problem is that the include files on your build system do not match those used by pfSense. Let me explain.

                              The way the pfSense team builds their image is on a custom builder machine. That machine has their customized FreeBSD kernel along with its include files. The packages repository is built within a Poudriere jail on this builder machine. The Poudriere jail installs the pfSense FreeBSD source tree including its header files. Everything happens in that jail. But the key thing here is that pfSense itself (the operating system as it were for pfSense) is built from the FreeBSD source tree. Packages, on the other hand, are built from the source code referenced in the Makefile and the header include files in /usr/include within the Poudriere jail.

                              But you are building outside of the jail, so the files that actually get used are the local headers (include files, really) installed on YOUR machine. These are in /usr/include. Those are the files I'm betting do not match up with the ones in the pfSense build tree. Don't get too hung up on the /usr/src tree since your build is not really using that. The files in there are for building the kernel, but you are not building the kernel. You are building a package. Packages need to have the correct include files in the /usr/include tree on the box where the packages are being built.

                              E 1 Reply Last reply Reply Quote 1
                              • E
                                encrypt1d @bmeeks
                                last edited by encrypt1d

                                @bmeeks

                                Packages need to have the correct include files in the /usr/include tree on the box where the packages are being built.

                                I tried the following two things based on that advice:

                                1. Outright replacement of /usr/include with only the contents of the pfSense
                                  /git/FreeBSD-src/include. This resulting in missing headers, and compile failure.

                                2. I set /usr/include set back to original content, then copied contents of the pfSense repo /git/FreeBSD-src/include into /usr/include (effectively a merge over top of the original). This compiles, links, and runs on the firewall, but with the same ioctl errors.

                                3. Copied all of the pfSense git repo contents (FreeBSD-src) into /usr. Build works, same ioctl issues.

                                Am I chasing my tail trying to do this outside a poudriere jail?

                                bmeeksB 1 Reply Last reply Reply Quote 1
                                • bmeeksB
                                  bmeeks @encrypt1d
                                  last edited by bmeeks

                                  @encrypt1d said in pfSense compile requirements for 3rd party software:

                                  @bmeeks

                                  Packages need to have the correct include files in the /usr/include tree on the box where the packages are being built.

                                  I tried the following two things based on that advice:

                                  1. Outright replacement of /usr/include with only the contents of the pfSense
                                    /git/FreeBSD-src/include. This resulting in missing headers, and compile failure.

                                  2. I set /usr/include set back to original content, then copied contents of the pfSense repo /git/FreeBSD-src/include into /usr/include (effectively a merge over top of the original). This compiles, links, and runs on the firewall, but with the same ioctl errors.

                                  3. Copied all of the pfSense git repo contents (FreeBSD-src) into /usr. Build works

                                  Am I chasing my tail trying to do this outside a poudriere jail?

                                  I think your experiement with the header files proves there are some differences in your build environment versus what the pfSense image and package builder uses.

                                  You can create your own pfSense build system. I have one I use when testing my package changes for Snort and Suricata. It won't successfully build a pfSense kernel or install image, but it builds packages just fine that I then install over on my pfSense virtual machine test boxes.

                                  I will give you the overall high-level steps. Warning -- this endeavor is not for everyone! It will very likely take some fiddling around to get things working. There is no great documentation of the steps (at least that I've found).

                                  NOTE: in the steps below I've given the full URL of the repo branch on GitHub. When you actually run the clone command, you will need to use the *.git file instead, then switch to the appropriate branch when building. So https://github.com/pfsense/pfsense.git

                                  1. Create a new directory on your builder. Clone this GitHub repo into a directory on your builder machine: https://github.com/pfsense/pfsense/tree/RELENG_2_5_2. I chose /usr/home/pfsense for my directory. Change into the directory you created and then clone the repo there. So when the clone completes, if you used my example path, you will have a /usr/home/pfsense/pfsense directory full of the PHP source code and various build configuration things in that last pfsense subdirectory.

                                  2. Next you need to clone the FreeBSD-ports repo. So using my example path, you would change into the /usr/home/pfsense directory and clone this repo: https://github.com/pfsense/FreeBSD-ports/tree/RELENG_2_5_2. This will create a ports tree of all the pfSense packages in /usr/home/pfsense/FreeBSD-ports.

                                  3. You now need to create a builder.conf file in the top-level of that final pfsense directory created by the first cloning step. There is a sample conf file there already called builder.conf.sample that you can copy from and customize.

                                  4. Make sure you are in that final pfsense directory and then issue this command to begin the setup of your builder environment:

                                  ./build.sh --setup
                                  

                                  Hopefully that runs to a successful completion. If not, you will need to troubleshoot using any error messages that print.

                                  1. Next you need to run the routine to create the Poudriere jail. This will take a long time depending on your hardware. On my modest virtual machine builders it takes over 4 hours. Here is the command:
                                  ./build.sh --setup-poudriere
                                  

                                  When that finishes, you should have a functional builder jail environment. To build the initial package tree run:

                                  ./build.sh --update-pkg-repo -a amd64.amd64
                                  

                                  That will build all the packages for the Intel/AMD architecture and store them in /usr/local/poudriere/packages.

                                  E 1 Reply Last reply Reply Quote 1
                                  • E
                                    encrypt1d @bmeeks
                                    last edited by

                                    @bmeeks
                                    I will give it a try, thanks.

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @encrypt1d
                                      last edited by bmeeks

                                      @encrypt1d said in pfSense compile requirements for 3rd party software:

                                      @bmeeks
                                      I will give it a try, thanks.

                                      If you get a working stock package builder, then the miniupnpd package should build and install on a pfSense firewall (of the same base pfSense version as the builder where the package was created).

                                      I know that it works, because I build Snort and Suricata binary packages (and the PHP GUI parts as well) in my package builder all the time. I even upload them to a web server that serves as a pkg repository so I can install my packages from SYSTEM > PACKAGE MANAGER in pfSense. I do that by adding an additional repo that references my local web server to the pkg repo configuration. I do that so I can test all phases of my packages including installation, removal, and updating.

                                      E 1 Reply Last reply Reply Quote 1
                                      • E
                                        encrypt1d @bmeeks
                                        last edited by encrypt1d

                                        @bmeeks

                                        In the build.conf file,
                                        Should these lines point to the FreeBSD official, or the pfSense version, ie should it be this:

                                        # Define FreeBSD repository, branch and specific commit
                                        export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
                                        export FREEBSD_BRANCH=RELENG_2_5_2
                                        

                                        or this:

                                        # Define FreeBSD repository, branch and specific commit
                                        export FREEBSD_REPO_BASE=https://github.com/freebsd/freebsd.git
                                        export FREEBSD_BRANCH=stable/10
                                        
                                        bmeeksB 1 Reply Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks @encrypt1d
                                          last edited by bmeeks

                                          @encrypt1d said in pfSense compile requirements for 3rd party software:

                                          @bmeeks

                                          In the build.conf file,
                                          Should these lines point to the FreeBSD official, or the pfSense version, ie should it be this:

                                          # Define FreeBSD repository, branch and specific commit
                                          export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
                                          export FREEBSD_BRANCH=RELENG_2_5_2
                                          

                                          or this:

                                          # Define FreeBSD repository, branch and specific commit
                                          export FREEBSD_REPO_BASE=https://github.com/freebsd/freebsd.git
                                          export FREEBSD_BRANCH=stable/10
                                          

                                          The FREEBSD_BRANCH line should point to the pfSense branch. So you have it right in the first example: RELENG_2_5_2.

                                          But as of just a little while ago, that is now RELENG_2_6_0 as they released 2.6.0 CE and 22.01 pfSense Plus this morning. So be careful and keep the branch in the builder synced up with the branch installed on your firewall.

                                          And if you swap branches, you will likely need to rebuild the FreeBSD jail used by Poudriere. Run the shell script with no arguments to see all the command options like this:

                                          ./build.sh
                                          

                                          One of the options is to update the poudriere jails.

                                          E 1 Reply Last reply Reply Quote 1
                                          • E
                                            encrypt1d @bmeeks
                                            last edited by

                                            @bmeeks

                                            Is it a requirement to be running a local repo server?

                                            Seems like it might be - I see in the logs it is trying to access online content to my "nonSense" build. There doesn't seem to be an option to disable that.

                                            pdating nonSense-core repository catalogue...
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/meta.txz: Forbidden
                                            repository nonSense-core has no meta file, using default settings
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.pkg: Forbidden
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.txz: Forbidden
                                            Unable to update repository nonSense-core
                                            Updating nonSense repository catalogue...
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/meta.txz: Forbidden
                                            repository nonSense has no meta file, using default settings
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.pkg: Forbidden
                                            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.txz: Forbidden
                                            Unable to update repository nonSense
                                            Error updating repositories!
                                            
                                            bmeeksB 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.