Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense compile requirements for 3rd party software

    Scheduled Pinned Locked Moved Development
    102 Posts 8 Posters 27.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeksB
      bmeeks @encrypt1d
      last edited by bmeeks

      @encrypt1d said in pfSense compile requirements for 3rd party software:

      @bmeeks

      In the build.conf file,
      Should these lines point to the FreeBSD official, or the pfSense version, ie should it be this:

      # Define FreeBSD repository, branch and specific commit
      export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
      export FREEBSD_BRANCH=RELENG_2_5_2
      

      or this:

      # Define FreeBSD repository, branch and specific commit
      export FREEBSD_REPO_BASE=https://github.com/freebsd/freebsd.git
      export FREEBSD_BRANCH=stable/10
      

      The FREEBSD_BRANCH line should point to the pfSense branch. So you have it right in the first example: RELENG_2_5_2.

      But as of just a little while ago, that is now RELENG_2_6_0 as they released 2.6.0 CE and 22.01 pfSense Plus this morning. So be careful and keep the branch in the builder synced up with the branch installed on your firewall.

      And if you swap branches, you will likely need to rebuild the FreeBSD jail used by Poudriere. Run the shell script with no arguments to see all the command options like this:

      ./build.sh
      

      One of the options is to update the poudriere jails.

      E 1 Reply Last reply Reply Quote 1
      • E
        encrypt1d @bmeeks
        last edited by

        @bmeeks

        Is it a requirement to be running a local repo server?

        Seems like it might be - I see in the logs it is trying to access online content to my "nonSense" build. There doesn't seem to be an option to disable that.

        pdating nonSense-core repository catalogue...
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/meta.txz: Forbidden
        repository nonSense-core has no meta file, using default settings
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.pkg: Forbidden
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.txz: Forbidden
        Unable to update repository nonSense-core
        Updating nonSense repository catalogue...
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/meta.txz: Forbidden
        repository nonSense has no meta file, using default settings
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.pkg: Forbidden
        pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.txz: Forbidden
        Unable to update repository nonSense
        Error updating repositories!
        
        bmeeksB 1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks @encrypt1d
          last edited by

          @encrypt1d said in pfSense compile requirements for 3rd party software:

          @bmeeks

          Is it a requirement to be running a local repo server?

          Seems like it might be - I see in the logs it is trying to access online content to my "nonSense" build. There doesn't seem to be an option to disable that.

          pdating nonSense-core repository catalogue...
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/meta.txz: Forbidden
          repository nonSense-core has no meta file, using default settings
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.pkg: Forbidden
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/packagesite.txz: Forbidden
          Unable to update repository nonSense-core
          Updating nonSense repository catalogue...
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/meta.txz: Forbidden
          repository nonSense has no meta file, using default settings
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.pkg: Forbidden
          pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-nonSense_v2_6_0/packagesite.txz: Forbidden
          Unable to update repository nonSense
          Error updating repositories!
          

          I think you can just comment out those lines in builder.conf by placing a "#" character at the start of each line. These little issues are what I meant when I said getting a builder configured takes a lot of patience. It is not a simple click, click, done sort of thing.

          Here is my builder.conf file from my RELEASE builder. I have two: one for DEVEL and another for RELEASE.

          # Define FreeBSD repository, branch and specific commit
          export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
          export FREEBSD_BRANCH="RELENG_2_5_2"
          #export GIT_FREEBSD_COSHA1=##HASH##
          
          # Do not clean FreeBSD obj dirs
          #export NO_CLEAN_FREEBSD_OBJ=YES
          
          # Do not clean FreeBSD src dir
          #export NO_CLEAN_FREEBSD_SRC=YES
          
          # rsync data to upload snapshots
          #export RSYNCIP="your.snapshot.server"
          #export RSYNCUSER="username"
          #export RSYNCPATH="/usr/local/www/snapshots"
          #export RSYNCLOGS="/usr/local/www/snapshots/logs"
          
          # rsync data to upload pkg repo
          export PKG_RSYNC_HOSTNAME="vm-pfpackages.themeeks.net"
          export PKG_RSYNC_USERNAME="root"
          export PKG_RSYNC_SSH_PORT="22"
          export PKG_RSYNC_DESTDIR="/usr/local/www/apache22/data/packages"
          
          export STAGING_HOSTNAME="pfsense-builder-RELEASE.themeeks.net"
          
          # Set these when doing RELEASE package builds
          #export PKG_FINAL_RSYNC_HOSTNAME="vm-pfpackages.themeeks.net"
          #export PKG_FINAL_RSYNC_DESTDIR="/usr/local/www/apache22/data/packages"
          
          # pkg repo URL
          export PKG_REPO_SERVER_DEVEL="pkg+https://beta.pfsense.org/packages"
          export PKG_REPO_SERVER_RELEASE="pkg+https://pkg.pfsense.org"
          export PKG_REPO_SERVER_STAGING="pkg+https://pkg.pfsense.org"
          
          # Command used to sign pkg repo
          export PKG_REPO_SIGN_KEY="/root/repo.key"
          export PKG_REPO_SIGNING_COMMAND="/root/repo_sign.sh ${PKG_REPO_SIGN_KEY}"
          
          # Added for private repo to skip rsync to Netgate RELEASE server
          export SKIP_FINAL_RSYNC=YES
          
          # Branch to replace pkg.conf template, defaults to $GIT_REPO_BRANCH_OR_TAG
          #export PKG_REPO_BRANCH_DEVEL="BRANCH"
          #export PKG_REPO_BRANCH_RELEASE="v2_3_2"
          
          E 1 Reply Last reply Reply Quote 2
          • E
            encrypt1d @bmeeks
            last edited by

            @bmeeks

            getting a builder configured takes a lot of patience. It is not a simple click, click, done sort of thing.

            I am committed to getting it to work, and I am very appreciative of all the help.

            Indeed I started with all of the repo lines commented out, yet it is still trying to access that online content. No matter what I do in the build.conf file, it tries to access a non-existent repo and then defaults out.

            Updating nonSense-core repository catalogue...
            pkg: http://release-staging.nyi.netgate.com/ce/packages/nonSense_v2_6_0_amd64-core/meta.txz: Forbidden
            repository nonSense-core has no meta file, using default settings
            

            My build.conf is as follows:

            export PRODUCT_NAME="nonSense"
            #export PRODUCT_URL="https://PRODUCT_URL/"
            
            # Define FreeBSD repository, branch and specific commit
            #export FREEBSD_REPO_BASE=https://github.com/freebsd/freebsd.git
            #export FREEBSD_BRANCH=stable/10p
            #export GIT_FREEBSD_COSHA1=##HASH##
            export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
            export FREEBSD_BRANCH=RELENG_2_5_2
            
            # Do not clean FreeBSD obj dirs
            #export NO_CLEAN_FREEBSD_OBJ=YES
            
            # rsync data to upload snapshots
            #export RSYNCIP="your.snapshot.server"
            #export RSYNCUSER="username"
            #export RSYNCPATH="/usr/local/www/snapshots"
            
            # rsync data to upload pkg repo
            #export PKG_RSYNC_HOSTNAME="pkg_repo_hostname"
            #export PKG_RSYNC_USERNAME="username"
            #export PKG_RSYNC_SSH_PORT="22"
            #export PKG_RSYNC_DESTDIR="/usr/local/www/beta/packages"
            export SKIP_FINAL_RSYNC=YES
            
            # pkg repo URL
            #export PKG_REPO_SERVER_DEVEL="pkg+http://packages-beta.netgate.com/packages"
            #export PKG_REPO_SERVER_RELEASE="pkg+http://packages.netgate.com"
            #export PKG_REPO_SERVER_STAGING="pkg+http://release-staging.netgate.com/ce/packages"
            export PKG_REPO_SERVER_DEVEL="pkg+https://beta.pfsense.org/packages"
            export PKG_REPO_SERVER_RELEASE="pkg+https://pkg.pfsense.org"
            export PKG_REPO_SERVER_STAGING="pkg+https://pkg.pfsense.org"
            # Command used to sign pkg repo
            export PKG_REPO_SIGN_KEY="/root/repo.key"
            export PKG_REPO_SIGNING_COMMAND="/root/repo_sign.sh ${PKG_REPO_SIGN_KEY}"
            
            # Branch to replace pkg.conf template, defaults to $GIT_REPO_BRANCH_OR_TAG
            #export PKG_REPO_BRANCH_DEVEL="BRANCH"
            #export PKG_REPO_BRANCH_RELEASE="BRANCH"
            
            bmeeksB 1 Reply Last reply Reply Quote 1
            • bmeeksB
              bmeeks @encrypt1d
              last edited by bmeeks

              @encrypt1d:
              You may wind up having to edit/modify the shell script code in ./tools/builder_common.sh to get things to work. The code in /builder.sh simply calls functions with ./tools/builder_common.sh to do the actual job you request.

              This whole build system is geared towards Netgate building the official images and packages. It really does not work out of the box for other things without editing. It's been a very long time (years) since I set my two boxes up, so I don't recall all the hoops I jumped through. Since then, I've used other commands to stay updated without totally rebuilding my two builder machines.

              You can try changing the PRODUCT_NAME to "pfSense" as a test. I think that also means you have to toggle the "Authorized_by_Netgate" switch or else tests in the functions that are called will bail out and print a copyright warning. This logic is all designed to prevent pirates from producing "pfSense" images and then installing and selling them. However, it does cause lots of roadblocks when all you want to do is simply build packages and you have no interest in building a kernel or pfSense install image.

              E 1 Reply Last reply Reply Quote 2
              • E
                encrypt1d @bmeeks
                last edited by encrypt1d

                @bmeeks
                Using your tip about the product name really helped eliminate a lot of errors, and I had to comment out two items in the tools/builder_common.sh. One was an imaginary new package it seemed to try and install called <productname>-builder, or in my case pfSense-builder. I commented out the install command. Since I had chosen the pfSense product name it also tried to clone the GNID repo (doesn't seem to exist?), which I don't think I care about, so I commented out those lines too.

                The --setup-poudriere option errors out right away, making me think I need to do some setup work on it first. I do have it installed, but made the assumption that the option in the build.sh would do all that for me.

                Do I need to do some pre-work on poudriere first?

                >>> Operation ./build.sh has started at Sun Feb 13 20:32:13 EST 2022
                >>> ERROR: ZFS tank zroot not found, please create it and try again...
                
                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @encrypt1d
                  last edited by bmeeks

                  @encrypt1d said in pfSense compile requirements for 3rd party software:

                  @bmeeks
                  Using your tip about the product name really helped eliminate a lot of errors, and I had to comment out two items in the tools/builder_common.sh. One was an imaginary new package it seemed to try and install called <productname>-builder, or in my case pfSense-builder. I commented out the install command. Since I had chosen the pfSense product name it also tried to clone the GNID repo (doesn't seem to exist?), which I don't think I care about, so I commented out those lines too.

                  The --setup-poudriere option errors out right away, making me think I need to do some setup work on it first. I do have it installed, but made the assumption that the option in the build.sh would do all that for me.

                  Do I need to do some pre-work on poudriere first?

                  >>> Operation ./build.sh has started at Sun Feb 13 20:32:13 EST 2022
                  >>> ERROR: ZFS tank zroot not found, please create it and try again...
                  

                  Oh, yes -- forgot to tell you that your builder needs to be running on ZFS because that's what the Poudriere setup expects. So you may need to rebuild your builder machine to use ZFS instead of the default UFS. My two builders are virtual machines on ESXi, and when I created my most recent one for RELEASE package building I had to tear it down and start over because I forgot to use ZFS on the initial FreeBSD install.

                  But sounds like you found the other gotcha points. The GNID thing is proprietary code, and I commented it out as well. That part of the script is only hit during initial setup or when you run the "--update-sources" subcommand.

                  E 2 Replies Last reply Reply Quote 2
                  • E
                    encrypt1d @bmeeks
                    last edited by

                    @bmeeks

                    Cool. I have all the instructions recorded, so a rebuild isn't too big a deal.

                    E 1 Reply Last reply Reply Quote 1
                    • E
                      encrypt1d @encrypt1d
                      last edited by

                      Looking good so far. Now I wait ;)

                      root@devbox:/build/pfsense # ./build.sh --setup
                      >>> Operation ./build.sh has started at Tue Feb 15 10:11:28 EST 2022
                      >>> Obtaining FreeBSD sources (RELENG_2_5_2)...
                      ====>> Cloning git repository https://github.com/pfsense/FreeBSD-src.git (RELENG_2_5_2)
                      Skipping clone of https://github.com/pfsense/gnid.git
                      Skipping install of pfSense-builder
                      >>> Operation ./build.sh has ended at Tue Feb 15 10:25:52 EST 2022
                      root@devbox:/build/pfsense # ./build.sh --setup-poudriere
                      >>> Operation ./build.sh has started at Tue Feb 15 10:27:16 EST 2022
                      >>> Creating ZFS filesystem zroot/poudriere... Done!
                      >>> Creating poudriere.conf
                      >>> Creating jail pfSense_v2_5_2_amd64, it may take some time... 
                      
                      1 Reply Last reply Reply Quote 0
                      • E
                        encrypt1d @bmeeks
                        last edited by

                        @bmeeks
                        While that is running, what are the next steps beyond the jail setup in order to build a specific port package?

                        bmeeksB 1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks @encrypt1d
                          last edited by bmeeks

                          @encrypt1d said in pfSense compile requirements for 3rd party software:

                          @bmeeks
                          While that is running, what are the next steps beyond the jail setup in order to build a specific port package?

                          I've never found the magic word for building a specific port. That's because the Netgate shell script is geared toward building the whole repository.

                          However, once you build the first round (which will take quite some time, especially the Rust port), then subsequent rounds will only build the packages that have been changed. Therefore, so long as you don't "refresh" your local Poudriere ports tree, none of the other packages will need to be rebuilt. You can just change your miniupnpd package and then when you run this:

                          ./build.sh --update-pkg-repo -a amd64.amd64
                          

                          it will only rebuild miniupnpd if that's the only package with a change. For your troubleshooting steps, assuming you will want to rebuild your package several times, you will need to delete the package from /usr/local/poudriere/data/packages/.. so Poudriere will see it missing and build it again. Either that, or in the Makefile for miniupnpd you can increment the version number to signal Poudriere to rebuild the port. In your case bumping the version number in the Makefile might be the fastest method.

                          Then when you have it working like you want, you can change the version number back to maybe just one past the current one in pfSense, delete the package from the path I mentioned above, then build it one last time.

                          E 1 Reply Last reply Reply Quote 2
                          • E
                            encrypt1d @bmeeks
                            last edited by

                            @bmeeks

                            I will still be able to use the patching technique @jimp suggested? I.e make extract, change files, make patch?

                            bmeeksB 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @encrypt1d
                              last edited by bmeeks

                              @encrypt1d said in pfSense compile requirements for 3rd party software:

                              @bmeeks

                              I will still be able to use the patching technique @jimp suggested? I.e make extract, change files, make patch?

                              Yes, that technique should still work to create the patch file or files. One convention commonly used is to consolidate all the changes for a given bug fix or feature add into a single patch file. Then you name the patch file something that aligns with what the patch does. So if you are adding a new feature or fixing some specific bug, then put all the patched files into a single *.diff patch and name it something that reflects the new feature added or the bug fixed.

                              E 1 Reply Last reply Reply Quote 1
                              • E
                                encrypt1d @bmeeks
                                last edited by

                                @bmeeks
                                The jail setup completed successfully. Apparently your machine is faster than my virtual machine, ouch. 11 hrs.

                                root@devbox:/build/pfsense # ./build.sh --setup-poudriere
                                >>> Operation ./build.sh has started at Tue Feb 15 10:27:16 EST 2022
                                >>> Creating ZFS filesystem zroot/poudriere... Done!
                                >>> Creating poudriere.conf
                                >>> Creating jail pfSense_v2_5_2_amd64, it may take some time... 
                                Done!
                                >>> Creating poudriere ports tree, it may take some time... Done!
                                >>> Poudriere is now configured!
                                >>> Operation ./build.sh has ended at Tue Feb 15 21:16:43 EST 2022
                                

                                Is my next step to run this?

                                ./build.sh none
                                

                                I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

                                bmeeksB 1 Reply Last reply Reply Quote 0
                                • bmeeksB
                                  bmeeks @encrypt1d
                                  last edited by bmeeks

                                  @encrypt1d said in pfSense compile requirements for 3rd party software:

                                  Is my next step to run this?

                                  ./build.sh none
                                  

                                  I kicked that off (and sorted out all the missing packages it needed) and would prefer to know if that isn't the right command before I let it run too long to find out otherwise.

                                  I run this command to build the package tree:

                                  ./build.sh --update-pkg-repo -a amd64.amd64
                                  

                                  That will build all the packages for AMD64/Intel architectures. You can just execute the shell script with no arguments to see all the available options like this:

                                  ./build.sh
                                  

                                  I just rebuilt the jail in my RELEASE builder since I had to update it to the latest FreeBSD 12.3 with the recent 2.6.0 pfSense release, and it took about as long as yours. Much longer than I remembered from the past.

                                  E 1 Reply Last reply Reply Quote 1
                                  • E
                                    encrypt1d @bmeeks
                                    last edited by

                                    @bmeeks

                                    Thanks again, it appears to be running.

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @encrypt1d
                                      last edited by

                                      @encrypt1d said in pfSense compile requirements for 3rd party software:

                                      @bmeeks

                                      Thanks again, it appears to be running.

                                      Some of the packages will take a long time to build, particularly the Rust language one. Just be patient. On my builder VM, Rust takes over 4 hours by itself. And it needs lots of RAM (more than 8 GB). What I did was create a very large extra swap partition for it to use during that build. Makes it take longer, but my ESXi host only has 32 GB of RAM to share among my active VMs.

                                      If Rust fails to build, it will probably be due to resource limits. If that happens, simply kick off the same build again (use the same command) and the process will pick up where it left off. It is smart enough not to rebuild everything from scratch every time.

                                      So if you see any package fail during the process, just let it keep going until the job stops. Then you can run it again to rebuild any failures. Like I said, most times the failures are due to resource exhaustion in my minimal builder VM.

                                      E 1 Reply Last reply Reply Quote 1
                                      • E
                                        encrypt1d @bmeeks
                                        last edited by

                                        @bmeeks

                                        That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

                                        bmeeksB 1 Reply Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks @encrypt1d
                                          last edited by bmeeks

                                          @encrypt1d said in pfSense compile requirements for 3rd party software:

                                          @bmeeks

                                          That's good to know, I only gave the VM 8 gig, so I will expect the failures. So first build is a multiday event by the sound of it? ;)

                                          Possibly, or at least a very long first day. After that, assuming you don't change any of the other ports' source code, each time you kick off a build it will only build your miniupnpd package (if you change the Makefile version) and a couple of small pfSense packages that are rebuilt on each run (these have timestamp info for the current build).

                                          E 1 Reply Last reply Reply Quote 1
                                          • E
                                            encrypt1d @bmeeks
                                            last edited by

                                            @bmeeks

                                            I've narrowed the build down to just 4 errors now, almost there.

                                            [00:00:31] [01] [00:00:03] Finished databases/sqlite3@default | sqlite3-3.35.5_1,1: Failed: fetch
                                            [00:03:15] [01] [00:02:44] Finished databases/mysql57-client | mysql57-client-5.7.34: Failed: fetch
                                            [00:37:23] [01] [00:00:03] Finished net-mgmt/zabbix52-agent | zabbix52-agent-5.2.6: Failed: fetch
                                            [00:38:53] [01] [00:01:30] Finished security/stunnel | stunnel-5.59,1: Failed: fetch
                                            

                                            The logs are showing 404 errors on the downloads. Seems like these aren't being hosted anymore. Any suggestions on what to do abut this?

                                            bmeeksB 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.