Barnyard2 and MariaDB



  • Is there a patch (or will there be a patch/upgrade) to deal with the Barnyard2/MariaDB issue of needing backticks to properly reference ref_system_id?

    I've looked all over the place with the assistance of Dr. Google, and all I can find is that a bug report has been filed, or that there's a patch but you have to clone a git repository and compile a bunch of stuff on your pfSense install. Frankly, not really cool with doing all that and possibly borking my 2 firewalls beyond belief.

    TiA



  • Barnyard2 has not been materially updated in FreeBSD ports in several years. By "materially" I mean actual changes to the C source code. There have been some minor tweaks to the package code done by the FreeBSD ports folks to keep the package compiling, but there have been no updates to the core source code to fix any Barnyard2 bugs (or to add new features).

    The Snort and Suricata packages simply tag the FreeBSD ports version of Barnyard2 as a runtime dependency so that during installation of Snort or Suricata the pkg utility pulls in and installs Barnyard2 as well.

    I say this to imply that any fixes to Barnyard2 appear unlikely unless some new maintainer takes over the FreeBSD port of Barnyard2.



  • Thanks.

    Short of picking up the Barnyard2 code myself (which I'm so rusty on C I couldn't do anyway), any suggestions?



  • No, not really. Suricata is going to discontinue unified2 binary logging in a coming future version, and Snort3 is headed that way as well with everybody switching over to JSON logging options. So the Barnyard2 tab is going to disappear from Suricata at some point in the near future. I suspect the same may happen with Snort a bit later. It depends on exactly what the Snort team finalizes in Snort3.

    So my best recommendation would be to start working on something based on ELK or Graylog that can injest JSON logs. Right now Snort 2.9.x can't produce JSON logs, but Snort3 will. Suricata can already produce loads of JSON logs.



  • I'm new to pfSense. I was bitten by the same bug. See post here.

    If I want to replace Barnyard2 binary manually, are there any doc on how to setup build env for pfSense port?

    Mine is SG-3100 which runs on ARMV7. Where can I find the cross-compile build tool stack?

    Thanks in advance.



  • @rickyzhang said in Barnyard2 and MariaDB:

    I'm new to pfSense. I was bitten by the same bug. See post here.

    If I want to replace Barnyard2 binary manually, are there any doc on how to setup build env for pfSense port?

    Mine is SG-3100 which runs on ARMV7. Where can I find the cross-compile build tool stack?

    Thanks in advance.

    Building armv7 binaries is very difficult now in pfSense unless you have native armv7 hardware to construct a poudriere builder on. There are some packages in the armv7 tree that won't build on qemu, the current emulator used in FreeBSD to do cross compiles. One of these is the Go language (if I recall correctly). That language is a dependency in several FreeBSD packages. So while it is theoretically possible to build most of the armv7 binaries using qemu in a FreeBSD poudriere builder, it would take a lot of trial and error and the disabling of a number of the packages that either don't build in qemu anymore, or that themselves depend on packages that won't build using qemu. So a lot of "ifs".

    My suggestion is to forget Barnyard2 in its current state. It has not been maintained in FreeBSD for the last several years, and thus probably has a lot of issues with the newer DB versions out now for MySQL and others.

    If you want to try anyway, then construct a FreeBSD machine using FreeBSD 12.0. A virtual machine works nicely. Next, clone the two pfSense repos from Github onto the FreeBSD machine.

    1. https://github.com/pfsense/pfsense
    2. https://github.com/pfsense/FreeBSD-ports

    After you clone the two repos, there is a build.sh script in pfsense/pfsense/tools that you can use to perform various tasks such as creating a Ports tree for pfSense-DEVEL and a corresponding Poudriere builder. It will take quite a bit of looking through the build.sh shell script and the other shell files it includes in order to figure out what to do in order to get packages building. There is basically no documentation. You will need to be rather skilled in building packages on FreeBSD using Poudriere (or else you will need to get up-to-speed yourself).



  • @bmeeks

    Thanks for your advice.

    Actually, the upstream Barnyard2 in Github has patched the MariaDB syntax issue. (MariaDB is not 100% compatible with MySQL now as I tried both in containers.) The fix seems easy to apply. The obstacle as you said is the build.

    Missing the documentation to setup dev env is the first problem. ARM cross platform build is the second one. But if I can get amd64 done, I can get the ARMV7 SD image and replicate it in QEMU.

    I do want to try to build the whole pfSense. I'm more familiar with Macport in Mac OS X where documentation is well written.

    After my skimming through the Barnyard2 port from pfSense Git repo(https://github.com/pfsense/FreeBSD-ports/tree/devel/security/barnyard2), I found that it only contains couple patch files and a Makefile.

    I don't know how can it get the original source code. It didn't have the whole copy of Barnyard2 source code or its repo code URL to pull from anywhere.

    Should I use FreeBSD 11.2 which match pfSense I use rather than FreeBSD 12.0? The FreeBSD port documentation doesn't explain how to replace official port with bespoke port tree like pfSense. That make me concerned if I should spend time on this project.



  • @rickyzhang said in Barnyard2 and MariaDB:

    @bmeeks

    Thanks for your advice.

    Actually, the upstream Barnyard2 in Github has patched the MariaDB syntax issue. (MariaDB is not 100% compatible with MySQL now as I tried both in containers.) The fix seems easy to apply. The obstacle as you said is the build.

    Missing the documentation to setup dev env is the first problem. ARM cross platform build is the second one. But if I can get amd64 done, I can get the ARMV7 SD image and replicate it in QEMU.

    I do want to try to build the whole pfSense. I'm more familiar with Macport in Mac OS X where documentation is well written.

    After my skimming through the Barnyard2 port from pfSense Git repo(https://github.com/pfsense/FreeBSD-ports/tree/devel/security/barnyard2), I found that it only contains couple patch files and a Makefile.

    I don't know how can it get the original source code. It didn't have the whole copy of Barnyard2 source code or its repo code URL to pull from anywhere.

    Should I use FreeBSD 11.2 which match pfSense I use rather than FreeBSD 12.0? The FreeBSD port documentation doesn't explain how to replace official port with bespoke port tree like pfSense. That make me concerned if I should spend time on this project.

    You can create a FreeBSD 11.2 virtual machine and then create your own builder system. You don't have to use the pfSense builder system. In fact, you will find some things missing from the pfSense builder if you intend to build an actual pfSense image. You will need to go into the shell script files and comment out several things in order to get a build of pfSense itself to actually complete, and even then some of the kernel patches will be missing. One item you will have to comment out is anything having to do with the GNID stuff (unique Netgate ID). Building packages for AMD64, on the other hand, is easy. Building pfSense itself, not so much.

    You will find the source code URL inside the Makefile included with each FreeBSD port. So build a FreeBSD 11.2 machine, replicate a FreeBSD ports tree to it (portsnap auto) and then you can examine the Barnyard2 build files. Inside the Makefile you will find the source code URL. Barnyard2 will be in /usr/ports/security/barnyard2. The URL will be shown on the line for "MASTER_SITES=". The first time you build Barnyard2, it will automatically pull down the source tarball and store it in /usr/ports/distfiles. It appears that the port maintainer for Barnyard2 on FreeBSD has not ported any of the Github updates over to the FreeBSD port in quite some time. That's what I meant by not being maintained anymore. The few changes that have been made were only to keep Barnyard2 building as the FreeBSD ports tree changed. There have been no changes to the actual Barnyard2 C source code files -- just changes to the Makefile to alter build dependencies.



  • @bmeeks said in Barnyard2 and MariaDB:

    /usr/ports/security/barnyard2

    I wrote a doc/note on how to setup FreeBSD dev env as your instructed. (Perhaps, I should find a wiki to place it your knowledge somewhere to share it)

    See link here: https://github.com/rickyzhang82/PiBa-NL-WIKI/wiki/Setup-FreeBSD-Development-Environment-for-pfSense-port

    After running portsnap auto, I check the Makfile in /usr/ports/security/barnyard2. There is no "MASTER_SITES=" in this Makefile like the rest of port does. For this, I will read the FreeBSD port doc and see if I can pull the source code and build it.

    But here is the main roadblock: how can I use pfSense port (https://github.com/pfsense/FreeBSD-ports) rather than the one from FreeBSD?

    For now I don't want to build the whole pfSense port but rather the one port only.

    BTW: The build.sh has gone from pfsense/pfsense/tools folder (see https://github.com/pfsense/pfsense/tree/master/tools)



  • My apologies on misleading you with the "MASTER_SITES=" line. That is the way most ports do it, but Barnyard2 is different. It's using a Github technique contained within these two lines:

    USE_GITHUB=	yes
    GH_ACCOUNT=	firnsy
    

    It's been quite some time since I've looked at the Makefile for Barnyard2.



  • @rickyzhang said in Barnyard2 and MariaDB:

    @bmeeks said in Barnyard2 and MariaDB:
    But here is the main roadblock: how can I use pfSense port (https://github.com/pfsense/FreeBSD-ports) rather than the one from FreeBSD?

    For now I don't want to build the whole pfSense port but rather the one port only.

    BTW: The build.sh has gone from pfsense/pfsense/tools folder (see https://github.com/pfsense/pfsense/tree/master/tools)

    The pfsense/pfsense/tools path was assuming you had cloned the Github repo onto a FreeBSD machine. That will be the path on your FreeBSD machine after the clone operation. And I was operating from memory, so I think I remembered the path wrong. build.sh is in pfsense/pfsense, but the main file containing all the function calls is in the pfsense/pfsense/tools path.

    I recommend creating the directory /usr/home/pfsense and then cloning the two repos I linked earlier into that path on your FreeBSD machine. There will be multiple branches within each repo. DEVEL is for pfSense-2.5 and RELENG_2.4.4 is for pfSense-2.4.

    After cloning, if you change to the pfsense/pfsense directory and execute the following commands, the script should build the Poudriere jails for you.

    ./build.sh --setup
    ./build.sh --setup-poudriere
    


  • @bmeeks

    I run make in /usr/ports/security/barnyard2. I starts to build all dependencies and pulling its own code.

    See below:

    ===>  License GPLv2 accepted by the user
    ===>   barnyard2-1.13_3 depends on file: /usr/local/sbin/pkg - found
    => firnsy-barnyard2-v2-1.13_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
    => Attempting to fetch https://codeload.github.com/firnsy/barnyard2/tar.gz/v2-1.13?dummy=/firnsy-barnyard2-v2-1.13_GH0.tar.gz
    fetch: https://codeload.github.com/firnsy/barnyard2/tar.gz/v2-1.13?dummy=/firnsy-barnyard2-v2-1.13_GH0.tar.gz: No address record
    => Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/firnsy-barnyard2-v2-1.13_GH0.tar.gz
    firnsy-barnyard2-v2-1.13_GH0.tar.gz           100% of  424 kB 1324 kBps 00m00s
    ===> Fetching all distfiles required by barnyard2-1.13_3 for building
    ===>  Extracting for barnyard2-1.13_3
    => SHA256 Checksum OK for firnsy-barnyard2-v2-1.13_GH0.tar.gz.
    ===>  Patching for barnyard2-1.13_3
    ===>  Applying FreeBSD patches for barnyard2-1.13_3
    ===>   barnyard2-1.13_3 depends on package: autoconf>=2.69 - not found
    ===>  License GPLv2+ GPLv3+ GFDL AUTOCONF_CONFIGURE_SCRIPT_EXCEPTION accepted by the user
    ===>   autoconf-2.69_3 depends on file: /usr/local/sbin/pkg - found
    => autoconf-2.69.tar.xz doesn't seem to exist in /usr/ports/distfiles/.
    => Attempting to fetch https://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz
    fetch: https://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz: Authentication error
    => Attempting to fetch https://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz
    fetch: https://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz: Authentication error
    => Attempting to fetch ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz
    autoconf-2.69.tar.xz                          100% of 1186 kB 2416 kBps 00m00s
    ===> Fetching all distfiles required by autoconf-2.69_3 for building
    ===>  Extracting for autoconf-2.69_3
    => SHA256 Checksum OK for autoconf-2.69.tar.xz.
    ===>  Patching for autoconf-2.69_3
    ===>  Applying FreeBSD patches for autoconf-2.69_3
    ===>   autoconf-2.69_3 depends on executable: gm4 - not found
    
    

    The source code indicate its from the upstream. It supposed what I read in Github with the MariaDB syntax patch. I will take a close look.

    But I think I do this build exercise in the wrong way. I have to manfully hit my keyboard so many times to answer gettex build option dialog during building of all its dependencies.



  • @rickyzhang :
    Yes, FreeBSD has a lot of "interactive" prompts that require hitting "Y" or ENTER. I hate it, but have never found a way around all of them, just some of them. The good news is that it will save your answers and use them from now on without prompting you again.



  • I built the Barnyard2 binary successfully from FreeBSD official port by running make in /usr/ports/security/barnyard2 folder. I think the same approach can be applied to pfSense's FreeBSD-ports by running make in cloned Git repo.

    Since I already clone both pfsense repo and FreeBSD-ports repo from Github under pfsense folder. See the folder structure below:

    [Ricky@freebsd ~/repo/github/pfsense]$ pwd
    /home/Ricky/repo/github/pfsense
    
    [Ricky@freebsd ~/repo/github/pfsense]$ ls
    FreeBSD-ports	pfsense
    

    I don't see build.sh script from pfsense/pfsense folder will use FreeBSD-ports repo cloned by me. In any case, I will give this a try if I fail.

    Missing build instruction is a real pain in the ass.



  • After building Barnyard2 from by running make in /usr/ports/security/barnyard2 folder for 20 mins, it took me less than a few seconds to build it from FreeBSD-ports under my cloned repo /home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2.

    I think for now I can start to debug the issue. But it is better that I can just build ARMv7 binary, instead amd64.

    Now I have to start to install pfSense in amd64 VM and test it. What a pain in the ass.



  • @rickyzhang :
    The build.sh script is not part of the FreeBSD-ports repo. It is part of the pfSense repo. That's why I gave you two links originally to clone. Here is a screenshot from my builder showing the build.sh script file.

    Path.png

    You need to fill in the proper values in pfsense/pfsense/build.conf and then execute the build.sh commands I posted earlier. That will create a proper environment for building pfSense packages using Poudriere. If you build packagea outside of that environment, you will likely find that they fail to load and run on your pfSense firewall due to path problems.



  • @rickyzhang said in Barnyard2 and MariaDB:

    /home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2

    I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running make. I may not need to use your approach.

    Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.

    BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.

    Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.



  • @rickyzhang said in Barnyard2 and MariaDB:

    @rickyzhang said in Barnyard2 and MariaDB:

    /home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2

    I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running make. I may not need to use your approach.

    Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.

    BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.

    Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.

    I don't want to share my entire build.conf file because it contains some sensitive items. But the main two parameters you need to set are these:

    # Define FreeBSD repository, branch and specific commit
    export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git
    export FREEBSD_BRANCH=RELENG_2_5
    

    In my case, I am building packages for DEVEL, so I use RELENG_2_5. If you want to build packages for pfSense-RELEASE, then you would use RELENG_2_4_4. You can find the relevant branches by looking at the branches available on Github for FreeBSD-ports in pfSense.

    Also I hope you realize that unless you do make pkg during your build, the binary you produce likely won't run on a pfSense firewall. You need to use the make pkg command in order to produce a package file that you can install using pkg on pfSense itself.

    And if you want to actually run your binary on your SG-3100 firewall, you must build it either under the qemu emulator environment or else create your FreeBSD builder on native ARM hardware.



  • Are you suggesting that I can't swap the binary file due to some security features like signature signing on the binary? That's new to me. I will look into make pkg command.

    Yes, I'm working on how to emulate FreeBSD on ARMV7 now. All ARM board I got either too slow like BeagleBone or 64 bit like RPI 4 or even they can't run FreeBSD at all like odroid XU4.

    I thought their build should work under corss-compile. But I can only saw some qemu string pops up in their build script (https://github.com/pfsense/pfsense/blob/master/tools/builder_common.sh)



  • @rickyzhang :
    My experience when trying to transfer and run a binary built on FreeBSD but outside of the pfSense Poudriere builder structure is that the binary would fail to run because the various paths (/usr/local/bin, /usr/etc, etc.) would be incorrect. Also would get various library loading failures. There are probably solutions to all of those, but I just found it easier to use the Poudriere environment within the pfSense builder structure.



  • @rickyzhang said in Barnyard2 and MariaDB:

    Yes, I'm working on how to emulate FreeBSD on ARMV7 now. All ARM board I got either too slow like BeagleBone or 64 bit like RPI 4 or even they can't run FreeBSD at all like odroid XU4.

    I thought their build should work under corss-compile. But I can only saw some qemu string pops up in their build script (https://github.com/pfsense/pfsense/blob/master/tools/builder_common.sh)

    If you run things on native ARM hardware, you don't need to "emulate" anything. There is a FreeBSD image for ARM hardware just like there is an image for AMD64. Install that on some true ARMv7 hardware and go with it. You only need the qemu emulator environment when you need to build ARM code on AMD64 hardware.



  • @bmeeks
    Those path like /usr/etc can be redefined in configure. Because both FreeBSD and Linux use ELF format. I bet FreeBSD can load dynamic library like Linux from anywhere. But maybe I'm wrong. We will see.

    Your way is definitely easier to avoid the hack I mentioned. Can you list a command that build only one package like Barnyard2, instead of everything?

    Building anything on a native ARM board is always a bad idea. It is way too slow. I once built a customized open-cv on odroid XU4 (way much faster than RPI2 and RPI4) . It took me a day to build it.

    I found it difficult to emulate ARM version FreeBSD in QEMU under amd64. The pre-built SD card image provided by FreeBSD doesn't work out-of-box in QEMU. You have to hack the FreeBSD kernel. It seems too much work to get it done. I gave up on this.

    But I did find another way: cross compile in amd64 for ARM. See https://wiki.freebsd.org/FreeBSD/arm/crossbuild



  • @bmeeks

    make package command works. I can install that package locally without problem. It loads the binary correctly:

    [Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ ldd /usr/local/bin/barnyard2
    /usr/local/bin/barnyard2:
    	libmysqlclient.so.20 => /usr/local/lib/mysql/libmysqlclient.so.20 (0x800a00000)
    	libz.so.6 => /lib/libz.so.6 (0x800fb9000)
    	libpcap.so.8 => /lib/libpcap.so.8 (0x8011d1000)
    	libm.so.5 => /lib/libm.so.5 (0x80142d000)
    	libc.so.7 => /lib/libc.so.7 (0x80165a000)
    	libssl.so.8 => /usr/lib/libssl.so.8 (0x801a16000)
    	libcrypto.so.8 => /lib/libcrypto.so.8 (0x801e00000)
    	librt.so.1 => /usr/lib/librt.so.1 (0x80226f000)
    	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x802475000)
    	libc++.so.1 => /usr/lib/libc++.so.1 (0x802678000)
    	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802946000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802b65000)
    	libthr.so.3 => /lib/libthr.so.3 (0x802d74000)
    	libelf.so.2 => /lib/libelf.so.2 (0x802f9c000)
    

    But I struggled with ARM cross compile. The wiki I quote suggest use the build flag like TARGET=arm TARGET_ARCH=armv6. But it doesn't work. It still shows amd64 binary.

    [Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ make -j8 package TARGET=arm TARGET_ARCH=armv6
    
    [Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ file work/stage/usr/local/bin/barnyard2
    work/stage/usr/local/bin/barnyard2: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, stripped
    

    Thanks for your time. I will ask the folks how to cross compile in FreeBSD embedded section.



  • @rickyzhang :
    You have to install qemu-user-static. That's what I've really been talking about the whole time I've mentioned qemu. I was shortening what I typed. I thought you understood already that's how you cross-compile ARM code on AMD64 hardware. That's what the pfSense builder environment does for you, and that is why I use it. It only quit working rather recently (as in the 1st quarter of this year) due to changes in some of the dependent packages such as Go. What happened is that some of the latest ports updates on FreeBSD no longer build properly when cross-compiled using qemu-user-static.

    If you want to build only a single package, you can alter the make.conf and poudriere.bulk files in the pfSense tree to accomplish that. There are also some port options specified for Barnyard2 that are needed.



  • @bmeeks

    There are two different things: emulation and cross compile.

    The emulation means emulating ARM instruction set under AMD64. That means you need ARM version FreeBSD OS and toolchain. That's why I struggled to run the pre-built RPI2 SD card image provided by FreeBSD in QEMU.

    Cross compile means your OS and toolchain still run as AMD64 native. It is the compiler (written in native AMD64 instruction) that generates the code which can run in ARM instruction set target platform (Think of this process as you speak foreign language without being in foreign country)

    I don't know the technical details qemu-user-static. But it sounds like it is an emulation rather than cross compile. Can you find a doc on this thingy?

    I thought passed in the flag TARGET=arm TARGET_ARCH=armv6 will force to use cross compiler toolchain.



  • @rickyzhang :

    The qemu-user-static package is the easiest way to cross-compile by using emulation. So I am really talking about both things (cross-compile and emulation). Here is a link: https://forums.freebsd.org/threads/building-arm-packages-with-poudriere-the-simple-way.52994/. And here is another link discussing a similar setup: https://www.dvatp.com/tech/armv6_freebsd_poudriere.

    The steps outlined in those links are basically what the pfSense package builder scripts are doing for you if you follow my earlier instructions to set that up.



  • If you cross compile, you don't need emulation.

    I'm not familiar with FreeBSD jail or poudriere. But it sounds like Linux docker container where it use the same kernel but different cgroup and an independent rootfs to isolate the build env. In any case, docker container always run as native binary. I bet FreeBSD should do the same. Correct me if I'm wrong.

    I already started the process like below to build my cross compiler tool chain: http://ray-freebsd.blogspot.com/2011/09/cross-compiling-ports-for-freebsd.html

    I will try your poudriere if there is too much trouble.

    Thanks in advance.



  • @rickyzhang :
    My understanding is that you need the emulation because some of the required tools are not available in AMD64 form or something like that. I've never investigated the "why" in much detail. I'm telling you the process that was given to me by the pfSense developer team a couple of years ago and that is incorporated into their build process. If you don't want to follow that advice, then OK. You can try the cross-compilation route. I will tell you that the pfSense team uses the method I described (well, that is until the last round of FreeBSD ports updates where some ports (Go being one of them) quit building properly using the emulation/cross-compiling tool chain.

    I understand the difference between emulation and cross-compiling, but I was not sure of your level of expertise so I was not specific about the details. Since you are not familiar with Poudriere, it sounds like your FreeBSD experience may be limited. In many ways the compilation and linking tools for FreeBSD lag behind those available for Linux (and some of the defaults are different, for example using llvm instead of gcc).



  • I may know why. I'm sure Golang is the one to be blamed. Because the latest version of Golang build needs to use lower version of Golang to bootstrap its build process. Imagine how that will work in cross compiling process. You cross compile a bootstrap Golang but only runs in ARM. You can't use that bootstrap version Golang in your AMD64 platform to build the latest version of Golang.

    I admit I have limited FreeBSD experience. There are tons of doc to read and catch it up. But I really appreciate your help and advice from FreeBSD community.

    No offense. I felt the same way as you did: FreeBSD did lag behind Linux in terms of tooling. But FreeBSD did have advantage to run as network equipment because Linux change so rapidly. You have no idea how many bugs are introduced because of those "new features".

    I got stuck now because I don't know:

    • How to get clang build tools. The svn repo doesn't include clang svn checkout https://svn.freebsd.org/base/releng/`uname -r | cut -d'-' -f1,1` /usr/src
    • How to have a clean slate so that I can clean up all configuration setting and artifacts from the AMD64 build of dependencies ports (a lot of them).


  • I was wrong about source code doesn't include clang. It is in usr.bin source code folder.

    But I'm right about poudriere regarding to its emulation nature. I built a poudriere jail as you instructed:

    sudo poudriere jails -c -j pfsense-port-11-2-armv6 -a arm.armv6 -m svn -v release/11.2.0
    

    I found that the compiler cc in the armv6 architecture is actually ARM binary. That means when we build anything inside the jail, we compile in ARM emulation in AMD64 platform. It is not going to be fast compared to cross compile.

    Ricky@freebsd ~ $ file /usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc
    
    /usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), statically linked, for FreeBSD 11.2, FreeBSD-style, stripped
    
    


  • Yes, the building of the ARM packages is much slower in the pfSense builder than the AMD64 packages. You can selectively build for each architecture independently using arguments for the build.sh script provided in the pfSense builder tools. I have been building only AMD64 versions of my Snort and Suricata packages because of the problems with Golang that you mentioned. The Golang issue prevents the ARM build process from completing successfully.

    However, if you turn off all of the ARM packages except for just the ones required for Barnyard2 the ARM build might suceed. I have not tried. You can turn packages "on" and "off" by manipulating the Poudriere "bulk" files. If using the pfSense build tree, those will be in pfsense/tools/conf. The make.conf file in that same path controls the various option knobs for each port. That's where you selectively enable or disable particular build options.



  • I'm still studying how to use poudriere. As I have tried the build process in AMD64, it is fairly easy to run make command in pfsense port tree to build Barnyard2 package. It also works correctly.

    I wrote my notes here. You can see that I listed two approaches to build ARM ports. In the 1st approach a.k.a cross compile approach, I succeeded in building cross compile tool chain for armv6 (This mean you can run the build process in native AMD64 way. It is much faster than emulation). But I haven't figured it out how to override default cc and link command by my cross compile tool chain. FreeBSD port Makefile does not work like Linux way. Please let me know if you know the answer.

    Regarding to poudriere emulation approach, I'm thinking:

    • Create a poudriere port from a local cloned pfsense port directory.
    • Choose Baryard2 package from a package list file.
    • Run poudriere bulk

    I read the build.sh script from pfsense. It is a lot of bash reading to figure out how it works. Gosh... pfsense make it hard for us to build it.



  • @rickyzhang said in Barnyard2 and MariaDB:

    I'm still studying how to use poudriere. As I have tried the build process in AMD64, it is fairly easy to run make command in pfsense port tree to build Barnyard2 package. It also works correctly.

    I wrote my notes here. You can see that I listed two approaches to build ARM ports. In the 1st approach a.k.a cross compile approach, I succeeded in building cross compile tool chain for armv6 (This mean you can run the build process in native AMD64 way. It is much faster than emulation). But I haven't figured it out how to override default cc and link command by my cross compile tool chain. FreeBSD port Makefile does not work like Linux way. Please let me know if you know the answer.

    Regarding to poudriere emulation approach, I'm thinking:

    • Create a poudriere port from a local cloned pfsense port directory.
    • Choose Baryard2 package from a package list file.
    • Run poudriere bulk

    I read the build.sh script from pfsense. It is a lot of bash reading to figure out how it works. Gosh... pfsense make it hard for us to build it.

    Just running make from inside the pfSense ports tree is not the same as running their build.sh script with the proper arguments. The build process uses Poudriere with a proper Poudriere Jail that has the correct port revisions within it needed to make things work on pfSense itself. Running the build within the jail is a key part of the process, especially if you want a final package file that you can install using pkg on the firewall with all the correct dependencies listed.

    pfSense builds their packages using the method I described to you. They do not use your cross-compile method because it has issues on FreeBSD as you are experiencing.



  • First of all, I think we should never call using poudriere jail to compile ARM in AMD64 platform is cross compile. It is emulation. Basically, it use qemu-arm-static to run ARM binary tool chain like cc in emulation mode. The performance really sucks even in my 8 cores i9-9900K CPU.

    I enabled the following in /usr/local/etc/poudriere.conf. But after 30 minutes run, it still compiling dependencies of Barnyard2.

    PARALLEL_JOBS=8
    ALLOW_MAKE_JOBS=yes
    

    The fastest way should be cross compile tool chain. I remember last time I used make command to build Barnyard2 in amd64. It took less than 10 minutes. I'm surprised FreeBSD community or pfsense has not improved it. I'm thinking of creating amd64 jail and override tool chain with cross compile ones. But that's my next step. First thing first: patch Barnyard2....

    Right now, my poudriere bulk got stuck in fetching textinfo package. I believed the server must be down. I will give it a try later.

    [00:29:11] [01] [00:00:15] Finished print/texinfo | texinfo-6.5,1: Failed: fetch
    [00:29:11] [01] [00:00:15] Skipping devel/autoconf | autoconf-2.69_1: Dependent port print/texinfo | texinfo-6.5,1 failed
    [00:29:11] [01] [00:00:15] Skipping devel/automake | automake-1.16.1: Dependent port print/texinfo | texinfo-6.5,1 failed
    [00:29:11] [01] [00:00:15] Skipping security/barnyard2 | barnyard2-1.13_1: Dependent port print/texinfo | texinfo-6.5,1 failed
    [00:29:11] [01] [00:00:15] Skipping devel/libtool | libtool-2.4.6: Dependent port print/texinfo | texinfo-6.5,1 failed
    [00:29:11] [01] [00:00:15] Skipping devel/m4 | m4-1.4.18,1: Dependent port print/texinfo | texinfo-6.5,1 failed
    


  • @bmeeks

    I got stuck in building Barnyard2 dependencies. I tried it again this morning. It still failed at fetching. I bet the file checksum or the size has been changed. Do you have a good suggestion?

    ===========================================================================
    =======================<phase: fetch-depends  >============================
    ===========================================================================
    =======================<phase: fetch          >============================
    ===>  License GPLv3+ accepted by the user
    => texinfo-6.5.tar.xz doesn't seem to exist in /portdistfiles/texinfo/6.5.
    => Attempting to fetch https://ftpmirror.gnu.org/texinfo/texinfo-6.5.tar.xz
    texinfo-6.5.tar.xz                                       0  B    0  Bps
    => htmlxref.cnf doesn't seem to exist in /portdistfiles/texinfo/6.5.
    => Attempting to fetch http://distcache.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf
    fetch: http://distcache.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf: Not Found
    => Attempting to fetch http://distcache.us-east.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf
    fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf: Not Found
    => Attempting to fetch http://distcache.eu.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf
    fetch: http://distcache.eu.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf: Not Found
    => Attempting to fetch http://distcache.us-west.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf
    fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/sunpoet/texinfo/6.5/htmlxref.cnf: Not Found
    => Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/texinfo/6.5/htmlxref.cnf
    fetch: http://distcache.FreeBSD.org/ports-distfiles/texinfo/6.5/htmlxref.cnf: size mismatch: expected 20137, actual 20076
    => Couldn't fetch it - please try to retrieve this
    => port manually into /portdistfiles/texinfo/6.5 and try again.
    *** Error code 1
    
    Stop.
    make: stopped in /usr/ports/print/texinfo
    =>> Cleaning up wrkdir
    ===>  Cleaning for texinfo-6.5,1
    build of print/texinfo | texinfo-6.5,1 ended at Tue Aug  6 08:14:30 UTC 2019
    build time: 00:00:13
    !!! build failure encountered !!!
    
    

    Do you have a detail step to use pfsense build process to compile to ARM target platform from AMD64 platform?

    I'm slow. I have to have some detailed step in the notes like this.

    I grep pfsense repo and see how they use poudriere. It doesn't seem like they use my 2nd approach in my notes where you create emulation ARM jail in poudriere. I don't know if they build in a beefy ARM board or they use cross compiler in AMD64 with ARM target.

    I saw your name in pfsense's commit. I beg you are quite active from community to send PR. You are the right person I should talk to.

    Thanks!



  • For some reason your builder is having problems downloading the source code for that port. You might want to try the download manually from other Internet locations. If the required tarball is in /usr/ports/distfiles, then the make process won't attempt to download it.

    I am the maintainer for the Snort and Suricata packages on pfSense, so yeah, I have a number of commits to the FreeBSD-ports tree in pfSense. That's also why I use their builder script when compiling my packages for testing before submitting them to the upstream master tree.

    Using the builder environment for packages is not that difficult. Here are the basic steps. I'm assuming you want to build your package for pfSense-2.4.4, so my instructions are based on that. However, I don't target that environment. I build for the DEVEL branch, so I'm not 100% sure that everything I have outlined below works on the RELENG_2_4_4 branch.

    1. Begin by installing qemu-user-static on your FreeBSD machine with this command:
    pkg install qemu-user-static
    
    1. Make sure you have cloned both the FreeBSD-ports and pfsense repos onto your FreeBSD machine.
    2. Change into the FreeBSD-ports directory and execute this command to switch to the pfSense-2.4.4 branch.
    git checkout RELENG_2_4_4
    
    1. Then go back up one directory level and change down into the pfsense directory.
    2. To see all the options for the build.sh script, execute it with no arguments like this:
    ./build.sh
    
    1. Execute this command to set up an intial builder environment:
    ./build.sh --setup
    
    1. After that completes, execute the command to pull down the latest FreeBSD source code in preparation for building a poudriere jail. Note that you will need to edit the file /pfsense/tools/builder_common.sh and comment out two sections in that file having to do with pulling down the source tree for the Netgate ID stuff (gnid) as that is proprietary source and source code update will fail when it can't pull down that code. Use an editor and search for the phrase "gnid" in the script file. You will find it used in two "if" blocks in the script. Comment out both "if" blocks entirely and then save the file. After editing and saving the file, execute this command:
    ./build.sh --update-sources
    
    1. When that finishes, it's time to actually create the Poudriere ports tree and the jails for building using this command (note, this will take at least an hour and probably nearly twice that long to complete depending on your hardware):
    ./build.sh --setup-poudriere
    
    1. To actually start a package build process after the ports tree and pourdriere jails are ready, execute this command:
    ./build.sh --update-pkg-repo
    
    1. There are some additional command line options for the previous command. You can use -a [arch] to control which CPU architectures to target. The default is to build both AMD64 and ARM packages. If you want to build say just AMD64 packages, the argument would be -a amd64.amd64. For just ARM packages, the argument would be -a arm.armv7.

    The completed packages will be stored in sub-directories under /usr/local/poudriere/data/packages on your FreeBSD machine.

    Because of the issues with some packages under qemu-user-static, you will want to edit the file pfsense/tools/conf/pfPorts/poudriere.bulk.exclude.arm.armv7 to remove all of the ports that depend on the problematic packages such as Golang. You can open and examine the file to see the required syntax. This may become a trial and error process to get everything excluded.

    The above process will work, but it might take a few iterations and some trial and error to get it all going. There is no official documentation, but if you wade through the builder_common.sh script you can figure out how things work by looking at the code.

    Also note that the setup process only has to be performed once. Thereafter, you can simply execute

    ./build.sh --update-pkg-repo
    

    to build a new version of a package. To make poudriere start a new build of a modified package, you will have to either bump the port version string or go to the /usr/local/poudriere/data/packages tree and remove the package file that you want to rebuild. Either of these two methods will alert poudriere that a build of that package is required.



  • @bmeeks

    I really appreciate your help! I will give a try tonight.

    Please let me know if you are OK with me to copy your steps to my wiki.

    I found the culprit why I got stuck. The file size of texinfo/6.5/htmlxref.cnf specified by print/texinfo/distinfo doesn't match what it downloads from Internet.

    ./distinfo:5:SIZE (texinfo/6.5/htmlxref.cnf) = 20137
    

    I check out my FreeBSD-ports repo on tag: v2.4.4_3 which match the release of my pfsense router current firmware. But it looks like the build can't replicate due to the dependency of hell.

    I will switch to devel branch. I found that it change the file size.



  • @rickyzhang said in Barnyard2 and MariaDB:

    v2.4.4_3

    I haven't tried your build.sh approach. But I succeeded in building Barnyard2 in poudriere ARM jail. I applied the make.conf from pfsense. So I don't need to answer the build options during the build.

    The whole build process took about 2hrs! But I went into dependency hell. I manually substitute original barnyard2 binary in /usr/local/bin with my new one. The new one needs libmysqlclient.so.20, while the original one needs libmysqlclient.so.18.

    Although I have new mysqlclient package, I don't think it is a good idea to replace it.

    [2.4.4-RELEASE][admin@pfSense.localdomain]/root/Download/mysql57: ldd /usr/local/bin/barnyard2.orig 
    /usr/local/bin/barnyard2.orig:
    	libmysqlclient.so.18 => /usr/local/lib/mysql/libmysqlclient.so.18 (0x20100000)
    	libz.so.6 => /lib/libz.so.6 (0x20087000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x200a6000)
    	libm.so.5 => /lib/libm.so.5 (0x20425000)
    	libbroccoli.so.5 => /usr/local/lib/libbroccoli.so.5 (0x2044a000)
    	libc.so.7 => /lib/libc.so.7 (0x20500000)
    	libssl.so.8 => /usr/lib/libssl.so.8 (0x2046f000)
    	libcrypto.so.8 => /lib/libcrypto.so.8 (0x20700000)
    	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x200f5000)
    	libc++.so.1 => /usr/lib/libc++.so.1 (0x2089e000)
    	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x204d1000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x20666000)
    	libthr.so.3 => /lib/libthr.so.3 (0x20675000)
    	libelf.so.2 => /lib/libelf.so.2 (0x206a2000)
    
    [2.4.4-RELEASE][admin@pfSense.localdomain]/root/Download: ldd arm-pkg/barnyard2
    arm-pkg/barnyard2:
    	libmysqlclient.so.20 => not found (0)
    	libz.so.6 => /lib/libz.so.6 (0x20089000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x200a8000)
    	libm.so.5 => /lib/libm.so.5 (0x200f7000)
    	libbroccoli.so.5 => /usr/local/lib/libbroccoli.so.5 (0x2011c000)
    	libc.so.7 => /lib/libc.so.7 (0x20200000)
    	libssl.so.8 => /usr/lib/libssl.so.8 (0x20141000)
    	libcrypto.so.8 => /lib/libcrypto.so.8 (0x20400000)
    

    The root of the problem is that my port tree is forked from devel branch which is different from v2.4.4.4_3.

    But obviously I have to manually fix Barnyard2 dependency distinfo file. The whole build in pfsense is not repeatable. Because its dependencies come from Internet. That doesn't seem to be a good sign.



  • I keep telling you what to do and you keep ignoring my advice... ☺. Build it within the pfSense builder environment using the build.sh script and the various arguments I gave you in the previous post farther up above.

    Yes, DEVEL in pfSense is based on FreeBSD 12.0 which has a slightly different version of various libraries. pfSense-2.4.4_3 is based on FreeBSD 11.2.

    You can probably switch to the RELENG_2_4_4 branch within the pfSense builder environment and build Barnyard2 from there. That is the current RELEASE branch. The initial build of any package is going to take a while because all of the dependencies have to built first. Subsequent builds of just the Barnyard2 module will be much faster.



  • @bmeeks

    I know I looks like an idiot. When in Rome, do as the Romans do. I will definitely follow your advice later.

    Before you give me your detailed instruction for pfsense, I can't find any documents from pfsense so I have to figure this out by myself. It is kind of painful.

    As a lazy developer, I look for an easy and quick way to build thing for pfSense.

    Because I'm more familiar with make command and autoconf way. So I tried this approach first and then hit the wall when do cross compiling.

    Later I tried 2nd approach: use poudriere jail. I have to read poudriere user guide. I figured out it is slow ARM emulation rather than cross compile. The pfsense build.sh use poudriere jail as well. So I'm very close to what Romans do now.

    I bet most Linux developer who wants to contribute to pfSense will go through the same path like me. I wish we could publish this in our forum so that people can avoid wasting time on build problem and focus on contributing.


Log in to reply