Barnyard2 and MariaDB
-
After building Barnyard2 from by running
make
in /usr/ports/security/barnyard2 folder for 20 mins, it took me less than a few seconds to build it from FreeBSD-ports under my cloned repo /home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2.I think for now I can start to debug the issue. But it is better that I can just build ARMv7 binary, instead amd64.
Now I have to start to install pfSense in amd64 VM and test it. What a pain in the ass.
-
@rickyzhang :
Thebuild.sh
script is not part of the FreeBSD-ports repo. It is part of the pfSense repo. That's why I gave you two links originally to clone. Here is a screenshot from my builder showing thebuild.sh
script file.You need to fill in the proper values in pfsense/pfsense/build.conf and then execute the
build.sh
commands I posted earlier. That will create a proper environment for building pfSense packages using Poudriere. If you build packagea outside of that environment, you will likely find that they fail to load and run on your pfSense firewall due to path problems. -
@rickyzhang said in Barnyard2 and MariaDB:
/home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2
I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running
make
. I may not need to use your approach.Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.
BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.
Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.
-
@rickyzhang said in Barnyard2 and MariaDB:
@rickyzhang said in Barnyard2 and MariaDB:
/home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2
I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running
make
. I may not need to use your approach.Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.
BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.
Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.
I don't want to share my entire
build.conf
file because it contains some sensitive items. But the main two parameters you need to set are these:# Define FreeBSD repository, branch and specific commit export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git export FREEBSD_BRANCH=RELENG_2_5
In my case, I am building packages for DEVEL, so I use RELENG_2_5. If you want to build packages for pfSense-RELEASE, then you would use RELENG_2_4_4. You can find the relevant branches by looking at the branches available on Github for FreeBSD-ports in pfSense.
Also I hope you realize that unless you do
make pkg
during your build, the binary you produce likely won't run on a pfSense firewall. You need to use themake pkg
command in order to produce a package file that you can install usingpkg
on pfSense itself.And if you want to actually run your binary on your SG-3100 firewall, you must build it either under the qemu emulator environment or else create your FreeBSD builder on native ARM hardware.
-
Are you suggesting that I can't swap the binary file due to some security features like signature signing on the binary? That's new to me. I will look into
make pkg
command.Yes, I'm working on how to emulate FreeBSD on ARMV7 now. All ARM board I got either too slow like BeagleBone or 64 bit like RPI 4 or even they can't run FreeBSD at all like odroid XU4.
I thought their build should work under corss-compile. But I can only saw some qemu string pops up in their build script (https://github.com/pfsense/pfsense/blob/master/tools/builder_common.sh)
-
@rickyzhang :
My experience when trying to transfer and run a binary built on FreeBSD but outside of the pfSense Poudriere builder structure is that the binary would fail to run because the various paths (/usr/local/bin, /usr/etc, etc.) would be incorrect. Also would get various library loading failures. There are probably solutions to all of those, but I just found it easier to use the Poudriere environment within the pfSense builder structure. -
@rickyzhang said in Barnyard2 and MariaDB:
Yes, I'm working on how to emulate FreeBSD on ARMV7 now. All ARM board I got either too slow like BeagleBone or 64 bit like RPI 4 or even they can't run FreeBSD at all like odroid XU4.
I thought their build should work under corss-compile. But I can only saw some qemu string pops up in their build script (https://github.com/pfsense/pfsense/blob/master/tools/builder_common.sh)
If you run things on native ARM hardware, you don't need to "emulate" anything. There is a FreeBSD image for ARM hardware just like there is an image for AMD64. Install that on some true ARMv7 hardware and go with it. You only need the qemu emulator environment when you need to build ARM code on AMD64 hardware.
-
@bmeeks
Those path like /usr/etc can be redefined in configure. Because both FreeBSD and Linux use ELF format. I bet FreeBSD can load dynamic library like Linux from anywhere. But maybe I'm wrong. We will see.Your way is definitely easier to avoid the hack I mentioned. Can you list a command that build only one package like Barnyard2, instead of everything?
Building anything on a native ARM board is always a bad idea. It is way too slow. I once built a customized open-cv on odroid XU4 (way much faster than RPI2 and RPI4) . It took me a day to build it.
I found it difficult to emulate ARM version FreeBSD in QEMU under amd64. The pre-built SD card image provided by FreeBSD doesn't work out-of-box in QEMU. You have to hack the FreeBSD kernel. It seems too much work to get it done. I gave up on this.
But I did find another way: cross compile in amd64 for ARM. See https://wiki.freebsd.org/FreeBSD/arm/crossbuild
-
make package
command works. I can install that package locally without problem. It loads the binary correctly:[Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ ldd /usr/local/bin/barnyard2 /usr/local/bin/barnyard2: libmysqlclient.so.20 => /usr/local/lib/mysql/libmysqlclient.so.20 (0x800a00000) libz.so.6 => /lib/libz.so.6 (0x800fb9000) libpcap.so.8 => /lib/libpcap.so.8 (0x8011d1000) libm.so.5 => /lib/libm.so.5 (0x80142d000) libc.so.7 => /lib/libc.so.7 (0x80165a000) libssl.so.8 => /usr/lib/libssl.so.8 (0x801a16000) libcrypto.so.8 => /lib/libcrypto.so.8 (0x801e00000) librt.so.1 => /usr/lib/librt.so.1 (0x80226f000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x802475000) libc++.so.1 => /usr/lib/libc++.so.1 (0x802678000) libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x802946000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802b65000) libthr.so.3 => /lib/libthr.so.3 (0x802d74000) libelf.so.2 => /lib/libelf.so.2 (0x802f9c000)
But I struggled with ARM cross compile. The wiki I quote suggest use the build flag like
TARGET=arm TARGET_ARCH=armv6
. But it doesn't work. It still shows amd64 binary.[Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ make -j8 package TARGET=arm TARGET_ARCH=armv6 [Ricky@freebsd ~/repo/github/pfsense/FreeBSD-ports/security/barnyard2]$ file work/stage/usr/local/bin/barnyard2 work/stage/usr/local/bin/barnyard2: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, stripped
Thanks for your time. I will ask the folks how to cross compile in FreeBSD embedded section.
-
@rickyzhang :
You have to installqemu-user-static
. That's what I've really been talking about the whole time I've mentioned qemu. I was shortening what I typed. I thought you understood already that's how you cross-compile ARM code on AMD64 hardware. That's what the pfSense builder environment does for you, and that is why I use it. It only quit working rather recently (as in the 1st quarter of this year) due to changes in some of the dependent packages such as Go. What happened is that some of the latest ports updates on FreeBSD no longer build properly when cross-compiled usingqemu-user-static
.If you want to build only a single package, you can alter the
make.conf
andpoudriere.bulk
files in the pfSense tree to accomplish that. There are also some port options specified for Barnyard2 that are needed. -
There are two different things: emulation and cross compile.
The emulation means emulating ARM instruction set under AMD64. That means you need ARM version FreeBSD OS and toolchain. That's why I struggled to run the pre-built RPI2 SD card image provided by FreeBSD in QEMU.
Cross compile means your OS and toolchain still run as AMD64 native. It is the compiler (written in native AMD64 instruction) that generates the code which can run in ARM instruction set target platform (Think of this process as you speak foreign language without being in foreign country)
I don't know the technical details
qemu-user-static
. But it sounds like it is an emulation rather than cross compile. Can you find a doc on this thingy?I thought passed in the flag
TARGET=arm TARGET_ARCH=armv6
will force to use cross compiler toolchain. -
The
qemu-user-static
package is the easiest way to cross-compile by using emulation. So I am really talking about both things (cross-compile and emulation). Here is a link: https://forums.freebsd.org/threads/building-arm-packages-with-poudriere-the-simple-way.52994/. And here is another link discussing a similar setup: https://www.dvatp.com/tech/armv6_freebsd_poudriere.The steps outlined in those links are basically what the pfSense package builder scripts are doing for you if you follow my earlier instructions to set that up.
-
If you cross compile, you don't need emulation.
I'm not familiar with FreeBSD jail or poudriere. But it sounds like Linux docker container where it use the same kernel but different cgroup and an independent rootfs to isolate the build env. In any case, docker container always run as native binary. I bet FreeBSD should do the same. Correct me if I'm wrong.
I already started the process like below to build my cross compiler tool chain: http://ray-freebsd.blogspot.com/2011/09/cross-compiling-ports-for-freebsd.html
I will try your poudriere if there is too much trouble.
Thanks in advance.
-
@rickyzhang :
My understanding is that you need the emulation because some of the required tools are not available in AMD64 form or something like that. I've never investigated the "why" in much detail. I'm telling you the process that was given to me by the pfSense developer team a couple of years ago and that is incorporated into their build process. If you don't want to follow that advice, then OK. You can try the cross-compilation route. I will tell you that the pfSense team uses the method I described (well, that is until the last round of FreeBSD ports updates where some ports (Go being one of them) quit building properly using the emulation/cross-compiling tool chain.I understand the difference between emulation and cross-compiling, but I was not sure of your level of expertise so I was not specific about the details. Since you are not familiar with Poudriere, it sounds like your FreeBSD experience may be limited. In many ways the compilation and linking tools for FreeBSD lag behind those available for Linux (and some of the defaults are different, for example using llvm instead of gcc).
-
I may know why. I'm sure Golang is the one to be blamed. Because the latest version of Golang build needs to use lower version of Golang to bootstrap its build process. Imagine how that will work in cross compiling process. You cross compile a bootstrap Golang but only runs in ARM. You can't use that bootstrap version Golang in your AMD64 platform to build the latest version of Golang.
I admit I have limited FreeBSD experience. There are tons of doc to read and catch it up. But I really appreciate your help and advice from FreeBSD community.
No offense. I felt the same way as you did: FreeBSD did lag behind Linux in terms of tooling. But FreeBSD did have advantage to run as network equipment because Linux change so rapidly. You have no idea how many bugs are introduced because of those "new features".
I got stuck now because I don't know:
- How to get clang build tools. The svn repo doesn't include clang
svn checkout https://svn.freebsd.org/base/releng/`uname -r | cut -d'-' -f1,1` /usr/src
- How to have a clean slate so that I can clean up all configuration setting and artifacts from the AMD64 build of dependencies ports (a lot of them).
- How to get clang build tools. The svn repo doesn't include clang
-
I was wrong about source code doesn't include clang. It is in
usr.bin
source code folder.But I'm right about poudriere regarding to its emulation nature. I built a poudriere jail as you instructed:
sudo poudriere jails -c -j pfsense-port-11-2-armv6 -a arm.armv6 -m svn -v release/11.2.0
I found that the compiler cc in the armv6 architecture is actually ARM binary. That means when we build anything inside the jail, we compile in ARM emulation in AMD64 platform. It is not going to be fast compared to cross compile.
Ricky@freebsd ~ $ file /usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc /usr/local/poudriere/jails/pfsense-port-11-2-armv6/usr/bin/cc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (FreeBSD), statically linked, for FreeBSD 11.2, FreeBSD-style, stripped
-
Yes, the building of the ARM packages is much slower in the pfSense builder than the AMD64 packages. You can selectively build for each architecture independently using arguments for the
build.sh
script provided in the pfSense builder tools. I have been building only AMD64 versions of my Snort and Suricata packages because of the problems with Golang that you mentioned. The Golang issue prevents the ARM build process from completing successfully.However, if you turn off all of the ARM packages except for just the ones required for Barnyard2 the ARM build might suceed. I have not tried. You can turn packages "on" and "off" by manipulating the Poudriere "bulk" files. If using the pfSense build tree, those will be in pfsense/tools/conf. The
make.conf
file in that same path controls the various option knobs for each port. That's where you selectively enable or disable particular build options. -
I'm still studying how to use poudriere. As I have tried the build process in AMD64, it is fairly easy to run
make
command in pfsense port tree to build Barnyard2 package. It also works correctly.I wrote my notes here. You can see that I listed two approaches to build ARM ports. In the 1st approach a.k.a cross compile approach, I succeeded in building cross compile tool chain for armv6 (This mean you can run the build process in native AMD64 way. It is much faster than emulation). But I haven't figured it out how to override default cc and link command by my cross compile tool chain. FreeBSD port Makefile does not work like Linux way. Please let me know if you know the answer.
Regarding to poudriere emulation approach, I'm thinking:
- Create a poudriere port from a local cloned pfsense port directory.
- Choose Baryard2 package from a package list file.
- Run poudriere bulk
I read the build.sh script from pfsense. It is a lot of bash reading to figure out how it works. Gosh... pfsense make it hard for us to build it.
-
@rickyzhang said in Barnyard2 and MariaDB:
I'm still studying how to use poudriere. As I have tried the build process in AMD64, it is fairly easy to run
make
command in pfsense port tree to build Barnyard2 package. It also works correctly.I wrote my notes here. You can see that I listed two approaches to build ARM ports. In the 1st approach a.k.a cross compile approach, I succeeded in building cross compile tool chain for armv6 (This mean you can run the build process in native AMD64 way. It is much faster than emulation). But I haven't figured it out how to override default cc and link command by my cross compile tool chain. FreeBSD port Makefile does not work like Linux way. Please let me know if you know the answer.
Regarding to poudriere emulation approach, I'm thinking:
- Create a poudriere port from a local cloned pfsense port directory.
- Choose Baryard2 package from a package list file.
- Run poudriere bulk
I read the build.sh script from pfsense. It is a lot of bash reading to figure out how it works. Gosh... pfsense make it hard for us to build it.
Just running
make
from inside the pfSense ports tree is not the same as running theirbuild.sh
script with the proper arguments. The build process uses Poudriere with a proper Poudriere Jail that has the correct port revisions within it needed to make things work on pfSense itself. Running the build within the jail is a key part of the process, especially if you want a final package file that you can install usingpkg
on the firewall with all the correct dependencies listed.pfSense builds their packages using the method I described to you. They do not use your cross-compile method because it has issues on FreeBSD as you are experiencing.
-
First of all, I think we should never call using poudriere jail to compile ARM in AMD64 platform is cross compile. It is emulation. Basically, it use qemu-arm-static to run ARM binary tool chain like cc in emulation mode. The performance really sucks even in my 8 cores i9-9900K CPU.
I enabled the following in
/usr/local/etc/poudriere.conf
. But after 30 minutes run, it still compiling dependencies of Barnyard2.PARALLEL_JOBS=8 ALLOW_MAKE_JOBS=yes
The fastest way should be cross compile tool chain. I remember last time I used
make
command to build Barnyard2 in amd64. It took less than 10 minutes. I'm surprised FreeBSD community or pfsense has not improved it. I'm thinking of creating amd64 jail and override tool chain with cross compile ones. But that's my next step. First thing first: patch Barnyard2....Right now, my poudriere bulk got stuck in fetching textinfo package. I believed the server must be down. I will give it a try later.
[00:29:11] [01] [00:00:15] Finished print/texinfo | texinfo-6.5,1: Failed: fetch [00:29:11] [01] [00:00:15] Skipping devel/autoconf | autoconf-2.69_1: Dependent port print/texinfo | texinfo-6.5,1 failed [00:29:11] [01] [00:00:15] Skipping devel/automake | automake-1.16.1: Dependent port print/texinfo | texinfo-6.5,1 failed [00:29:11] [01] [00:00:15] Skipping security/barnyard2 | barnyard2-1.13_1: Dependent port print/texinfo | texinfo-6.5,1 failed [00:29:11] [01] [00:00:15] Skipping devel/libtool | libtool-2.4.6: Dependent port print/texinfo | texinfo-6.5,1 failed [00:29:11] [01] [00:00:15] Skipping devel/m4 | m4-1.4.18,1: Dependent port print/texinfo | texinfo-6.5,1 failed