Barnyard2 and MariaDB
-
Is there a patch (or will there be a patch/upgrade) to deal with the Barnyard2/MariaDB issue of needing backticks to properly reference ref_system_id?
I've looked all over the place with the assistance of Dr. Google, and all I can find is that a bug report has been filed, or that there's a patch but you have to clone a git repository and compile a bunch of stuff on your pfSense install. Frankly, not really cool with doing all that and possibly borking my 2 firewalls beyond belief.
TiA
-
Barnyard2 has not been materially updated in FreeBSD ports in several years. By "materially" I mean actual changes to the C source code. There have been some minor tweaks to the package code done by the FreeBSD ports folks to keep the package compiling, but there have been no updates to the core source code to fix any Barnyard2 bugs (or to add new features).
The Snort and Suricata packages simply tag the FreeBSD ports version of Barnyard2 as a runtime dependency so that during installation of Snort or Suricata the
pkg
utility pulls in and installs Barnyard2 as well.I say this to imply that any fixes to Barnyard2 appear unlikely unless some new maintainer takes over the FreeBSD port of Barnyard2.
-
Thanks.
Short of picking up the Barnyard2 code myself (which I'm so rusty on C I couldn't do anyway), any suggestions?
-
No, not really. Suricata is going to discontinue unified2 binary logging in a coming future version, and Snort3 is headed that way as well with everybody switching over to JSON logging options. So the Barnyard2 tab is going to disappear from Suricata at some point in the near future. I suspect the same may happen with Snort a bit later. It depends on exactly what the Snort team finalizes in Snort3.
So my best recommendation would be to start working on something based on ELK or Graylog that can injest JSON logs. Right now Snort 2.9.x can't produce JSON logs, but Snort3 will. Suricata can already produce loads of JSON logs.
-
I'm new to pfSense. I was bitten by the same bug. See post here.
If I want to replace Barnyard2 binary manually, are there any doc on how to setup build env for pfSense port?
Mine is SG-3100 which runs on ARMV7. Where can I find the cross-compile build tool stack?
Thanks in advance.
-
@rickyzhang said in Barnyard2 and MariaDB:
I'm new to pfSense. I was bitten by the same bug. See post here.
If I want to replace Barnyard2 binary manually, are there any doc on how to setup build env for pfSense port?
Mine is SG-3100 which runs on ARMV7. Where can I find the cross-compile build tool stack?
Thanks in advance.
Building armv7 binaries is very difficult now in pfSense unless you have native armv7 hardware to construct a poudriere builder on. There are some packages in the armv7 tree that won't build on qemu, the current emulator used in FreeBSD to do cross compiles. One of these is the Go language (if I recall correctly). That language is a dependency in several FreeBSD packages. So while it is theoretically possible to build most of the armv7 binaries using qemu in a FreeBSD poudriere builder, it would take a lot of trial and error and the disabling of a number of the packages that either don't build in qemu anymore, or that themselves depend on packages that won't build using qemu. So a lot of "ifs".
My suggestion is to forget Barnyard2 in its current state. It has not been maintained in FreeBSD for the last several years, and thus probably has a lot of issues with the newer DB versions out now for MySQL and others.
If you want to try anyway, then construct a FreeBSD machine using FreeBSD 12.0. A virtual machine works nicely. Next, clone the two pfSense repos from Github onto the FreeBSD machine.
After you clone the two repos, there is a
build.sh
script in pfsense/pfsense/tools that you can use to perform various tasks such as creating a Ports tree for pfSense-DEVEL and a corresponding Poudriere builder. It will take quite a bit of looking through thebuild.sh
shell script and the other shell files it includes in order to figure out what to do in order to get packages building. There is basically no documentation. You will need to be rather skilled in building packages on FreeBSD using Poudriere (or else you will need to get up-to-speed yourself). -
Thanks for your advice.
Actually, the upstream Barnyard2 in Github has patched the MariaDB syntax issue. (MariaDB is not 100% compatible with MySQL now as I tried both in containers.) The fix seems easy to apply. The obstacle as you said is the build.
Missing the documentation to setup dev env is the first problem. ARM cross platform build is the second one. But if I can get amd64 done, I can get the ARMV7 SD image and replicate it in QEMU.
I do want to try to build the whole pfSense. I'm more familiar with Macport in Mac OS X where documentation is well written.
After my skimming through the Barnyard2 port from pfSense Git repo(https://github.com/pfsense/FreeBSD-ports/tree/devel/security/barnyard2), I found that it only contains couple patch files and a Makefile.
I don't know how can it get the original source code. It didn't have the whole copy of Barnyard2 source code or its repo code URL to pull from anywhere.
Should I use FreeBSD 11.2 which match pfSense I use rather than FreeBSD 12.0? The FreeBSD port documentation doesn't explain how to replace official port with bespoke port tree like pfSense. That make me concerned if I should spend time on this project.
-
@rickyzhang said in Barnyard2 and MariaDB:
Thanks for your advice.
Actually, the upstream Barnyard2 in Github has patched the MariaDB syntax issue. (MariaDB is not 100% compatible with MySQL now as I tried both in containers.) The fix seems easy to apply. The obstacle as you said is the build.
Missing the documentation to setup dev env is the first problem. ARM cross platform build is the second one. But if I can get amd64 done, I can get the ARMV7 SD image and replicate it in QEMU.
I do want to try to build the whole pfSense. I'm more familiar with Macport in Mac OS X where documentation is well written.
After my skimming through the Barnyard2 port from pfSense Git repo(https://github.com/pfsense/FreeBSD-ports/tree/devel/security/barnyard2), I found that it only contains couple patch files and a Makefile.
I don't know how can it get the original source code. It didn't have the whole copy of Barnyard2 source code or its repo code URL to pull from anywhere.
Should I use FreeBSD 11.2 which match pfSense I use rather than FreeBSD 12.0? The FreeBSD port documentation doesn't explain how to replace official port with bespoke port tree like pfSense. That make me concerned if I should spend time on this project.
You can create a FreeBSD 11.2 virtual machine and then create your own builder system. You don't have to use the pfSense builder system. In fact, you will find some things missing from the pfSense builder if you intend to build an actual pfSense image. You will need to go into the shell script files and comment out several things in order to get a build of pfSense itself to actually complete, and even then some of the kernel patches will be missing. One item you will have to comment out is anything having to do with the GNID stuff (unique Netgate ID). Building packages for AMD64, on the other hand, is easy. Building pfSense itself, not so much.
You will find the source code URL inside the Makefile included with each FreeBSD port. So build a FreeBSD 11.2 machine, replicate a FreeBSD ports tree to it (
portsnap auto
) and then you can examine the Barnyard2 build files. Inside the Makefile you will find the source code URL. Barnyard2 will be in /usr/ports/security/barnyard2. The URL will be shown on the line for "MASTER_SITES=". The first time you build Barnyard2, it will automatically pull down the source tarball and store it in /usr/ports/distfiles. It appears that the port maintainer for Barnyard2 on FreeBSD has not ported any of the Github updates over to the FreeBSD port in quite some time. That's what I meant by not being maintained anymore. The few changes that have been made were only to keep Barnyard2 building as the FreeBSD ports tree changed. There have been no changes to the actual Barnyard2 C source code files -- just changes to the Makefile to alter build dependencies. -
@bmeeks said in Barnyard2 and MariaDB:
/usr/ports/security/barnyard2
I wrote a doc/note on how to setup FreeBSD dev env as your instructed. (Perhaps, I should find a wiki to place it your knowledge somewhere to share it)
See link here: https://github.com/rickyzhang82/PiBa-NL-WIKI/wiki/Setup-FreeBSD-Development-Environment-for-pfSense-port
After running
portsnap auto
, I check the Makfile in /usr/ports/security/barnyard2. There is no "MASTER_SITES=" in this Makefile like the rest of port does. For this, I will read the FreeBSD port doc and see if I can pull the source code and build it.But here is the main roadblock: how can I use pfSense port (https://github.com/pfsense/FreeBSD-ports) rather than the one from FreeBSD?
For now I don't want to build the whole pfSense port but rather the one port only.
BTW: The build.sh has gone from pfsense/pfsense/tools folder (see https://github.com/pfsense/pfsense/tree/master/tools)
-
My apologies on misleading you with the "MASTER_SITES=" line. That is the way most ports do it, but Barnyard2 is different. It's using a Github technique contained within these two lines:
USE_GITHUB= yes GH_ACCOUNT= firnsy
It's been quite some time since I've looked at the Makefile for Barnyard2.
-
@rickyzhang said in Barnyard2 and MariaDB:
@bmeeks said in Barnyard2 and MariaDB:
But here is the main roadblock: how can I use pfSense port (https://github.com/pfsense/FreeBSD-ports) rather than the one from FreeBSD?For now I don't want to build the whole pfSense port but rather the one port only.
BTW: The build.sh has gone from pfsense/pfsense/tools folder (see https://github.com/pfsense/pfsense/tree/master/tools)
The pfsense/pfsense/tools path was assuming you had cloned the Github repo onto a FreeBSD machine. That will be the path on your FreeBSD machine after the clone operation. And I was operating from memory, so I think I remembered the path wrong.
build.sh
is in pfsense/pfsense, but the main file containing all the function calls is in the pfsense/pfsense/tools path.I recommend creating the directory /usr/home/pfsense and then cloning the two repos I linked earlier into that path on your FreeBSD machine. There will be multiple branches within each repo. DEVEL is for pfSense-2.5 and RELENG_2.4.4 is for pfSense-2.4.
After cloning, if you change to the pfsense/pfsense directory and execute the following commands, the script should build the Poudriere jails for you.
./build.sh --setup ./build.sh --setup-poudriere
-
I run
make
in /usr/ports/security/barnyard2. I starts to build all dependencies and pulling its own code.See below:
===> License GPLv2 accepted by the user ===> barnyard2-1.13_3 depends on file: /usr/local/sbin/pkg - found => firnsy-barnyard2-v2-1.13_GH0.tar.gz doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch https://codeload.github.com/firnsy/barnyard2/tar.gz/v2-1.13?dummy=/firnsy-barnyard2-v2-1.13_GH0.tar.gz fetch: https://codeload.github.com/firnsy/barnyard2/tar.gz/v2-1.13?dummy=/firnsy-barnyard2-v2-1.13_GH0.tar.gz: No address record => Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/firnsy-barnyard2-v2-1.13_GH0.tar.gz firnsy-barnyard2-v2-1.13_GH0.tar.gz 100% of 424 kB 1324 kBps 00m00s ===> Fetching all distfiles required by barnyard2-1.13_3 for building ===> Extracting for barnyard2-1.13_3 => SHA256 Checksum OK for firnsy-barnyard2-v2-1.13_GH0.tar.gz. ===> Patching for barnyard2-1.13_3 ===> Applying FreeBSD patches for barnyard2-1.13_3 ===> barnyard2-1.13_3 depends on package: autoconf>=2.69 - not found ===> License GPLv2+ GPLv3+ GFDL AUTOCONF_CONFIGURE_SCRIPT_EXCEPTION accepted by the user ===> autoconf-2.69_3 depends on file: /usr/local/sbin/pkg - found => autoconf-2.69.tar.xz doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch https://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz fetch: https://ftpmirror.gnu.org/autoconf/autoconf-2.69.tar.xz: Authentication error => Attempting to fetch https://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz fetch: https://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz: Authentication error => Attempting to fetch ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz autoconf-2.69.tar.xz 100% of 1186 kB 2416 kBps 00m00s ===> Fetching all distfiles required by autoconf-2.69_3 for building ===> Extracting for autoconf-2.69_3 => SHA256 Checksum OK for autoconf-2.69.tar.xz. ===> Patching for autoconf-2.69_3 ===> Applying FreeBSD patches for autoconf-2.69_3 ===> autoconf-2.69_3 depends on executable: gm4 - not found
The source code indicate its from the upstream. It supposed what I read in Github with the MariaDB syntax patch. I will take a close look.
But I think I do this build exercise in the wrong way. I have to manfully hit my keyboard so many times to answer gettex build option dialog during building of all its dependencies.
-
@rickyzhang :
Yes, FreeBSD has a lot of "interactive" prompts that require hitting "Y" or ENTER. I hate it, but have never found a way around all of them, just some of them. The good news is that it will save your answers and use them from now on without prompting you again. -
I built the Barnyard2 binary successfully from FreeBSD official port by running
make
in /usr/ports/security/barnyard2 folder. I think the same approach can be applied to pfSense's FreeBSD-ports by runningmake
in cloned Git repo.Since I already clone both pfsense repo and FreeBSD-ports repo from Github under pfsense folder. See the folder structure below:
[Ricky@freebsd ~/repo/github/pfsense]$ pwd /home/Ricky/repo/github/pfsense [Ricky@freebsd ~/repo/github/pfsense]$ ls FreeBSD-ports pfsense
I don't see
build.sh
script from pfsense/pfsense folder will use FreeBSD-ports repo cloned by me. In any case, I will give this a try if I fail.Missing build instruction is a real pain in the ass.
-
After building Barnyard2 from by running
make
in /usr/ports/security/barnyard2 folder for 20 mins, it took me less than a few seconds to build it from FreeBSD-ports under my cloned repo /home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2.I think for now I can start to debug the issue. But it is better that I can just build ARMv7 binary, instead amd64.
Now I have to start to install pfSense in amd64 VM and test it. What a pain in the ass.
-
@rickyzhang :
Thebuild.sh
script is not part of the FreeBSD-ports repo. It is part of the pfSense repo. That's why I gave you two links originally to clone. Here is a screenshot from my builder showing thebuild.sh
script file.You need to fill in the proper values in pfsense/pfsense/build.conf and then execute the
build.sh
commands I posted earlier. That will create a proper environment for building pfSense packages using Poudriere. If you build packagea outside of that environment, you will likely find that they fail to load and run on your pfSense firewall due to path problems. -
@rickyzhang said in Barnyard2 and MariaDB:
/home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2
I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running
make
. I may not need to use your approach.Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.
BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.
Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.
-
@rickyzhang said in Barnyard2 and MariaDB:
@rickyzhang said in Barnyard2 and MariaDB:
/home/Ricky/repo/github/pfsense/FreeBSD-ports/security/barnyard2
I see. I already cloned both pfsense and FreeBSD-ports. I can compile Barnyard2 from both FreeBSD official ports and pfsense's own port a.k.a FreeBSD-ports repo by running
make
. I may not need to use your approach.Now I need to figure it out how to test my change. Replicating my physical SG-3100 pfsense setup in a amd64 virtual machine is not fun. I think I should try QEMU now to run FreeBSD in ARMV7 and replay what I have done in amd64 FreeBSD VM.
BTW, can you share your sample build.conf? From the sample build.conf.sample file, I don't see how they point to local FreeBSD-ports repo.
Thanks so much for your guidance. We definitely need a place somewhere to gather your wisdom and help any newbies.
I don't want to share my entire
build.conf
file because it contains some sensitive items. But the main two parameters you need to set are these:# Define FreeBSD repository, branch and specific commit export FREEBSD_REPO_BASE=https://github.com/pfsense/FreeBSD-src.git export FREEBSD_BRANCH=RELENG_2_5
In my case, I am building packages for DEVEL, so I use RELENG_2_5. If you want to build packages for pfSense-RELEASE, then you would use RELENG_2_4_4. You can find the relevant branches by looking at the branches available on Github for FreeBSD-ports in pfSense.
Also I hope you realize that unless you do
make pkg
during your build, the binary you produce likely won't run on a pfSense firewall. You need to use themake pkg
command in order to produce a package file that you can install usingpkg
on pfSense itself.And if you want to actually run your binary on your SG-3100 firewall, you must build it either under the qemu emulator environment or else create your FreeBSD builder on native ARM hardware.
-
Are you suggesting that I can't swap the binary file due to some security features like signature signing on the binary? That's new to me. I will look into
make pkg
command.Yes, I'm working on how to emulate FreeBSD on ARMV7 now. All ARM board I got either too slow like BeagleBone or 64 bit like RPI 4 or even they can't run FreeBSD at all like odroid XU4.
I thought their build should work under corss-compile. But I can only saw some qemu string pops up in their build script (https://github.com/pfsense/pfsense/blob/master/tools/builder_common.sh)
-
@rickyzhang :
My experience when trying to transfer and run a binary built on FreeBSD but outside of the pfSense Poudriere builder structure is that the binary would fail to run because the various paths (/usr/local/bin, /usr/etc, etc.) would be incorrect. Also would get various library loading failures. There are probably solutions to all of those, but I just found it easier to use the Poudriere environment within the pfSense builder structure.