Barnyard2 and MariaDB

rickyzhang

I totally agree with your last statement.

We always face a trade-off between productivity and performance. Now we write as less as possible in low level assembly language because high level abstract language can improve productivity and portability at the cost of performance loss.

Optimization is great in theory but premature one is not in practice. Since ARMv6, ARM cpu supports unaligned memory access in their hardware like intel CPU does.

It is complete clang compiler's fault that they thought they are so smart to generate optimized target code but planting a random timing bomb, instead. In the end, what is more important: optimized executable but randomly fail or reliable executable but slow? The answer is obvious.

bmeeks

@rickyzhang said in Barnyard2 and MariaDB:

...Since ARMv6, ARM cpu supports unaligned memory access in their hardware like intel CPU does.

That is not 100% correct. ARMv7 and later support unaligned access for most opcodes, but not all of them. You still have the LDM and STM opcodes (and I think there are two others as well) that never support unaligned access (meaning the CPU microcode engine will not automatically adjust those opcodes for unaligned access).

The SG-3100 actually uses an armv7 processor and it is the clang/llvm compiler's choice of using the LDM and STM opcodes when optimizing that causes the crash. If the armv7 CPU did an auto-fixup for every opcode, then it would not matter which opcodes the compiler chose to use when optimizing.

rickyzhang

ARM doc already specified that multiple load/store LDM and STM doesn't support unaligned memory access. So why compiler still optimize this operation? This is a typical premature optimization.

The __packed qualifier in pointer casting suppose inform compiler don't use optimized load/store. Or put it in this way: use alignment byte = 1 in __packed qualifier. In this case, the compiler still can use optimized load/store in other suitable use case rather than we completely turn off optimization.

I checked the proposed solution to do a proper pointer casting in C to avoid unaligned memory access: copy the source buffer by memcpy function. The memcpy function always allocate the heap at the maximum multiplier aligned address. So the casting always works. But it is an overhead. You have to do clean up otherwise memory leaks. I don't think it is a simple patch any more.

I think it is not ARM but clang should fix their way to generate target machine code.

bmeeks

@rickyzhang said in Barnyard2 and MariaDB:

ARM doc already specified that multiple load/store LDM and STM doesn't support unaligned memory access. So why compiler still optimize this operation? This is a typical premature optimization.

The __packed qualifier in pointer casting suppose inform compiler don't use optimized load/store. Or put it in this way: use alignment byte = 1 in __packed qualifier. In this case, the compiler still can use optimized load/store in other suitable use case rather than we completely turn off optimization.

I checked the proposed solution to do a proper pointer casting in C to avoid unaligned memory access: copy the source buffer by memcpy function. The memcpy function always allocate the heap at the maximum multiplier aligned address. So the casting always works. But it is an overhead. You have to do clean up otherwise memory leaks. I don't think it is a simple patch any more.

I think it is not ARM but clang should fix their way to generate target machine code.

I agree with you that clang/llvm could do better, but I bet its developers will throw down the philosophical argument that the bad C code should be fixed instead of their compiler generating workarounds for the bad coding, and that what the compiler is doing is "ethically" correct when optimizing as the end result is faster more efficient machine code (of course if that machine code crashes, then just how well off are you?). You will experience zealots on both sides of the argument, unfortunately.

I'm not trying to argue with you. Just pointing out you should probably throw in the towel like I have for Snort and Suricata and just compile your Barnyard2 code on ARM hardware with optimizations disabled.

rickyzhang

Yes, I'm done now. It is quite a learning process. I really appreciate your advice and guidance.

I disabled optimization on Friday. Barnyard2 ran OK. Crash free for two days. Maria DB are loaded with alerts.

I'm ready to send the PR. See my branch: https://github.com/rickyzhang82/FreeBSD-ports/commits/patch-barnyard2-v2-4-4-3

I patched and tested it on FreeBSD-ports repo v2.4.4_3 release tag with make.conf from pfsense repo v2.4.4_3 tag.

I saw devel branch in FreeBSD-ports repo and folks send their PR to devel branch. But I don't see devel branch in pfsense.

Please correct me if I'm wrong:

I should send my PR to FreeBSD-ports on devel branch.
Use make.conf from pfsense in branch RELENG_2_4_4.

bmeeks

@rickyzhang :
I submit my changes against devel in FreeBSD-ports and against master in pfSense. I suspect there are no FreeBSD 11.2 vs FreeBSD 12.0 dependency issues with Barnyard2, so I would submit against devel and master as I mentioned. The pfSense team will then cherry pick and port the changes over to 2.4.4_3 for you. Their preference is to take FreeBSD-ports changes into devel, let them stew in the pfSense-DEVEL branch a bit, and then port them over to RELEASE.

What are you doing on the Barnyard2 configuration side (I'm talking about the barnyard2.conf file) to use the Maria DB? Is it just a drop-in for MySQL with no changes in the Barnyard2 configuration? If not, then you should also create the necessary Snort and Suricata GUI package patches to make the new DB work with them.

rickyzhang

No Snort configuration GUI change are required. I used their pfsense web UI to configure Barnyard2 . (Although I'd like their UI can configure MariaDB port as well, I lived with 3306 default port)

The package didn't ship with data base initialization script. You have to set up Maria DB and run DB creation manually.

rickyzhang

@bmeeks

One more thing I want to share: I found the way to cross compiling rather than emulation.

I saw 6x faster speed up in build time.

See my notes here.