Wan Down

andersondeda

Re: Suricata "fail 'head > kring->rtail && head < kring->rhead'"

Hello good morning everyone.
I'm facing similar problems to the one reported by member Marc05.
My scenario is a little different and I would like to have a little more information before trying to take any secondary action.

My scenario:
Pfsense 2.6 on HA.
Hardware has 64GB RAM and 48 CPUs.
Meerkat 6.0.4_1
Suricata is in inline mode

I used this configuration for a few hours and everything worked fine for me.

When I activated the TLS Log option I started having problems reported in this topic
https://forum.netgate.com/topic/165601/help-understanding-a-crash-kernel-panic

and in this one too

https://forum.netgate.com/topic/165041/suricata-fail-head-kring-rtail-head-kring-rhead/4

My doubt is.
If I update pfsense to 2.7 and consequently FreeBSD to version 14, could the problem be solved?
I ask this for the following reason, I saw some answers stating that the problem could be with Netmap and consequently with FreeBSD.

Additional information about the scenario.
We are only looking at the WAN interface.
We have many virtual servers on our network providing content.

bmeeks

You have an old version of both Suricata and FreeBSD there. It is correct that several changes have occurred to netmap within both of those software pieces. I worked with the upstream Suricata team to make a few key changes in the netmap portions of the Suricata binary to improve its thread concurrency when using host stack rings.

Upgrading could very well address your problem, but it is a wise move to stay up-to-date on your firewall software anyway.

With that number of CPUs and RAM you will need to greatly increase the TCP Stream Memcap value in Suricata if you have not already done so. I suspect you have increased it markedly from the default or else Suricata would not even start.

andersondeda

@bmeeks Hello and thank you for your response.
I will follow your instructions and update pf sense and also suricata.

You're right, I had to increase the 'Stream Memory Cap' to 10737418240.
I left the Reassembly Memory Cap at the same value.

I confess that I have difficulty finding supporting documentation, I feel the need to know the purpose of all these options.

I will perform the upgrade and return with more information, probably only tomorrow.

bmeeks

@andersondeda said in Wan Down:

I confess that I have difficulty finding supporting documentation, I feel the need to know the purpose of all these options.

I agree that the documentation for many of Suricata's internal settings is a bit opaque at times. But the upstream team does have a goal of improving the documentation. The official docs site has a sub-section for each Suricata version. Here is the link to the version for 6.0.13 (which is current in pfSense 2.7.0 right now): https://docs.suricata.io/en/suricata-6.0.13/.

Some details about the Stream Engine settings can be found here: https://docs.suricata.io/en/suricata-6.0.13/configuration/suricata-yaml.html#stream-engine. A quick and dirty explanation is TCP Stream Memory is used to buffer packets associated with a specific TCP flow and reassemble them into a contiguous block that the signature comparision engine can process to look for matches to rules. Because multiple CPU cores means multiple threads with each thread needing its own separate TCP stream reassembly area, the amount of needed RAM increases quickly as the CPU core count increases.