Suricata Signature Group Header MPM Context Definition

zer0systems

Can someone please further explain (beyond the same help text found everywhere) what "MPM context" means in the case of this particular Suricata property? I understand the mechanism, just not the definition. The standard explanation is as seen below:

Choose a Signature Group Header multi-pattern matcher context. Default is Auto.
AUTO means Suricata selects between Full and Single based on the MPM algorithm chosen. FULL means every Signature Group has its own MPM context. SINGLE means all Signature Groups share a single MPM context. Using FULL can improve performance at the expense of significant memory consumption.

I understand what sig groups are, but not their "headers". Does it mean the "context" of the MPM is either Full or Single? Does that define context?

It also mentions "based on the MPM algorithm chosen". What is Hyperscan was "always" chosen? Would the "context" always be the same.

This very well be an incredibly obvious answer and I'm just not connecting the dots but any help would be appreciated. Thanks.

bmeeks

The Suricata documentation for the pattern matcher is not so great. I'm referring to the upstream docs, too. All the Suricata package on pfSense does is copy what is in the official docs. Here is all I could find on the MPM (Multi-Pattern Matcher) Context setting: https://suricata.readthedocs.io/en/suricata-6.0.1/performance/tuning-considerations.html#detect-sgh-mpm-context-auto-single-full.

The pattern matchers in both Suricata and Snort are a type of dark secret. I don't think anyone has ever fully understood them outside of the actual code developer. The best thing is to leave them at their default values. This is especially true on most firewalls where you don't have 32 GB or more of RAM. Changing the MPM settings can cause the use of RAM to skyrocket. And I mean skyrocket to rapidly running completely out of memory on a 32 GB box!

Resist the urge to tinker with the pattern matcher settings. Leave them at their default values. I mean no disrepect with this comment, but if you have to ask what it is, you probably shouldn't monkey with it ... .

zer0systems

@bmeeks

I completely agree, and I've seen the documentation you linked, thank you. I've actually had great success with tuning Suricata thus far, there's just always that setting or two you wonder about (or would like to grasp a bit more). 32GB?, via Suricata alone? - perhaps with 1000 clients. On a 16GB box with all of Suricata's settings X4, including the available firewall states being increased by 400% your only using about 20% of that 16GB if provisioned properly - that's why they offer the adjustments. I also mean no disrespect, but offering the old "if you don't know" answer is disappointing. When you don't understand something, you try to understand it. If we didn't, no one would truly understand the marrow of anything, in this case being the MPM library. If I would be using Suricata's "defaults", or all of pfSense's for that matter, I would be wasting RAM, as in your 32GB explanation. Yet again, no disrespect, I really do appreciate any help or assistance, but "defaults"?

bmeeks

@zer0systems said in Suricata Signature Group Header MPM Context Definition:

@bmeeks

I completely agree, and I've seen the documentation you linked, thank you. I've actually had great success with tuning Suricata thus far, there's just always that setting or two you wonder about (or would like to grasp a bit more). 32GB?, via Suricata alone? - perhaps with 1000 clients. On a 16GB box with all of Suricata's settings X4, including the available firewall states being increased by 400% your only using about 20% of that 16GB if provisioned properly - that's why they offer the adjustments. I also mean no disrespect, but offering the old "if you don't know" answer is disappointing. When you don't understand something, you try to understand it. If we didn't, no one would truly understand the marrow of anything, in this case being the MPM library. If I would be using Suricata's "defaults", or all of pfSense's for that matter, I would be wasting RAM, as in your 32GB explanation. Yet again, no disrespect, I really do appreciate any help or assistance, but "defaults"?

I have seen a number of folks posting here who have crashed Suricata by tinkering with the settings, so just offering the advice in case you were not aware. Some users like to tweak things just because an adjustment is there. Same for Snort. Its MPM is especially sensitive to tweaking. The defaults are the best in pretty much every circumstance.

If you want to experiment with different settings, you certainly are free to do so. And if you find one that works better, post back and share with the community. But the defaults are chosen by the creators of Suricata and Snort for a good reason -- they generally work best unless there is some peculiar extenuating circumstance in a given environment.

I am the creator of the Suricata package on pfSense (and the maintainer of the Snort package). I put the various adjustments in the GUI because they are available as choices in the underlying binary's configuration file (suricata.yaml for Suricata and snort.conf for Snort). The documentation from upstream for both pieces of software is a bit lacking in terms of a full explanation for some of the configurable options. But the developers of the binary choose their defaults to yield optimal performance in most cases.

I don't know in any detail what the various selections in the MPM do. I don't think anyone truly does except the guy who wrote that code in the binary.