Starting suricata, failing netmap, on "oversized" HW (multiple processors/cores + RAM)

DaddyGo

Hello,

This configuration was put together a few days ago and is currently on the test bench...

We are preparing to replace the HW of our central pfSense unit and also return to the CE “line”.

even before….: - I know it's unnecessary to have that much power for an NGFW, but what we do is to -we are looking for an old server HW (from the shelf) that we have been using for other purposes, but it is over 5 years old, and use it as a central firewall for a few more years, so the HW components are a given.
(so far, this is a proven and economical way to use server HWs)

Our current pfSense unit is an old Cisco C220M3 (2x4C/8T = 16CPUs + ECC DDR3 32GB + 6 x I350 + Intel X520, which we are replacing with a C220M4 (2x12C/24T = 48CPUs + ECC DDR4 64GB + 6 x I350 + 2 x Intel X520).

As I have already experienced with the C220M3 and I have also read about the Suricata memory when using many CPU cores and RAM: https://forum.netgate.com/topic/148365/suricata-does-not-start-the-interface
I raised the Suricata stream memory cap to a higher value early in the setup, this always solved the problem that Suricata did not start in IPS mode on the interfaces.

The full NGFW installation is complete, and everything is working as expected (2.7.2 CE) - only Suricata refuses to run only two interfaces in IPS mode.

When I add the third interface in IPS mode only one (int.) remains running and the other interfaces stop with the following error: "Error: netmap: opening devname netmap:igb2-4/T failed: Cannot allocate memory" and/or ......................netmap*:ix**2-3/T* - it makes absolutely no difference whether we are talking about igb or ix interfaces…

here is a “Pastebin” with the Suricata logs + ‘sysctl -a | grep netmap’ when I try to run it on three interfaces: https://pastebin.com/tLsDPyBK

I have already changed the “Run Mode” - from AutoFP to Workers, does not bring change.
I read these links below, but I don't know how they relate to the new version 7.0.2_2 on 2.7.2CE, and 6.0.13 on 23.05.1-RELEASE this int. stop problem does not arise, here 4 IPS interfaces run without problems /w Cisco C220M3 (of course with increased stream memory cap)
https://forum.suricata.io/t/starting-suricata-failing-netmap/3108/2
https://redmine.openinfosecfoundation.org/issues/5744

The question is why on an oversized HW, like the C220M4 in question, with the increased “stream mem cap”, why can only run a maximum of 2 IPS interfaces, if I activate the third one, the other two will stop and only one will run in the next…

PS:
Hello @bmeeks Bill, you're the expert on this, have you ever come across this question? Thanks in advance.

bmeeks

Although the fix is intended for ZenArmor, it may help with your case. The default available kernel netmap buffer space may not be large enough on your oversized platform.

Scroll down the page a bit at this link to find the System Tunable you can try for pfSense: https://www.zenarmor.com/docs/troubleshooting/packet-engine.

A Google search for "netmap cannot allocate memory" will bring up some additional links you can research.

DaddyGo

@bmeeks said in Starting suricata, failing netmap, on "oversized" HW (multiple processors/cores + RAM):

find the System Tunable you can try for pfSense

I have done these so far, without any positive results...

I changed the following parameters to:

dev.netmap.buf_num=200000
dev.netmap.ring_num=800
dev.netmap.buf_size=2048 and 4096

but true the Zenarmor link says 1 000 000 for "dev.netmap.buf_num", I'll try that tomorrow - and I'll tell you what happened

bmeeks

@DaddyGo said in Starting suricata, failing netmap, on "oversized" HW (multiple processors/cores + RAM):

@bmeeks said in Starting suricata, failing netmap, on "oversized" HW (multiple processors/cores + RAM):

find the System Tunable you can try for pfSense

I have done these so far, without any positive results...

I changed the following parameters to:

dev.netmap.buf_num=200000
dev.netmap.ring_num=800
dev.netmap.buf_size=2048 and 4096

but true the Zenarmor link says 1 000 000 for "dev.netmap.buf_num", I'll try that tomorrow - and I'll tell you what happened

I suspect a reboot will be required for those changes to take effect. At least rebooting would be good insurance.

DaddyGo

@bmeeks said in Starting suricata, failing netmap, on "oversized" HW (multiple processors/cores + RAM):

I suspect a reboot will be required

Yeah I thought so too, since it's still just in test mode, it's often restarted without consequence...