7.0.2_3 update "broke" IPS (netamap) on LAN interface (?)

tinfoilmatt

depending on your Suricata packet capture settings, you may just need to increase netmap thread buffer sizes. this can be done via System / Adavanced / System Tunables.

any jumbo pathways (i.e. packets or frames) anywhere on relevant infrastructure (i.e. the pfSense host itself, or any directly-connected routing or switching devices)? we've used dev.netmap.buf_size="9280" in environments with jumbo paths. but you might be able to get Suricata to come up with ony "4096" if using normal/default MTUs. reboot after adding the tunable (and with Suricata reenabled so you can confirm everything comes up).

you may also need to do some additional system or Suricata tuning. again, depends on environment and Suricata pcap settings. you could disable Suricata pcaps altogether—Services / Suricata / Interfaces / [interface] / Settings / Logging Settings... uncheck "Enable Packet Log"—before attemtpting anything else just to confirm that that's what's preventing Suricata from passing traffic.

FWIW, we've also relied on that brief window while Suricata is initiliazing to bail on attempted reconfigs. :)

DaddyGo

@bmeeks said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

Please let me know if this helps. It is likely a tweak I need to make as a new default in the package if it works.

Hello Bill, thanks for the tweak.

A little bit now the snake bites its own tail situation I'm in, as I'm not on the site.

I have a mgmt. interface to pfSense (w/o Suricata), but there is no internet there. And on the LAN, if I reinstall Suricata, because keep config was set, - I guess the interface will be down again when Suricata starts.

So it takes me a while to configure the mgmt. network over the backup 4G internet to go to pfSense on an interface where Suricata is not configured, so I can make sure I can reach it even if Suricata shuts down the LAN.

I'm definitely going to try the tweak one of the following nights (production environment) because I need the Suricata as there are 250 endpoints behind it.

I'll be back with more news soon, in the meantime thanks for your quick reply and help.

DaddyGo

@cyberconsultants said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

you may just need to increase netmap thread buffer sizes. this can be done via System / Adavanced / System Tunables.

Hi,

Thanks, but it was already a set up, working Suricata, with tunings, as it should be.
I've been pushing IPS/IDS stuff for a couple of years now.

bmeeks

@DaddyGo said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

I'll be back with more news soon, in the meantime thanks for your quick reply and help.

I understand. Just post back here once you're able to try the additional setting tweak. The combination of setting the livedev-use-for-tracking to false AND leaving the default midstream-policy to ignore seems to have done the trick for the OPNsense folks.

DaddyGo

@bmeeks said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

Just post back here once you're able to try the additional setting tweak.

Hello Bill,

Unfortunately, it didn't work for me.
After further investigation, what I can say for sure is that this is (7.0.2_3 update) not a problem on system with (ixl) Intel X700 series...

The test speaks for itself - I show you a series of pictures:

immediately came these erro logs, by email, - dozens of...

pfs / Intel 710 with 7.0.2_3:

bmeeks

@DaddyGo:
So, to be sure I am understanding correctly --

Systems with Intel X700 NICs work fine. So, what NIC type is in the failing systems?

Or did I misunderstand your post?

DaddyGo

@bmeeks said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

Or did I misunderstand your post?

You did not misunderstand, we have several systems where we use Suricata in IPS mode.

Out of curiosity, I ran the update (7.0.2_3) on a system with an Intel XL-710 BM1 controller, i.e. a (ixl) driver.
https://www.supermicro.com/manuals/other/AOC-STG-i4S.pdf

No problem here (Supermicro server), so it could be driver dependent...(?)

The problematic system uses Intel X520, or two of them, original Cisco N2XX-AIPCI01.
https://www.cisco.com/c/dam/en/us/products/collateral/interfaces-modules/ucs-virtual-interface-card-1225/ethernet-x520-server-adapter.pdf

UCS Server Firmware: huu-4.1(2m) - this serverFW includes the slightly Cisco-tailored NIC driver, once upon a time I compared with original Intel there is no significant difference...

bmeeks

@DaddyGo:
Yes, very much sounds like a driver issue. Also be aware that a given "generic" driver in FreeBSD such as ix or igb will usually work with several slightly different firmware versions/revisions of that NIC family. But therein lies a possibility for a particular firmware version from that family to produce issues (bugs).

DaddyGo

@bmeeks said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

Yes, very much sounds like a driver issue.

Today I was reading a lot of old "netmap" related FreeBSD forum threads and there the X520 drives were described as quite stable from a netmap point of view, unlike the XL710.

Now the situation seems to have reversed. :-)

This installation had a harder start at the beginning...
https://forum.netgate.com/topic/184932/starting-suricata-failing-netmap-on-oversized-hw-multiple-processors-cores-ram

I am trying to further refine the netmap settings, this is what I use:
https://github.com/luigirizzo/netmap/issues/783

this is what I am trying to achieve

dev.netmap.buf_num=1000000
dev.netmap.ring_num=1024
dev.netmap.buf_size=2048 and 4096

but I can't get the "default" dev.netmap.ring_num=200 value in sys tunables to change,...
sysctl OIDs, but stays at 200, after the tunnig, - maybe I should put in Loader Tunables too? (loader.conf.local)

can't find info on this... exactly where it should be located
although Steve has written in the past that it is difficult to get information about this, especially for kernel components
https://forum.netgate.com/topic/69184/advanced-system-tunables-vs-boot-loader-conf-local-and-boot-loader-conf/4

so it's in the sys tunables with 1024, but stays at 200...

bmeeks

@DaddyGo said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

but I can't get the "default" dev.netmap.ring_num=200 value in sys tunables to change,...
sysctl OIDs, but stays at 200, after the tunnig,

I'm not an expert in NIC drivers, but I do know that in some cases the driver itself can refuse to accept certain larger values for some parameters. There is an open Suricata Redmine Issue right now for the DPDK interface (used only on Linux) that is experiencing a problem setting a parameter to the value required. It won't go high enough because the NIC driver itself refuses to use the larger values.

DaddyGo

@bmeeks said in 7.0.2_3 update "broke" IPS (netamap) on LAN interface (?):

It won't go high enough because the NIC driver itself refuses to use the larger values.

Okay I'll play with this a bit more I saw somewhere that under CentOS this NIC goes up to 1024, as this is therefore the upper limit:

[NETMAP_RING_POOL] = {
		.name 	= "%s_ring",
		.objminsize = sizeof(struct netmap_ring),
		.objmaxsize = 32*PAGE_SIZE,
		.nummin     = 2,
		.nummax	    = 1024,