Suricata inline mode with Netgate 6100

[pfsjap]

I didn't find any recent posts related to this, so would like to ask, if Suricata inline mode should work with Netgate 6100MAX on pfSense+ 23.05?

With Netmap Threads set to auto the last two entries in suricata.log are:

3/6/2023 -- 09:52:50 - <Info> -- Going to use 4 thread(s)
3/6/2023 -- 09:52:50 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:igc2-0/R@conf:host-rings=4 failed: Invalid argument

With Netmap Threads set to 1 the last two entries in suricata.log are:

3/6/2023 -- 10:04:54 - <Info> -- Going to use 1 thread(s)
3/6/2023 -- 10:04:54 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - opening devname netmap:igc2/R failed: Invalid argument

[bmeeks]

Hmm... that looks like a NIC driver problem. Perhaps Suricata thinks the NIC is exposing more netmap rings than it actually is ??

In any case, I would say using Inline IPS Mode is not going to work there. You will have to stick with Legacy Mode.

One of the wrinkles with NIC drivers in FreeBSD is that a given driver class is supposed to support multiple hardware generations of a NIC. But in practice that support may not be 100% when it comes to the latest chips.

[pfsjap]

@bmeeks Since there seems to be bug, I would rather have it fixed, than resort to something else.

Can I somehow help to debug this?

[bmeeks]

@pfsjap said in Suricata inline mode with Netgate 6100:

Can I somehow help to debug this?

Unless you can compile your own debug version of the Suricata binary for testing, then likely not. Upstream there is minimal interest in FreeBSD as most of the effort is concentrated towards Linux distros. FreeBSD is supported, but there are few users and most of the upstream developers are not fluent in the internals of FreeBSD either.

This is almost certainly a driver issue as other igc NIC families work. As I mentioned, there are subtle differences between NICs even when they use the same igc driver family in FreeBSD.

[pfsjap]

@bmeeks said in Suricata inline mode with Netgate 6100:

This is almost certainly a driver issue as other igc NIC families work. As I mentioned, there are subtle differences between NICs even when they use the same igc driver family in FreeBSD.

In that case wouldn't it be Netgate's responsibility to find out the cause?

[RobbieTT]

@pfsjap said in Suricata inline mode with Netgate 6100:

In that case wouldn't it be Netgate's responsibility to find out the cause?

I'm sure that they would like to have a look at it, even if it is an upstream issue. Clearly they don't own or have any responsibility for Suricata but there is an off-chance they may be able to help.

️

[pfsjap]

Wasn't a driver issue after all. MTU of this interface was 9000 and netmap buffer size (dev.netmap.buf_size) was 2048 (default). After setting buffer size to 9100, Suricata started in inline mode.

Found this tunable in here.

3/6/2023 -- 23:13:46 - <Info> -- Going to use 4 thread(s)
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 9] netmap:igc2-0/R@conf:host-rings=4 igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 10] netmap:igc2^0/T@conf:host-rings=4 igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 11] netmap:igc2-1/R igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 13] netmap:igc2^1/T igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 14] netmap:igc2-2/R igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 15] netmap:igc2^2/T igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 16] netmap:igc2-3/R igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 17] netmap:igc2^3/T igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- Going to use 4 thread(s)
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 18] netmap:igc2^0/R@conf:host-rings=4 igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 19] netmap:igc2-0/T@conf:host-rings=4 igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 20] netmap:igc2^1/R igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 21] netmap:igc2-1/T igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 22] netmap:igc2^2/R igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 23] netmap:igc2-2/T igc2 opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 24] netmap:igc2^3/R igc2^ opened
3/6/2023 -- 23:13:47 - <Info> -- devname [fd: 25] netmap:igc2-3/T igc2 opened
3/6/2023 -- 23:13:47 - <Notice> -- all 8 packet processing threads, 2 management threads initialized, engine started.

[bmeeks]

@pfsjap said in Suricata inline mode with Netgate 6100:

Wasn't a driver issue after all. MTU of this interface was 9000 and netmap buffer size (dev.netmap.buf_size) was 2048 (default). After setting buffer size to 9100, Suricata started in inline mode.

Found this tunable in here.

Ah! Good detective work.

The error message certainly was not helpful in this instance. It could have said "out of memory" or "insufficent buffer size" you would think. This error comes from the netmap device code within FreeBSD and has nothing to do with Suricata's use of netmap. Not many folks are using MTU sizes larger than 1500, though.