Suricata causing kernel error "netmap_grab_packets bad pkt at"
-
HARDWARE
The igb driver supports Gigabit Ethernet adapters based on the Intel
82575 and 82576 controller chips:+o Intel Gigabit ET Dual Port Server Adapter (82576)
+o Intel Gigabit VT Quad Port Server Adapter (82575)
+o Intel Single, Dual and Quad Gigabit Ethernet Controller (82580)
+o Intel i210 and i211 Gigabit Ethernet Controller
+o Intel i350 and i354 Gigabit Ethernet Controller -
Just looking for some assistance on this issue. Only seeing this issue when running inline mode, doesn't happen if I switch it to legacy. All offloading options are disabled under the advanced tab. Easy to replicate, have the same issues on 3 different systems.
Jan 21 20:33:58 kernel 438.215029 [1162] netmap_grab_packets bad pkt at 536 len 2331
Jan 21 20:33:58 kernel 438.168943 [1162] netmap_grab_packets bad pkt at 526 len 2331
Jan 21 20:32:40 kernel 360.586684 [1162] netmap_grab_packets bad pkt at 895 len 2163
Jan 21 20:32:40 kernel 360.310778 [1162] netmap_grab_packets bad pkt at 877 len 2164
Jan 21 20:32:40 kernel 360.219529 [1162] netmap_grab_packets bad pkt at 855 len 2164
Jan 21 20:32:40 kernel 360.198430 [1162] netmap_grab_packets bad pkt at 850 len 2164
Jan 21 20:32:40 kernel 360.197684 [1162] netmap_grab_packets bad pkt at 846 len 2164I don't mean to sound condescending with this reply, but it has been posted here over and over and over, since the inline IPS mode was introduced, that only certain network drivers work with netmap, and that netmap support is required of your hardware in order for inline IPS mode to function. The very error message you are seeing tells you what is wrong – your hardware (and thus the NIC driver your system is using) is not supported with netmap. When you see any error message with netmap in it, that pretty much screams your particular NIC does not support it.
That error is a network driver issue and has nothing at all to do with Suricata. Netmap is a special technology recently added to FreeBSD and even more recently as an option in pfSense. However, it has been clearly stated it only works with a handful of network drivers. You can search Google and the FreeBSD site to find network drivers for FreeBSD that support netmap. They go buy you enough of those cards to use on all the network interfaces in your firewalls and Suricata will then work using inline IPS mode.
Bill
Hi Bill,
In regards to what was said. After the update I started to see those kernel bad packets from time to time,
but the Suricata Inline works, because it drops packets and create alerts, highlighted in red. I can assure you that before the update I did not see any kernel bad packets. The issue now is that I see 4 or 5 alerts or drops depending on the rule, and before I saw like 30 or 40. Also after I reload Suricata on both interfaces, it takes like 30 minutes to 1 hour in order to see any Alerts again, and before it was almost instantly.Jan 22 12:16:47 suricata[4100]: [Drop] [1:2500108:4220] ET COMPROMISED Known Compromised or Hostile Host Traffic group 55 [Classification: Misc Attack] [Priority: 2] {UDP} 93.115.85.39:23893 -> 172.17.0.3:52303
Jan 22 12:14:59 suricata[6615]: [1:2210010:2] SURICATA STREAM 3way handshake wrong seq wrong ack [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 172.18.100.205:35869 -> 54.241.249.159:443
Jan 22 12:11:47 suricata[6615]: [1:2210056:1] SURICATA STREAM bad window update [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 79.114.30.111:63858 -> 172.18.0.10:52303
Jan 22 12:11:46 suricata[4100]: [1:2210056:1] SURICATA STREAM bad window update [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 79.114.30.111:63858 -> 172.17.0.3:52303
Jan 22 12:10:49 suricata[6615]: [1:2200094:1] SURICATA zero length padN option [Classification: (null)] [Priority: 3] {IPV6-ICMP} 0000:0000:0000:0000:0000:0000:0000:0000:143 -> ff02:0000:0000:0000:0000:0000:0000:0016:0
Jan 22 12:10:49 suricata[6615]: [1:2200094:1] SURICATA zero length padN option [Classification: (null)] [Priority: 3] {IPV6-ICMP} 0000:0000:0000:0000:0000:0000:0000:0000:143 -> ff02:0000:0000:0000:0000:0000:0000:0016:0
Jan 22 12:09:49 kernel: 789.124810 [1162] netmap_grab_packets bad pkt at 400 len 3705
Jan 22 12:02:22 php-fpm[17732]: /suricata/suricata_interfaces.php: Successful login for user 'admin' from: 172.18.0.10
Jan 22 12:02:15 php-fpm[17732]: /suricata/suricata_interfaces.php: Session timed out for user 'admin' from: 172.18.0.10
Jan 22 12:00:00 php: [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Jan 22 12:00:00 php: [pfBlockerNG] Starting cron process.
Jan 22 11:49:27 kernel: 567.230077 [1162] netmap_grab_packets bad pkt at 278 len 2285
Jan 22 11:49:27 kernel: 567.205067 [1162] netmap_grab_packets bad pkt at 273 len 2285 -
I354 which isn't terribly old, and I would assume would support netmap. The other has a Intel Quad port adapter, I apologize I'm not sure the model right off.
You absolutely know more about this than I do, so I hear what you're saying. But is the subsection of HW that supports netmap that small?
Yes, the sub-section of hardware that fully supports Netmap is quite limited. It is actually the driver that has to be fully supportive and not the hardware itself. I am not a FreeBSD expert, but judging from the posts on other sections of the forum about network issues with various NIC drivers, it seems to me FreeBSD is having some problems with the newest kernels. If you search in the other forums (not the IDS/IPS sub-forum) you can find examples of folks having various problems with different cards, especially with buffers.
This last update did move to the 3.1.2 version of the Suricata binary. That version had some netmap fixes for FreeBSD from the guy who added the netmap option to Suricata. When using inline IPS mode, you are using a completely unchanged binary from upstream. The package you see and interract with on pfSense is simply a GUI tool that generates the suricata.yaml and a few other configuration files the Suricatan binary uses. The actual scanning engine and blocking all happen within that binary piece that I did not modify from upstream.
Bill
-
I think we need a sticky here, pointing people having issues with netmap to FreeBSD upstream and not the poor pfSense Suricata package maintainer. Other serious issues with netmap include:
Broken VLANs - https://redmine.pfsense.org/issues/6690
Broken Traffic Shaper - https://redmine.pfsense.org/issues/6023There's nothing that bmeeks could do here, stop bugging him! :P
-
I think we need a sticky here, pointing people having issues with netmap to FreeBSD upstream and not the poor pfSense Suricata package maintainer. Other serious issues with netmap include:
Broken VLANs - https://redmine.pfsense.org/issues/6690
Broken Traffic Shaper - https://redmine.pfsense.org/issues/6023There's nothing that bmeeks could do here, stop bugging him! :P
Thanks dok! ;). It's true the netmap issues are outside my area of expertise. All we do in the Suricata package is add a couple of lines in the configuration file of Suricata to "turn it on". All of the real meat for netmap happens in FreeBSD itself.
Bill
-
Thanks for the reply and explanation, I understand a little better about how the package is implemented now. I'm also somewhat glad to hear that this isn't something I did incorrectly.
-
I don't mean to sound condescending with this reply, but it has been posted here over and over and over, since the inline IPS mode was introduced, that only certain network drivers work with netmap, and that netmap support is required of your hardware in order for inline IPS mode to function. The very error message you are seeing tells you what is wrong – your hardware (and thus the NIC driver your system is using) is not supported with netmap. When you see any error message with netmap in it, that pretty much screams your particular NIC does not support it.
That error is a network driver issue and has nothing at all to do with Suricata. Netmap is a special technology recently added to FreeBSD and even more recently as an option in pfSense. However, it has been clearly stated it only works with a handful of network drivers. You can search Google and the FreeBSD site to find network drivers for FreeBSD that support netmap. They go buy you enough of those cards to use on all the network interfaces in your firewalls and Suricata will then work using inline IPS mode.
Bill
Bill - Just out of curiosity, could NICs that don't support netmap native be forced to use emulation as suggested in the FreeBSD documentation?
Some aspect of the operation of netmap are controlled through sysctl
variables on FreeBSD (dev.netmap.) and module parameters on Linux
(/sys/module/netmap_lin/parameters/):dev.netmap.admode: 0
Controls the use of native or emulated adapter mode. 0 uses the
best available option, 1 forces native and fails if not avail-
able, 2 forces emulated hence never fails. -
I think we need a sticky here, pointing people having issues with netmap to FreeBSD upstream and not the poor pfSense Suricata package maintainer. Other serious issues with netmap include:
Broken VLANs - https://redmine.pfsense.org/issues/6690
Broken Traffic Shaper - https://redmine.pfsense.org/issues/6023There's nothing that bmeeks could do here, stop bugging him! :P
Hello @doktornotor ,
I don't know what is the policy of pfSense, and also I don't know if you are the proper person to ask, but being just a question I hope you will not get angry.
Do you think recompiling the Kernel in order to add some Intel drivers will solve the netmap issue (will this make some Intel NICs to be supported to work in Inline mode) ? Or maybe in the form of a kernel module, to be loaded if needed?
This will deviate from FreeBsd policy or pfSense policy?
-
I don't mean to sound condescending with this reply, but it has been posted here over and over and over, since the inline IPS mode was introduced, that only certain network drivers work with netmap, and that netmap support is required of your hardware in order for inline IPS mode to function. The very error message you are seeing tells you what is wrong – your hardware (and thus the NIC driver your system is using) is not supported with netmap. When you see any error message with netmap in it, that pretty much screams your particular NIC does not support it.
That error is a network driver issue and has nothing at all to do with Suricata. Netmap is a special technology recently added to FreeBSD and even more recently as an option in pfSense. However, it has been clearly stated it only works with a handful of network drivers. You can search Google and the FreeBSD site to find network drivers for FreeBSD that support netmap. They go buy you enough of those cards to use on all the network interfaces in your firewalls and Suricata will then work using inline IPS mode.
Bill
Bill - Just out of curiosity, could NICs that don't support netmap native be forced to use emulation as suggested in the FreeBSD documentation?
Some aspect of the operation of netmap are controlled through sysctl
variables on FreeBSD (dev.netmap.) and module parameters on Linux
(/sys/module/netmap_lin/parameters/):dev.netmap.admode: 0
Controls the use of native or emulated adapter mode. 0 uses the
best available option, 1 forces native and fails if not avail-
able, 2 forces emulated hence never fails.I don't know. I am not familiar with the FreeBSD kernel internals nor any of the tunable parameters.
Bill
-
I don't know. I am not familiar with the FreeBSD kernel internals nor any of the tunable parameters.
Bill
Ok thank you sir, appreciate the response and assistance.
-
I don't mean to sound condescending with this reply, but it has been posted here over and over and over, since the inline IPS mode was introduced, that only certain network drivers work with netmap, and that netmap support is required of your hardware in order for inline IPS mode to function. The very error message you are seeing tells you what is wrong – your hardware (and thus the NIC driver your system is using) is not supported with netmap. When you see any error message with netmap in it, that pretty much screams your particular NIC does not support it.
That error is a network driver issue and has nothing at all to do with Suricata. Netmap is a special technology recently added to FreeBSD and even more recently as an option in pfSense. However, it has been clearly stated it only works with a handful of network drivers. You can search Google and the FreeBSD site to find network drivers for FreeBSD that support netmap. They go buy you enough of those cards to use on all the network interfaces in your firewalls and Suricata will then work using inline IPS mode.
Bill
Bill - Just out of curiosity, could NICs that don't support netmap native be forced to use emulation as suggested in the FreeBSD documentation?
Some aspect of the operation of netmap are controlled through sysctl
variables on FreeBSD (dev.netmap.) and module parameters on Linux
(/sys/module/netmap_lin/parameters/):dev.netmap.admode: 0
Controls the use of native or emulated adapter mode. 0 uses the
best available option, 1 forces native and fails if not avail-
able, 2 forces emulated hence never fails.I don't know. I am not familiar with the FreeBSD kernel internals nor any of the tunable parameters.
Bill
Hello @bmeeks
As stated by @doktornotor you're not the appropiate person to ask this, but I don't know whom to ask.
I found this on Free-Bsd, I will put some quotes, and the link:
"The drivers for common NICs are already present in the GENERIC kernel"…"If the driver for the NIC is not present in GENERIC, but a driver is available, the driver will need to be loaded before the NIC can be configured and used."...
"This may be accomplished in one of two ways:
* The easiest way is to load a kernel module for the NIC using kldload(8). To also automatically load the driver at boot time, add the appropriate line to /boot/loader.conf. Not all NIC drivers are available as modules.
* Alternatively, statically compile support for the NIC into a custom kernel. "
The link is here (section 11.5.1 ):
https://www.freebsd.org/doc/en/books/handbook/config-network-setup.html
Can you direct me to the proper person to ask, if this will solve the netmap issues?
Thanks
-
Just looking for some assistance on this issue. Only seeing this issue when running inline mode, doesn't happen if I switch it to legacy. All offloading options are disabled under the advanced tab. Easy to replicate, have the same issues on 3 different systems.
Jan 21 20:33:58 kernel 438.215029 [1162] netmap_grab_packets bad pkt at 536 len 2331
Jan 21 20:33:58 kernel 438.168943 [1162] netmap_grab_packets bad pkt at 526 len 2331
Jan 21 20:32:40 kernel 360.586684 [1162] netmap_grab_packets bad pkt at 895 len 2163
Jan 21 20:32:40 kernel 360.310778 [1162] netmap_grab_packets bad pkt at 877 len 2164
Jan 21 20:32:40 kernel 360.219529 [1162] netmap_grab_packets bad pkt at 855 len 2164
Jan 21 20:32:40 kernel 360.198430 [1162] netmap_grab_packets bad pkt at 850 len 2164
Jan 21 20:32:40 kernel 360.197684 [1162] netmap_grab_packets bad pkt at 846 len 2164I don't mean to sound condescending with this reply, but it has been posted here over and over and over, since the inline IPS mode was introduced, that only certain network drivers work with netmap, and that netmap support is required of your hardware in order for inline IPS mode to function. The very error message you are seeing tells you what is wrong – your hardware (and thus the NIC driver your system is using) is not supported with netmap. When you see any error message with netmap in it, that pretty much screams your particular NIC does not support it.
That error is a network driver issue and has nothing at all to do with Suricata. Netmap is a special technology recently added to FreeBSD and even more recently as an option in pfSense. However, it has been clearly stated it only works with a handful of network drivers. You can search Google and the FreeBSD site to find network drivers for FreeBSD that support netmap. They go buy you enough of those cards to use on all the network interfaces in your firewalls and Suricata will then work using inline IPS mode.
Bill
FYI - Suricata seems to generate lots of these errors for me on supported hardware/drivers. I'm using Intel 82575/82576:
SUPPORTED DEVICES
netmap natively supports the following devices:On FreeBSD: em(4), igb(4), ixgbe(4), lem(4), re(4).
ref: https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
-
FYI - Suricata seems to generate lots of these errors for me on supported hardware/drivers. I'm using Intel 82575/82576:
https://bugs.freebsd.org/bugzilla/ - and no, it's not Suricata issue. Will not get fixed here.
-
FYI - Suricata seems to generate lots of these errors for me on supported hardware/drivers. I'm using Intel 82575/82576:
https://bugs.freebsd.org/bugzilla/ - and no, it's not Suricata issue. Will not get fixed here.
The statement that @RadOD made was a reply to @bmeeks, (I think : ) ) that told us that, when we see that kind of error, it means netmap doesn't support our NICs. The issue happens to me also with Suricata 3.1.2, and it didn't happen with previous version.
So you say that it's a bug, and @bmeeks says that it happens with cards that are not supported.
This is my understanding, and it's confusing, meaning that 2 veteran users stating different things (No pointing fingers here, just want to be inline)
Sorry, if it's just me.
-
Got these errors sometimes multiple times in a minute on two pfSense SG-8860 firewalls (igb interfaces) with Suricata in inline mode. They are now being reverted to legacy mode due to multiple problems.
-
I also am using supported hardware and get quite a few of these bad pkt errors as well. I think I am going back to legacy mode for now. It is better than it was a year ago when inline really bugged things up. I will go back to it in the future. Real shame since legacy doesn't stop everything you want.
