Suricata InLine with igb NICs
-
Here are a few Netmap-related links I've found. There are some references in these about various netmap errors and issues, particularly around stripping of VLAN tags and problems with flow control. Lots of the issues are NIC driver specific.
https://github.com/luigirizzo/netmap/blob/master/LINUX/README
http://freebsd.1045724.x6.nabble.com/Netmap-ixgbe-stripping-Vlan-tags-td5838105.html
-
@bmeeks said in Suricata InLine with igb NICs:
Here are a few Netmap-related links I've found. There are some references in these about various netmap errors and issues, particularly around stripping of VLAN tags and problems with flow control. Lots of the issues are NIC driver specific.
https://github.com/luigirizzo/netmap/blob/master/LINUX/README
http://freebsd.1045724.x6.nabble.com/Netmap-ixgbe-stripping-Vlan-tags-td5838105.html
I've read through the man pages and netmap code several times now (which is why I'm so confident I know what the errors mean: https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap.c#L1169 )
VLAN tag stripping isn't an issue for me, but there was an interesting bit in that link:
When you switch an interface to netmap mode it does a soft-reset first. That reverts the vlanhwfilter configuration to default on. It's not netmap that does it but the driver. It seems to happen in or around ixgbe_setup_vlan_hw_support().
I just tested this on igb and em drivers, and both keep the vlanhwfilter setting across a netmap restart (along with other settings -- checksum offloading most importantly).
I think that the "bad pkt" error is "resolved" for both of us and probably more broadly. My guess is that the remaining errors are normal network noise that is just noisier than usual because the code writes to syslog for every dropped (because malformed) packet.
A flood of "bad pkt" errors would result if a user had a misconfiguration or an incompatible card (eg MTU set to 9000 to support jumbo frames with a netmap.buf_size of default 2048 would result in huge numbers of "bad pkt" errors) -- and so we're thinking that "bad pkt" means something isn't working correctly. Really it just means what it says -- a bad packet was received that is in violation of both our MTU and our buf_size. The packet should be dropped. Why someone is sending us a packet of size 4939 when we're advertising an MTU of 1500 is a good question.
To really get to the bottom of it I would need to capture and decode one of the oversized packets. Without having to capture enormous amounts of traffic the best way I can think of would be to try to recompile netmap/suricata with an expanded error message that output the packet in base64 to the log for analysis. I'll see how complicated is.
I started a write-up on how to troubleshoot netmap errors and got discouraged when my initial post was rejected by Akismet. If I write it up and send it your way, can you get it posted Bill?
-
@boobletins said in Suricata InLine with igb NICs:
I started a write-up on how to troubleshoot netmap errors and got discouraged when my initial post was rejected by Akismet. If I write it up and send it your way, can you get it posted Bill?
Sure! Write it up in a format that would make a good Sticky Post to put at the top of this forum along with the others. Give it a title to make it clear what it's about. Be sure to give yourself the credit in the notes, and you can even ask one of the Forum moderators to post the sticky for you. I have asked them to post mine in the past. You can ask @jimp or @johnpoz if they will make it a Sticky Post in this forum. If you run into difficulties or have a question, just let me know.
-
Nice work.
-
@boobletins said in Suricata InLine with igb NICs:
I'll see how complicated is.
It looks like this would require a kernel rebuild which I'm not really up to -- I'd then have to run that experimental build on my firewall (and I've never built one before).
-
@boobletins said in Suricata InLine with igb NICs:
@boobletins said in Suricata InLine with igb NICs:
I'll see how complicated is.
It looks like this would require a kernel rebuild which I'm not really up to -- I'd then have to run that experimental build on my firewall (and I've never built one before).
Enabling debugging or extra error messages from within netmap itself will require rebuilding the kernel module. Though I've never done it, you might be able to build a compatible module with debugging enabled using a vanilla FreeBSD 11.2 machine. Then just copy the kernel module over to pfSense. If you have virtual machines, you could do this with not much risk. Just save a snapshot, install the new netmap kernel module and give it a try. If it breaks badly, just restore the previous snapshot.
Turning on debugging with Suricata is relatively easy, but I don't think any really useful information will be gleaned from Suricata itself. I think this issue is between the NIC drivers, the netmap kernel module and the kernel itself.
-
That's how I built the hyperscan module -- a fresh VM and then followed the directions.
If we think it's possible to do something similar and just copy over the netmap module, then I can try that.
-
@boobletins said in Suricata InLine with igb NICs:
That's how I built the hyperscan module -- a fresh VM and then followed the directions.
If we think it's possible to do something similar and just copy over the netmap module, then I can try that.
I think it should work just fine. There is nothing customized about the netmap module in pfSense. They just enable the module to be built along with the kernel. I believe there are some minor pfSense-specific tweaks to the FreeBSD kernel code, but nothing to the netmap module itself.
-
boobletins...From bmeeks post here: https://helpmanual.io/man4/netmap-freebsd/, which you likely already read, I wanted to get your take on this, "The only parameter worth modifying is dev.netmap.buf_num as it impacts the total amount of memory used by netmap." The default value is 163840; I checked my pfSense and the value matched. I'm wondering if we should change this value as well as the dev.netmap.buf_size? Just a thought.
-
You can certainly do that -- mine is higher than default, but it won't help with any "bad pkt" errors if that's what you're trying to solve.
Really what you would be doing there is buying yourself a larger buffer if Suricata starts falling behind -- you've set aside more RAM in case of a backlog of packets.