Suricata inline mode IPS and VLANS

NRgia

Hello @bmeeks
Before I posted I read most of your other hints like this one:

https://forum.netgate.com/topic/138564/suricata-inline-mode-trunk-interface

The issue is the same as in the above topic,but with pfSense version 2.4.5 and Suricata 5.0.2. When I start IPS mode using Netmap on the trunk LAN, the LAN interface and all VLANs will stop responding after an intial bootup(I cann't manage pfSense or use any devices). I have however a window of opportunity to disable Suricata after booting, so the interface will function for a minute or so, but only after a reboot.

My LAN config is the following:
Network interfaces.jpg

ix2 - LAN trunk
ix2.20 - VLAN 20
ix2.30 - VLAN 30

The board is : A2SDi-4C-HLN4F
The NICs are Intel X553 and the chip manual can be found here:
https://www.intel.com/content/www/us/en/products/docs/processors/atom/c-series/atom-c3000-family-programmer-reference-manual.html

I will insert some relevant dmesg info:

ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xdd600000-0xdd7fffff,0xdd804000-0xdd807fff at device 0.0 on pci7
ix2: Using MSI-X interrupts with 5 vectors
ix2: Ethernet address: ac:1f:6b:45:fa:8a
ix2: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xdd400000-0xdd5fffff,0xdd800000-0xdd803fff at device 0.1 on pci7
ix3: Using MSI-X interrupts with 5 vectors
ix3: Ethernet address: ac:1f:6b:45:fa:8b
ix3: netmap queues/slots: TX 4/2048, RX 4/2048

If no VLANS are active, Netmap will work on both interfaces.
For the moment I couldn't add any logs with netmap active on LAN interface, bacause I will get locked out from managing pfSense, but I think I can connect through opnVPN via WAN and take a log that way?

Please let me know if this is an issue with Netmap?
I am reffering to your comment on this bug on pfSense redmine here: https://redmine.pfsense.org/issues/6690
which is unsolved?

Let me know what is your insight about this.

Thank you as always.

bmeeks

I don't see the ix NIC listed as being supported by netmap on the FreeBSD page here: https://www.freebsd.org/cgi/man.cgi?query=netmap. Scroll down to Supported Devices. I see ixgbe and ixl listed, but not ix.

If the ix driver is not supported by netmap, then that will likely be the cause of your issue.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

I don't see the ix NIC listed as being supported by netmap on the FreeBSD page here: https://www.freebsd.org/cgi/man.cgi?query=netmap. Scroll down to Supported Devices. I see ixgbe and ixl listed, but not ix.

If the ix driver is not supported by netmap, then that will likely be the cause of your issue.

AFAIK according to Intel driver page, there is no ixgbe driver for FreeBsd, only ix :

The documentation can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/ethernet-controller-x550-feature-support-matrix.pdf - under Table 5

Also if I disable the VLANS, it will work in Inline mode. I'm not questioning you on this...just trying to understand, why is it working without VLANS? If the driver isn't compatible, it shouldn't work at all in Inline mode...or I'm not understanding something?

Also according to this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233771
ix it's inlcuded in ixgbe, it's just a different name. - Please see comment 3 - and check the driver link, is ix
Thanks

bmeeks

@NRgia: VLANs and netmap have not played well together in the past. Most usually it is because the NIC drivers fiddle with the VLAN tags themselves. At least that is my understanding. I'm not a kernel networking guru.

I will say this, being a supported driver on FreeBSD and having netmap support are not necessarily the same thing.

Suricata on pfSense, when used with Inline IPS Mode is a straight stock binary. That means it is using the binary code from upstream 100% as-is with no modifications whatsoever. So issues like VLANs not working or problems with limiters really needs to be taken to Suricata upstream. They wil ask you for lots of details, though, when reporting a bug or issue. So be ready. The Redmine bug site for Suricata is here: https://redmine.openinfosecfoundation.org/projects/suricata.

Word of warning --- don't use the word "pfSense" anywhere in your bug report. Just say "I'm running Suricata 5.0.2 on FreeBSD-STABLE using netmap inline mode." I'm telling you this to avoid them sending you in a circle right back here, because if you say "pfSense" that's all they will hear and will be inclined to say "take it to the those guys". But in the case of Suricata Inline IPS Mode, pfSense is not modifying the Suricata code.

NRgia

@bmeeks I know, I saw your past comments, that's why I asked directly. I thought something changed after a year or two. Thanks for the tip, I'll try to take it with them.

Do you know how can I tell which version of netmap is included with Free-BSD 11.3-stable, or where to ask?

Thanks again

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks I know, I saw your past comments, that's why I asked directly. I thought something changed after a year or two. Thanks for the tip, I'll try to take it with them.

Do you know how can I tell which version of netmap is included with Free-BSD 11.3-stable, or where to ask?

Thanks again

No, off the top of my head I don't recall the netmap version for FreeBSD 11.3-stable. I do know that it is shown in one of the include files from the FreeBSD sources, though. I can maybe research which one a bit later. I have to be away the remainder of today.

NRgia

No worries, answer when you have time to spare.
Thank you for your support

NRgia

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

As per your and others comments from here https://redmine.pfsense.org/issues/6690 it affects VLAN tags, which is a common setup unfortunately. Sorry to see it will not get fixed. I know it's a Netmap thing. Maybe when you have time, and know where to look for the Netmap version...I want to crawl some Netmap forums, or Githubs, or ask away about this, or find the version which is affected
Thanks

bmeeks

@NRgia:
The hub for everything netmap is this guy's Github: https://github.com/luigirizzo/netmap. He is the netmap inventor and primary maintainer. I think he did all the FreeBSD work as well, but not 100% sure on that. So that would be a place to start with the VLAN questions.

I've done some research on this in the distant past, and my memory is foggy now, but I seem to recall that some NICs' onboard hardware does things with VLAN tags BEFORE handing off to the netmap device. So the netmap device does not get to see the tags. That would, of course, interfere with what Suricata is trying to do.

I found that there is a defined NETMAP_API_VERSION tag in some of the netmap include files. That might be the "version" thing I was remembering. As I said above, it's been quite some time since I last dug into netmap.

NRgia

@bmeeks It's more than ok, thanks

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

VLANs can be good I guess, but the better solution is usually to have actual independent physical interfaces for each subnet. This would especially be true when you want to utilize an IPS with different rules for the different protected subnets.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

Go have a look at this Sticky Post at the top of this forum: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces. You can do this on FreeBSD using the ifconfig command in many cases.

NRgia

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which should I disable further, if "rx_vlan" is off or not, from what I see

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which if "rx_vlan" is off or not, from what I see

I would lean towards VLAN_HWFILTER being a potential problem. But turning that off might break all of your VLAN routing.

You may have to think about abandoning Inline Mode if you have to use the VLANs. There are just too many issues with the netmap device itself and VLANs. These issues are totally outside the realm of Suricata. All Suricata does is call an API function to open a netmap device tunnel on an interface.

Oh, and that link I gave you to the Sticky Post was not about VLANs. It was just to illustrate use of the ifconfig command. That user was working on some different issues and tuning around those problems.

NRgia

@bmeeks I will try to compose a mail to Luigi, before doing something to my pfSense. Maybe I get lucky and he will respond. VLANS I can't abandon, I need them, for the moment I disabled Suricata on LAN, and run it only on WAN, I don't have any other choice for now
Thanks again :)