Suricata inline mode IPS and VLANS

bmeeks

I don't see the ix NIC listed as being supported by netmap on the FreeBSD page here: https://www.freebsd.org/cgi/man.cgi?query=netmap. Scroll down to Supported Devices. I see ixgbe and ixl listed, but not ix.

If the ix driver is not supported by netmap, then that will likely be the cause of your issue.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

I don't see the ix NIC listed as being supported by netmap on the FreeBSD page here: https://www.freebsd.org/cgi/man.cgi?query=netmap. Scroll down to Supported Devices. I see ixgbe and ixl listed, but not ix.

If the ix driver is not supported by netmap, then that will likely be the cause of your issue.

AFAIK according to Intel driver page, there is no ixgbe driver for FreeBsd, only ix :

The documentation can be found here:
https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/ethernet-controller-x550-feature-support-matrix.pdf - under Table 5

Also if I disable the VLANS, it will work in Inline mode. I'm not questioning you on this...just trying to understand, why is it working without VLANS? If the driver isn't compatible, it shouldn't work at all in Inline mode...or I'm not understanding something?

Also according to this https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233771
ix it's inlcuded in ixgbe, it's just a different name. - Please see comment 3 - and check the driver link, is ix
Thanks

bmeeks

@NRgia: VLANs and netmap have not played well together in the past. Most usually it is because the NIC drivers fiddle with the VLAN tags themselves. At least that is my understanding. I'm not a kernel networking guru.

I will say this, being a supported driver on FreeBSD and having netmap support are not necessarily the same thing.

Suricata on pfSense, when used with Inline IPS Mode is a straight stock binary. That means it is using the binary code from upstream 100% as-is with no modifications whatsoever. So issues like VLANs not working or problems with limiters really needs to be taken to Suricata upstream. They wil ask you for lots of details, though, when reporting a bug or issue. So be ready. The Redmine bug site for Suricata is here: https://redmine.openinfosecfoundation.org/projects/suricata.

Word of warning --- don't use the word "pfSense" anywhere in your bug report. Just say "I'm running Suricata 5.0.2 on FreeBSD-STABLE using netmap inline mode." I'm telling you this to avoid them sending you in a circle right back here, because if you say "pfSense" that's all they will hear and will be inclined to say "take it to the those guys". But in the case of Suricata Inline IPS Mode, pfSense is not modifying the Suricata code.

NRgia

@bmeeks I know, I saw your past comments, that's why I asked directly. I thought something changed after a year or two. Thanks for the tip, I'll try to take it with them.

Do you know how can I tell which version of netmap is included with Free-BSD 11.3-stable, or where to ask?

Thanks again

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks I know, I saw your past comments, that's why I asked directly. I thought something changed after a year or two. Thanks for the tip, I'll try to take it with them.

Do you know how can I tell which version of netmap is included with Free-BSD 11.3-stable, or where to ask?

Thanks again

No, off the top of my head I don't recall the netmap version for FreeBSD 11.3-stable. I do know that it is shown in one of the include files from the FreeBSD sources, though. I can maybe research which one a bit later. I have to be away the remainder of today.

NRgia

No worries, answer when you have time to spare.
Thank you for your support

NRgia

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

As per your and others comments from here https://redmine.pfsense.org/issues/6690 it affects VLAN tags, which is a common setup unfortunately. Sorry to see it will not get fixed. I know it's a Netmap thing. Maybe when you have time, and know where to look for the Netmap version...I want to crawl some Netmap forums, or Githubs, or ask away about this, or find the version which is affected
Thanks

bmeeks

@NRgia:
The hub for everything netmap is this guy's Github: https://github.com/luigirizzo/netmap. He is the netmap inventor and primary maintainer. I think he did all the FreeBSD work as well, but not 100% sure on that. So that would be a place to start with the VLAN questions.

I've done some research on this in the distant past, and my memory is foggy now, but I seem to recall that some NICs' onboard hardware does things with VLAN tags BEFORE handing off to the netmap device. So the netmap device does not get to see the tags. That would, of course, interfere with what Suricata is trying to do.

I found that there is a defined NETMAP_API_VERSION tag in some of the netmap include files. That might be the "version" thing I was remembering. As I said above, it's been quite some time since I last dug into netmap.

NRgia

@bmeeks It's more than ok, thanks

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

VLANs can be good I guess, but the better solution is usually to have actual independent physical interfaces for each subnet. This would especially be true when you want to utilize an IPS with different rules for the different protected subnets.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

Go have a look at this Sticky Post at the top of this forum: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces. You can do this on FreeBSD using the ifconfig command in many cases.

NRgia

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which should I disable further, if "rx_vlan" is off or not, from what I see

bmeeks

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which if "rx_vlan" is off or not, from what I see

I would lean towards VLAN_HWFILTER being a potential problem. But turning that off might break all of your VLAN routing.

You may have to think about abandoning Inline Mode if you have to use the VLANs. There are just too many issues with the netmap device itself and VLANs. These issues are totally outside the realm of Suricata. All Suricata does is call an API function to open a netmap device tunnel on an interface.

Oh, and that link I gave you to the Sticky Post was not about VLANs. It was just to illustrate use of the ifconfig command. That user was working on some different issues and tuning around those problems.

NRgia

@bmeeks I will try to compose a mail to Luigi, before doing something to my pfSense. Maybe I get lucky and he will respond. VLANS I can't abandon, I need them, for the moment I disabled Suricata on LAN, and run it only on WAN, I don't have any other choice for now
Thanks again :)

NRgia

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which if "rx_vlan" is off or not, from what I see

I would lean towards VLAN_HWFILTER being a potential problem. But turning that off might break all of your VLAN routing.

You may have to think about abandoning Inline Mode if you have to use the VLANs. There are just too many issues with the netmap device itself and VLANs. These issues are totally outside the realm of Suricata. All Suricata does is call an API function to open a netmap device tunnel on an interface.

Oh, and that link I gave you to the Sticky Post was not about VLANs. It was just to illustrate use of the ifconfig command. That user was working on some different issues and tuning around those problems.

Hello Bill,

I know it's an old post, but I wanted to tell you that you were partially right about my issue, which seems that hunts many people.

I found this bug:
https://github.com/luigirizzo/netmap/issues/703

I did not understand if it's fixed or not, but I looked to this PR afterwards:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236584

After I disabled the following:

ifconfig ix2 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum

I can start Suricata in inline mode, on parent interface that have other VLANs.

I though it was ok to share, if this is not known already.
Tested on pfsense 2.5.0, but I don't think it matters.