Suricata inline mode IPS and VLANS

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks I know, I saw your past comments, that's why I asked directly. I thought something changed after a year or two. Thanks for the tip, I'll try to take it with them.

Do you know how can I tell which version of netmap is included with Free-BSD 11.3-stable, or where to ask?

Thanks again

No, off the top of my head I don't recall the netmap version for FreeBSD 11.3-stable. I do know that it is shown in one of the include files from the FreeBSD sources, though. I can maybe research which one a bit later. I have to be away the remainder of today.

[NRgia]

No worries, answer when you have time to spare.
Thank you for your support

[NRgia]

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

[NRgia]

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

In the mean time I found this on pfSense reddit:
https://www.reddit.com/r/PFSENSE/comments/foucva/snort_inline_feature_25/

If it happens to Snort also, I think we can exclude Suricata.

Yes, my suspicions are it's something with netmap and the NIC drivers and/or the FreeBSD kernel.

As per your and others comments from here https://redmine.pfsense.org/issues/6690 it affects VLAN tags, which is a common setup unfortunately. Sorry to see it will not get fixed. I know it's a Netmap thing. Maybe when you have time, and know where to look for the Netmap version...I want to crawl some Netmap forums, or Githubs, or ask away about this, or find the version which is affected
Thanks

[bmeeks]

@NRgia:
The hub for everything netmap is this guy's Github: https://github.com/luigirizzo/netmap. He is the netmap inventor and primary maintainer. I think he did all the FreeBSD work as well, but not 100% sure on that. So that would be a place to start with the VLAN questions.

I've done some research on this in the distant past, and my memory is foggy now, but I seem to recall that some NICs' onboard hardware does things with VLAN tags BEFORE handing off to the netmap device. So the netmap device does not get to see the tags. That would, of course, interfere with what Suricata is trying to do.

I found that there is a defined NETMAP_API_VERSION tag in some of the netmap include files. That might be the "version" thing I was remembering. As I said above, it's been quite some time since I last dug into netmap.

[NRgia]

@bmeeks It's more than ok, thanks

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

VLANs can be good I guess, but the better solution is usually to have actual independent physical interfaces for each subnet. This would especially be true when you want to utilize an IPS with different rules for the different protected subnets.

[NRgia]

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

[NRgia]

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks It's more than ok, thanks

Reading quickly through a few issues on that link I sent you indicates that in general, when you have VLANs with netmap, you should run netmap on the parent interface only. So translating that to pfSense would mean running Suricata on the parent interface and not each VLAN. However, that method might not fit into the scenario where you want different rules in place for each VLAN.

I only run Netmap on the parent interface, in my picture the parent interface is ix2 - LAN, which is not tagged. On VLANS I'm not running Netmap, VLANs where created for the IOT devices, and Guests devices.

I also did some reading and found this:

"if you are receiving VLAN-tagged packets, netmap applications (with patched drivers) may not see the VLAN tag because receive VLAN offloading is enabled (and so VLAN tags are stripped by the NIC). To disable it use

ethtool -K eth0 rxvlan off"

taken from here:
https://github.com/luigirizzo/netmap/blob/38cbc64d005029324cd63896fe9ad644d7d9d2bd/LINUX/README.md

but I don't think it's that easy, and there is no ethtool on pfSense

I think I've also read that some NIC drivers won't honor that command because the feature is "enabled" inside firmware on the card and cannot be changed without reflashing the firmware.

How can I try to set that on pfSense, should I set something in bootloader.conf or it's another tool, to try it out ?

Go have a look at this Sticky Post at the top of this forum: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces. You can do this on FreeBSD using the ifconfig command in many cases.

[NRgia]

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which should I disable further, if "rx_vlan" is off or not, from what I see

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which if "rx_vlan" is off or not, from what I see

I would lean towards VLAN_HWFILTER being a potential problem. But turning that off might break all of your VLAN routing.

You may have to think about abandoning Inline Mode if you have to use the VLANs. There are just too many issues with the netmap device itself and VLANs. These issues are totally outside the realm of Suricata. All Suricata does is call an API function to open a netmap device tunnel on an interface.

Oh, and that link I gave you to the Sticky Post was not about VLANs. It was just to illustrate use of the ifconfig command. That user was working on some different issues and tuning around those problems.

[NRgia]

@bmeeks I will try to compose a mail to Luigi, before doing something to my pfSense. Maybe I get lucky and he will respond. VLANS I can't abandon, I need them, for the moment I disabled Suricata on LAN, and run it only on WAN, I don't have any other choice for now
Thanks again :)

[NRgia]

@bmeeks said in Suricata inline mode IPS and VLANS:

@NRgia said in Suricata inline mode IPS and VLANS:

@bmeeks Thanks, I read that in the past, and I applied that info.

I only have the following:

I don't know which if "rx_vlan" is off or not, from what I see

I would lean towards VLAN_HWFILTER being a potential problem. But turning that off might break all of your VLAN routing.

You may have to think about abandoning Inline Mode if you have to use the VLANs. There are just too many issues with the netmap device itself and VLANs. These issues are totally outside the realm of Suricata. All Suricata does is call an API function to open a netmap device tunnel on an interface.

Oh, and that link I gave you to the Sticky Post was not about VLANs. It was just to illustrate use of the ifconfig command. That user was working on some different issues and tuning around those problems.

Hello Bill,

I know it's an old post, but I wanted to tell you that you were partially right about my issue, which seems that hunts many people.

I found this bug:
https://github.com/luigirizzo/netmap/issues/703

I did not understand if it's fixed or not, but I looked to this PR afterwards:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236584

After I disabled the following:

ifconfig ix2 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum

I can start Suricata in inline mode, on parent interface that have other VLANs.

I though it was ok to share, if this is not known already.
Tested on pfsense 2.5.0, but I don't think it matters.

[bmeeks]

@NRgia said in Suricata inline mode IPS and VLANS:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236584

After I disabled the following:

ifconfig ix2 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum

I can start Suricata in inline mode, on parent interface that have other VLANs.

I though it was ok to share, if this is not known already.
Tested on pfsense 2.5.0, but I don't think it matters.

I think this is a driver-specific issue. The em series driver appears to be the one not honoring the disable option. Other drivers do (like your ix series).

There are other features of FreeBSD networking that do not play well with the netmap device either. One of them is limiters (packet shaping). Several folks have reported that when that is enabled in pfSense (FreeBSD, actually), network traffic stops on the interface. Also have reports of the traffic graph function not working when netmap is running on an interface.