Important Info: Inline IPS Mode with Suricata and VLANs

bmeeks

@bigjohns97 said in Important Info: Inline IPS Mode with Suricata and VLANs:

So my question is, if defining the VLANs at the ESXi level do you still only need to do the parent interface? or do you need to define an instance for each VLAN?

Not sure I'm fully understanding your question. When you say "defining the VLANs at the ESXi level", do you mean configuring VLAN operation for the ESXi virtual switch? If so, then the answer is "yes, you would run Suricata on the physical interface" because the pfSense guest is tagging/processing the traffic. The ESXi virtual switch then just behaves as any hardware switch would and routes the traffic according to the VLAN tags applied by pfSense (when talking about trunk ports).

To simplify, when pfSense is involved in the VLAN tagging operation by either adding the necessary tags, or routing traffic via the tags, then Suricata needs to run on the physical interface and not the VLAN virtual interface in pfSense. When I say "physical interface", that can be one of three possibilities as follows:

For pfSense running on a bare metal machine, that means the physical NIC such as em0, igc1, etc.
For pfSense running on a hypervisor with passthrough NICs, that's the same as #1 above because the guest OS will directly communicate with the hardware NIC via the passthrough.
For pfSense running on a hypervisor with virtual hardware, you still run on the parent interface of the virtual hardware such as vmx0, vtx1, hn0, etc.

The VLAN ID nomenclature I referenced in the original post is how pfSense and FreeBSD identify VLAN interfaces. pfSense will use the physical interface name with a period and the VLAN ID appended.

bigjohns97

@bmeeks That's just it though, when you define the vlan id on the vswitch the pfsense guest just sees it as another interface/network, it doesn't see any of the VLAN tags as they are stripped at the vswitch. It still does the routing based upon layer 3 but it isn't aware of any VLAN id's because they are defined at the hypervisor level.

bmeeks

@bigjohns97 said in Important Info: Inline IPS Mode with Suricata and VLANs:

@bmeeks That's just it though, when you define the vlan id on the vswitch the pfsense guest just sees it as another interface/network, it doesn't see any of the VLAN tags as they are stripped at the vswitch. It still does the routing based upon layer 3 but it isn't aware of any VLAN id's because they are defined at the hypervisor level.

Okay. I've never used ESXi at that level, so have no experience there. If pfSense simply sees it as either a different "physical" interface or just another IP stack, then you would run Suricata on whatever "interface" pfSense is showing on its INTERFACES menu.

Netmap operation is something that is happening inside the guest OS, so the rules I spelled out earlier apply only when the guest OS sees the VLAN tags themselves and needs to use them.

Marc05

You have to use Virtual Guest Tagging (tag 4095) on the port group:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-3BB93F2C-3872-4F15-AEA9-90DEDB6EA145.html

cyb3rtr0nian

@bmeeks I was thinking about this post and I have a possible solution although it is not elegant. I would like to hear your thoughts on this.

I like to optimize security so I’m naturally interested in Suricata and I want to keep using it in IPS mode.

Proposed solution:

You use Suricata on the WAN interface, Inline IPS mode.
You don’t use Suricata on the LAN interface(s) of this PfSense. But configure VLANs in this Pfsense.
You place a managed switch in between capable of 802.1Q. You configure the ports that lead to clients as untagged.
You create a second PfSense x Suricata machine. This one only passed through traffic like a switch and uses Suricata Inline IPS on the LAN side of things, which is untagged traffic only as the managed switch has removed the tag already for traffic passed on to this Pfsense. And the managed switch will reapply the tag when untagged traffic returns from this second PfSense x Suricata box.

I think this should work theoretically. I’ll try it out.

It might be a little overkill but I like to keep both VLANs and Inline IPS mode, so I was thinking ‘the tagging part is the problem’, the managed switch will place VLAN tags on the traffic it ingests and passed through to the main PfSense, because you’ve set that up in its configuration.

So the second PfSense x Suricata machine should be configured only for Suricata and not for routing or anything.

What do you think of this solution? I think it resolves the problem, you keep your VLANs and you keep Inline IPS mode.

I ofcourse don’t suggest to purchase a large number of PfSense x Suricata machines as this might be costly (or you could buy a smaller sized, but powerful enough machine for that section of the network), but at least for my small setup I think this will work. I’ll just apply the same GID:SID Mgmt files on both Suricata’s as I want to block most things and only allow what I need.

bmeeks

@cyb3rtr0nian said in Important Info: Inline IPS Mode with Suricata and VLANs:

@bmeeks I was thinking about this post and I have a possible solution although it is not elegant. I would like to hear your thoughts on this.

I like to optimize security so I’m naturally interested in Suricata and I want to keep using it in IPS mode.

Proposed solution:

You use Suricata on the WAN interface, Inline IPS mode.

You don’t use Suricata on the LAN interface(s) of this PfSense. But configure VLANs in this Pfsense.

You place a managed switch in between capable of 802.1Q. You configure the ports that lead to clients as untagged.

You create a second PfSense x Suricata machine. This one only passed through traffic like a switch and uses Suricata Inline IPS on the LAN side of things, which is untagged traffic only as the managed switch has removed the tag already for traffic passed on to this Pfsense. And the managed switch will reapply the tag when untagged traffic returns from this second PfSense x Suricata box.

I think this should work theoretically. I’ll try it out.

It might be a little overkill but I like to keep both VLANs and Inline IPS mode, so I was thinking ‘the tagging part is the problem’, the managed switch will place VLAN tags on the traffic it ingests and passed through to the main PfSense, because you’ve set that up in its configuration.

So the second PfSense x Suricata machine should be configured only for Suricata and not for routing or anything.

What do you think of this solution? I think it resolves the problem, you keep your VLANs and you keep Inline IPS mode.

I ofcourse don’t suggest to purchase a large number of PfSense x Suricata machines as this might be costly (or you could buy a smaller sized, but powerful enough machine for that section of the network), but at least for my small setup I think this will work. I’ll just apply the same GID:SID Mgmt files on both Suricata’s as I want to block most things and only allow what I need.

I think you should ditch the second pfSense and Suricata instance in this design and instead use a dedicated Linux box running Suricata using AF_PACKET mode for inline IPS. Suricata is much more performant on Linux now. Of course this would mean configuring Suricata on the Linux box using the CLI. You would have no GUI.

The problem with VLANs and Suricata on pfSense is the requirement to use the netmap kernel device for inline IPS mode. That device runs best on physical interfaces. To run on virtual interfaces such as VLANs it must run in an emulated mode that is much slower. Another issue is the requirement to use a host stack interface in the pfSense model. Suricata performs best when it can route traffic directly between two physical interfaces using inline IPS mode.

cyb3rtr0nian

@bmeeks said in Important Info: Inline IPS Mode with Suricata and VLANs:

@cyb3rtr0nian said in Important Info: Inline IPS Mode with Suricata and VLANs:

@bmeeks I was thinking about this post and I have a possible solution although it is not elegant. I would like to hear your thoughts on this.

I like to optimize security so I’m naturally interested in Suricata and I want to keep using it in IPS mode.

Proposed solution:

You use Suricata on the WAN interface, Inline IPS mode.

You don’t use Suricata on the LAN interface(s) of this PfSense. But configure VLANs in this Pfsense.

You place a managed switch in between capable of 802.1Q. You configure the ports that lead to clients as untagged.

You create a second PfSense x Suricata machine. This one only passed through traffic like a switch and uses Suricata Inline IPS on the LAN side of things, which is untagged traffic only as the managed switch has removed the tag already for traffic passed on to this Pfsense. And the managed switch will reapply the tag when untagged traffic returns from this second PfSense x Suricata box.

I think this should work theoretically. I’ll try it out.

It might be a little overkill but I like to keep both VLANs and Inline IPS mode, so I was thinking ‘the tagging part is the problem’, the managed switch will place VLAN tags on the traffic it ingests and passed through to the main PfSense, because you’ve set that up in its configuration.

So the second PfSense x Suricata machine should be configured only for Suricata and not for routing or anything.

What do you think of this solution? I think it resolves the problem, you keep your VLANs and you keep Inline IPS mode.

I ofcourse don’t suggest to purchase a large number of PfSense x Suricata machines as this might be costly (or you could buy a smaller sized, but powerful enough machine for that section of the network), but at least for my small setup I think this will work. I’ll just apply the same GID:SID Mgmt files on both Suricata’s as I want to block most things and only allow what I need.

I think you should ditch the second pfSense and Suricata instance in this design and instead use a dedicated Linux box running Suricata using AF_PACKET mode for inline IPS. Suricata is much more performant on Linux now. Of course this would mean configuring Suricata on the Linux box using the CLI. You would have no GUI.

That’s useful information, better performance is always welcome. CLI is fine. I’m familiar with Linux, although still learning. I’ll learn the relevant CLI commands.

The problem with VLANs and Suricata on pfSense is the requirement to use the netmap kernel device for inline IPS mode. That device runs best on physical interfaces.

I understand, but I encountered the problem even when using Inline IPS mode on a physical parent interface on the LAN. Because I read what you said about not enabling IPS mode on a separate VLAN interface, instead on the parent physical interface. For me that would be fine, I don’t mind all rules will be the same for all VLANs. But in Legacy mode everything works like a charm, but as soon as Inline IPS mode is enabled all VLANs break and the VLAN aware AP becomes unreachable (also after everything is rebooted). (Also, I have other physical LAN interfaces running Inline mode not related to these VLANs, running fine without any problems.)

I think because this emulating netmap kernel device is used, the VLAN tag gets stripped from the traffic so it becomes untagged (and then it’s no longer clear where it needs to go because no VLAN tag), because it only understands untagged traffic.

So hence my proposal to move the IPS mode for LAN to the section of the network that is untagged.

To run on virtual interfaces such as VLANs it must run in an emulated mode that is much slower. Another issue is the requirement to use a host stack interface in the pfSense model. Suricata performs best when it can route traffic directly between two physical interfaces using inline IPS mode.

I don’t want to run Suricata on the discriminate VLANs, just on the parent interface. I heard hardware offloading might be a problem, but I already disabled that function. I get your advice regarding slower performance that is also undesirable, but I didn’t even get it to send any traffic after Inline was enabled, and as soon as I enable Legacy everything works perfectly again on the LAN interfaces.

Any thoughts or is the separate box the way to go here?

bmeeks

@cyb3rtr0nian said in Important Info: Inline IPS Mode with Suricata and VLANs:

I think because this emulating netmap kernel device is used, the VLAN tag gets stripped from the traffic so it becomes untagged (and then it’s no longer clear where it needs to go because no VLAN tag), because it only understands untagged traffic.

This is correct. The netmap implementation in FreeBSD does not deal with VLAN tags. They do indeed get stripped out. When I first read about the netmap device and was considering a true inline IPS mode of operation for Suricata (and Snort), netmap seemed like a perfect fit. But as I dived deeper into it and began creating the necessary interface code I came to realize that the FreeBSD netmap implementation is not ideally suited for inline IPS where you need to create a pipe between a physical NIC and the kernel (via the host stack). The host stack interface is what we use on pfSense to avoid consuming two physical interface ports for each network connection (two physical ports, IN and OUT, for the LAN or any other interface where you wanted to use Inline IPS Mode).

In the Linux implementation I mentioned, you would use two physical ports (for example, em0 and em1). Then Suricata would bridge those two physical ports and police the traffic between them by copying packets from one port to the other but analyzing them first. Any packets matching a DROP rule would not get copied. That's true Inline IPS operation, but it requires two physical ports per instance. That's not desirable for the vast majority of pfSense users.

cyb3rtr0nian

This is correct. The netmap implementation in FreeBSD does not deal with VLAN tags. They do indeed get stripped out.

Such a shame though, I mean Pfsense+Suricata is an amazing product already, but it would be even more amazing if we could combine both the segmentation of VLANs with the policing of Suricata inline IPS don’t you think? I have it running in IDS mode with several VLANs on the untagged interface. (But I still prefer the In-line IPS mode. ) Will there ever be a different emulation device used in the future capable of reading VLAN tags you think?

I think I’ll create the appliance with 2 physical NICs for a more critical section of the network for the moment as I don’t want to lose IPS mode.

bmeeks

@cyb3rtr0nian said in Important Info: Inline IPS Mode with Suricata and VLANs:

This is correct. The netmap implementation in FreeBSD does not deal with VLAN tags. They do indeed get stripped out.

Such a shame though, I mean Pfsense+Suricata is an amazing product already, but it would be even more amazing if we could combine both the segmentation of VLANs with the policing of Suricata inline IPS don’t you think? I have it running in IDS mode with several VLANs on the untagged interface. (But I still prefer the In-line IPS mode. ) Will there ever be a different emulation device used in the future capable of reading VLAN tags you think?

I seriously doubt the netmap limitations within FreeBSD itself with regard to VLANs will be fixed in the future because it would require rewriting substantial portions of the kernel network stack code. That would potentially break other things.

One thing currently implemented in the pfSense Suricata package is a software-enforced ban on using emulated netmap adapters. This was done in the interest of speed (and because of some other quirks that once existed with the emulated netmap adapter). The creation of IPS mode on an interface is restricted unless the NIC driver supports native netmap operation. VLAN interfaces are inherently a virtual interface in the operating system and thus can never pass the "native mode" test for netmap operation. The pfSense Suricata package silently adjusts any VLAN-enabled IPS interfaces to use the physical NIC instead of the VLAN virtual interface.

There have been a few beneficial updates to the netmap adapter code in recent editions of FreeBSD. None of them address the fundamental slowdown you experience when using a host stack endpoint, though.

It would be possible to make some test edits to the PHP source code of the package to enable IPS operation with virtual VLAN interfaces using the emuated netmap adapter. I suspect throughput will be pretty abysmal unless you have a very powerful CPU, but someone may want to test it. I do not have a VLAN setup in my private LAN. I keep it pretty simple, and I no longer have a large test lab either. But if you want to tinker around with emulated netmap, I can share which sections of PHP code to edit. It's fairly trivial. There is one caveat, though. The emulated adapter with VLANs will only work on the newest versions of pfSense Plus (and possibly 2.8.0 DEVEL once those snapshots resume). It will not work without throwing a syntax error on pfSense 2.7.2 CE and versions earlier. The libnetmap code update that allows proper parsing of VLAN interfaces by netmap was only added to pfSense with the move to FreeBSD 15-CURRENT. That code fix is not present in earlier pfSense kernels.