Alerts received on incorrect interface

bwhaley · Feb 16, 2023, 11:41 PM

I'm currently running a 6100 appliance. Earlier today, I reconfigured my 3 networks to all come into my appliance via 10Gb DAQ on ix1 (ix1, ix1.10, ix1.20|Secure,IoT,MoB respectively). Secure is untagged, IoT is VLAN 10 and MoB is VLAN 20. All of my devices on the networks appear to have pulled the correct IP Addresses from my pfSense DHCP scopes. I noticed however, that I started seeing some random Snort alerts for benign events showing up under the wrong interface. Events that were sourced from an IP address on the MoB network were showing under the Secure interface.

In some cases, it showed under both interfaces. I removed Snort and installed Suricata to see if I saw the same results, and while they were not as frequent, I did still see some alerts that were under the wrong interface.

l completely uninstalled Snort, including the settings and reinstalled but still see the same issue. If I restore the network configuration with each network/vlan on its own discrete interface, the issue does not happen.

The following questions came to mind: Is inspecting multiple networks trunked into a single interface an issue? Is the untagged network on the interface possibly an issue?

Thanks for any suggestions you guys may have.

bmeeks · Feb 17, 2023, 12:01 AM

Snort automatically puts the monitored interfaces in promiscuous mode. When the interface is a VLAN, that means Snort sees the traffic from all VLANs defined on the same parent interface. So in your case ix1 is the parent, and any VLANs defined there will be seen by Snort.

~~There is an option on the INTERFACE SETTINGS tab to toggle promiscuous mode off,~~ but I'm not sure how well that will actually work in the case of VLANs because within FreeBSD a VLAN is created as a virtual interface on top of a physical parent interface.

Edit: scratch my earlier suggestion about the promiscuous mode switch. Only Suricata has that one.

Unless you need radically different rules on the various VLANs, I usually recommend folks run the IDS/IPS on the physical parent. That produces a single instance of the daemon and thus conserves CPU cycles and RAM. And with promiscuous mode, the instance on the parent interface sees all the VLAN traffic anyway.

If you must run substantially different rules on each VLAN, then you really have no option but to put each on a separate physical interface.

bwhaley · Feb 17, 2023, 12:02 AM

@bmeeks Thank you for the quick response. I will remove the VLAN interfaces and leave just the physical ix1 interface.

Thanks again!

bmeeks · Feb 17, 2023, 12:04 AM

@bwhaley said in Alerts received on incorrect interface:

@bmeeks Thank you for the quick response. I will remove the VLAN interfaces and leave just the physical ix1 interface.

Thanks again!

You're welcome! Hope all is well with you.

sstatjm · Mar 10, 2023, 2:48 PM

I am wondering if you could guide me. With some step by step instructions with it then on Suricata.

Since suricata should only run on the parent interface only and not create anything for the vlans that run on the same parent interface.

bmeeks · Mar 10, 2023, 2:57 PM

@sstatjm said in Alerts received on incorrect interface:

I am wondering if you could guide me. With some step by step instructions with it then on Suricata.

Since suricata should only run on the parent interface only and not create anything for the vlans that run on the same parent interface.

A typical VLAN setup might look like this assuming you have the following conditions:

Physical interface = em1 (this might be igb, or ix, or something else - it is determined by your particular NIC hardware).
VLAN100 = LAN VLAN
VLAN200 = IoT VLAN
VLAN300 = Guest Wireless

Within pfSense, if you had the configuration above, you would see a total of four interfaces available: em1, em1.100, em1.200, and em1.300. In the Suricata GUI they would also show with any assigned friendly interface names/descriptions.

So, when we say run Suricata on the parent interface, that would be em1 in this example. Suricata would then see all the traffic that runs across the physical em1 interface including each VLAN's traffic.

sstatjm · Mar 10, 2023, 3:12 PM

So if mine looks like this

Should i delete the last 4 and only leave the LAN only?

bmeeks · Mar 10, 2023, 3:26 PM

@sstatjm:
Yes, so long as you are running the exact same rule sets on each VLAN, then you can conserve resources by running a single instance on the parent interface.

In most cases the single instance is fine.

sstatjm · Mar 10, 2023, 3:47 PM

Should I turn promiscuous mode back on also?

bmeeks · Mar 10, 2023, 3:54 PM

@sstatjm said in Alerts received on incorrect interface:

Should I turn promiscuous mode back on also?

Yes, you can.

sstatjm · Mar 10, 2023, 5:01 PM

ok great I will give that a try. I am also wondering if running it in inline mode would be a problem. Before I ran it on all interfaces and everything crashed. And I had to rebuild from scratch

bmeeks · Mar 10, 2023, 7:00 PM

@sstatjm said in Alerts received on incorrect interface:

ok great I will give that a try. I am also wondering if running it in inline mode would be a problem. Before I ran it on all interfaces and everything crashed. And I had to rebuild from scratch

I do not recommend using Inline IPS Mode with VLANs. The netmap kernel device used for Inline Mode and VLANs do not play well together. Some work is being done in the FreeBSD kernel, but it is not ready yet.

sstatjm · Mar 10, 2023, 8:37 PM

this is what suricata looks like now

I deleted all the VLANs i had there before

michmoor · Mar 10, 2023, 8:54 PM

@bmeeks
I have multiple VLANs running on a LAGG.
How do i get suricata running on the parent interface if the parent interfaec is technically a LAG?
I cant assign the interfaec per documentation.

bmeeks · Mar 11, 2023, 12:33 AM

@michmoor said in Alerts received on incorrect interface:

@bmeeks
I have multiple VLANs running on a LAGG.
How do i get suricata running on the parent interface if the parent interfaec is technically a LAG?
I cant assign the interfaec per documentation.

Suricata is not really setup to work with LAGG interfaces. It might sort of work, but things can be weird. Interfaces such as LAGG, GIF, and even VLANs are actually virtual interfaces down at the kernel level. As such, the shared libraries such as libpcap used for IDS and Legacy Mode blocking, and the netmap kernel device used for Inline IPS Mode, don't work well with the virtual interfaces. That's because they are designed to work with hardware NIC drivers directly.

Some work is underway by others to make netmap and LAGG and Bridge interfaces behave better together. But none of that is in pfSense yet (nor anywhere else).

Having these IDS/IPS packages coded and working for complex setups is asking a bit much for volunteer maintainers and free software. To make that work well takes customized software both at the OS level and in userland. That's why the big boys charge so much for their products that do this.

michmoor · Apr 11, 2023, 4:42 PM

@bmeeks You really should give write-ups on the inner workings of pfSense and the packages. You give really good details on why something works the way it does.
That said, the work around will be to run an instance of Suricata on each VLAN.
Appreciate the response. Appreciate the work involved.

sgw · Apr 11, 2023, 6:34 PM

I think I also see this on a Netgate 7100.

WAN ... lagg0.4090
LAN ... lagg0.4091

I have snort on LAN and have the strange behavior that I see blocked IPs, but zero alerts on that interface!

So no way to suppress things right now. and somehow WAN traffic snorted when I only want to process LAN traffic.

hints welcome ;-)

edit: I also see blocking when blocking is disabled on all snort interfaces.
This is on an updated pfSense-23.01, snort installed and configured only 2 days ago (from scratch).

edit 2: removed snort and installed suricata. So far this seems to work much better. Although I haven't yet seen a single alert and/or block (alerting only configured now). But no blocks without alerts so far.

bmeeks · Apr 12, 2023, 1:28 PM

@sgw said in Alerts received on incorrect interface:

I think I also see this on a Netgate 7100.

WAN ... lagg0.4090
LAN ... lagg0.4091

I have snort on LAN and have the strange behavior that I see blocked IPs, but zero alerts on that interface!

So no way to suppress things right now. and somehow WAN traffic snorted when I only want to process LAN traffic.

hints welcome ;-)

edit: I also see blocking when blocking is disabled on all snort interfaces.
This is on an updated pfSense-23.01, snort installed and configured only 2 days ago (from scratch).

edit 2: removed snort and installed suricata. So far this seems to work much better. Although I haven't yet seen a single alert and/or block (alerting only configured now). But no blocks without alerts so far.

If you saw blocks with no corresponding alert, then it is highly likely you had another uncontrolled (or zombie) Snort process running on the interface. This can happen sometimes. You wind up with duplicate Snort instances on the same interface, but only one of those instances responds to- and provides updates to- the GUI.

LAGG interfaces are really not well supported. Ditto for Bridge interfaces. The IDS/IPS packages expect plain vanilla interfaces. They also do not always behave as expected with VLANs. All of these: VLANs, LAGGs, and Bridges are actually virtual interfaces created by the FreeBSD operating system. The IDS/IPS code expects "real, physical" interfaces. It can perhaps work with a virtual interface, but results will be spotty.

Some work is underway by others to patch the netmap kernel device so that it better supports LAGG, Bridge, and VLAN interfaces.

sgw · Apr 12, 2023, 4:57 AM

@bmeeks thanks for the explanation.
Looking at that system it seems to work OK. For sure I don't know what maybe is missing from the shown alerts etc

I am in the process of trying to learn the basics and decide if and where to use IDS/IPS. My customers mostly don't have any internal IT staff, so who maintains it, who even looks at the alerts etc?

So I am running it in alerting-only and try to see what happens in their subnets. And I learn things :-) thanks

bmeeks · Apr 12, 2023, 1:31 PM

@sgw said in Alerts received on incorrect interface:

My customers mostly don't have any internal IT staff, so who maintains it, who even looks at the alerts etc?

In that circumstance, I would be extremely reluctant to deploy an IDS/IPS in blocking mode. Any IDS/IPS is subject to generate false positives, and if no qualified admin is there to immediately assess the alert and disable or otherwise remediate the resulting block, the users will be very frustrated.

And there is a warning for simple alerting mode as well. If the firewall is a device without a spinning hard disk (or a very good quality server-grade SSD), the amount of logged data from an IDS/IPS can result in quickly exhausting the write cycles of the disk. Prime example would be a smallish eMMC type disk.