Alerts received on incorrect interface
-
I'm currently running a 6100 appliance. Earlier today, I reconfigured my 3 networks to all come into my appliance via 10Gb DAQ on ix1 (ix1, ix1.10, ix1.20|Secure,IoT,MoB respectively). Secure is untagged, IoT is VLAN 10 and MoB is VLAN 20. All of my devices on the networks appear to have pulled the correct IP Addresses from my pfSense DHCP scopes. I noticed however, that I started seeing some random Snort alerts for benign events showing up under the wrong interface. Events that were sourced from an IP address on the MoB network were showing under the Secure interface.
In some cases, it showed under both interfaces. I removed Snort and installed Suricata to see if I saw the same results, and while they were not as frequent, I did still see some alerts that were under the wrong interface.
l completely uninstalled Snort, including the settings and reinstalled but still see the same issue. If I restore the network configuration with each network/vlan on its own discrete interface, the issue does not happen.The following questions came to mind: Is inspecting multiple networks trunked into a single interface an issue? Is the untagged network on the interface possibly an issue?
Thanks for any suggestions you guys may have.
-
Snort automatically puts the monitored interfaces in promiscuous mode. When the interface is a VLAN, that means Snort sees the traffic from all VLANs defined on the same parent interface. So in your case
ix1
is the parent, and any VLANs defined there will be seen by Snort.There is an option on the INTERFACE SETTINGS tab to toggle promiscuous mode off,but I'm not sure how well that will actually work in the case of VLANs because within FreeBSD a VLAN is created as a virtual interface on top of a physical parent interface.Edit: scratch my earlier suggestion about the promiscuous mode switch. Only Suricata has that one.
Unless you need radically different rules on the various VLANs, I usually recommend folks run the IDS/IPS on the physical parent. That produces a single instance of the daemon and thus conserves CPU cycles and RAM. And with promiscuous mode, the instance on the parent interface sees all the VLAN traffic anyway.
If you must run substantially different rules on each VLAN, then you really have no option but to put each on a separate physical interface.
-
@bmeeks Thank you for the quick response. I will remove the VLAN interfaces and leave just the physical ix1 interface.
Thanks again!
-
@bwhaley said in Alerts received on incorrect interface:
@bmeeks Thank you for the quick response. I will remove the VLAN interfaces and leave just the physical ix1 interface.
Thanks again!
You're welcome! Hope all is well with you.
-
-
-
I am wondering if you could guide me. With some step by step instructions with it then on Suricata.
Since suricata should only run on the parent interface only and not create anything for the vlans that run on the same parent interface.
-
@sstatjm said in Alerts received on incorrect interface:
I am wondering if you could guide me. With some step by step instructions with it then on Suricata.
Since suricata should only run on the parent interface only and not create anything for the vlans that run on the same parent interface.
A typical VLAN setup might look like this assuming you have the following conditions:
Physical interface =
em1
(this might be igb, or ix, or something else - it is determined by your particular NIC hardware).
VLAN100
= LAN VLAN
VLAN200
= IoT VLAN
VLAN300
= Guest WirelessWithin pfSense, if you had the configuration above, you would see a total of four interfaces available:
em1
,em1.100
,em1.200
, andem1.300
. In the Suricata GUI they would also show with any assigned friendly interface names/descriptions.So, when we say run Suricata on the parent interface, that would be
em1
in this example. Suricata would then see all the traffic that runs across the physicalem1
interface including each VLAN's traffic. -
So if mine looks like this
Should i delete the last 4 and only leave the LAN only? -
@sstatjm:
Yes, so long as you are running the exact same rule sets on each VLAN, then you can conserve resources by running a single instance on the parent interface.In most cases the single instance is fine.
-
Should I turn promiscuous mode back on also?
-
@sstatjm said in Alerts received on incorrect interface:
Should I turn promiscuous mode back on also?
Yes, you can.
-
ok great I will give that a try. I am also wondering if running it in inline mode would be a problem. Before I ran it on all interfaces and everything crashed. And I had to rebuild from scratch
-
@sstatjm said in Alerts received on incorrect interface:
ok great I will give that a try. I am also wondering if running it in inline mode would be a problem. Before I ran it on all interfaces and everything crashed. And I had to rebuild from scratch
I do not recommend using Inline IPS Mode with VLANs. The netmap kernel device used for Inline Mode and VLANs do not play well together. Some work is being done in the FreeBSD kernel, but it is not ready yet.
-
this is what suricata looks like now
I deleted all the VLANs i had there before
-
@bmeeks
I have multiple VLANs running on a LAGG.
How do i get suricata running on the parent interface if the parent interfaec is technically a LAG?
I cant assign the interfaec per documentation. -
@michmoor said in Alerts received on incorrect interface:
@bmeeks
I have multiple VLANs running on a LAGG.
How do i get suricata running on the parent interface if the parent interfaec is technically a LAG?
I cant assign the interfaec per documentation.Suricata is not really setup to work with LAGG interfaces. It might sort of work, but things can be weird. Interfaces such as LAGG, GIF, and even VLANs are actually virtual interfaces down at the kernel level. As such, the shared libraries such as
libpcap
used for IDS and Legacy Mode blocking, and the netmap kernel device used for Inline IPS Mode, don't work well with the virtual interfaces. That's because they are designed to work with hardware NIC drivers directly.Some work is underway by others to make netmap and LAGG and Bridge interfaces behave better together. But none of that is in pfSense yet (nor anywhere else).
Having these IDS/IPS packages coded and working for complex setups is asking a bit much for volunteer maintainers and free software. To make that work well takes customized software both at the OS level and in userland. That's why the big boys charge so much for their products that do this.
-
@bmeeks You really should give write-ups on the inner workings of pfSense and the packages. You give really good details on why something works the way it does.
That said, the work around will be to run an instance of Suricata on each VLAN.
Appreciate the response. Appreciate the work involved. -
I think I also see this on a Netgate 7100.
WAN ... lagg0.4090
LAN ... lagg0.4091I have snort on LAN and have the strange behavior that I see blocked IPs, but zero alerts on that interface!
So no way to suppress things right now. and somehow WAN traffic snorted when I only want to process LAN traffic.
hints welcome ;-)
edit: I also see blocking when blocking is disabled on all snort interfaces.
This is on an updated pfSense-23.01, snort installed and configured only 2 days ago (from scratch).edit 2: removed snort and installed suricata. So far this seems to work much better. Although I haven't yet seen a single alert and/or block (alerting only configured now). But no blocks without alerts so far.
-
@sgw said in Alerts received on incorrect interface:
I think I also see this on a Netgate 7100.
WAN ... lagg0.4090
LAN ... lagg0.4091I have snort on LAN and have the strange behavior that I see blocked IPs, but zero alerts on that interface!
So no way to suppress things right now. and somehow WAN traffic snorted when I only want to process LAN traffic.
hints welcome ;-)
edit: I also see blocking when blocking is disabled on all snort interfaces.
This is on an updated pfSense-23.01, snort installed and configured only 2 days ago (from scratch).edit 2: removed snort and installed suricata. So far this seems to work much better. Although I haven't yet seen a single alert and/or block (alerting only configured now). But no blocks without alerts so far.
If you saw blocks with no corresponding alert, then it is highly likely you had another uncontrolled (or zombie) Snort process running on the interface. This can happen sometimes. You wind up with duplicate Snort instances on the same interface, but only one of those instances responds to- and provides updates to- the GUI.
LAGG interfaces are really not well supported. Ditto for Bridge interfaces. The IDS/IPS packages expect plain vanilla interfaces. They also do not always behave as expected with VLANs. All of these: VLANs, LAGGs, and Bridges are actually virtual interfaces created by the FreeBSD operating system. The IDS/IPS code expects "real, physical" interfaces. It can perhaps work with a virtual interface, but results will be spotty.
Some work is underway by others to patch the netmap kernel device so that it better supports LAGG, Bridge, and VLAN interfaces.
-
@bmeeks thanks for the explanation.
Looking at that system it seems to work OK. For sure I don't know what maybe is missing from the shown alerts etcI am in the process of trying to learn the basics and decide if and where to use IDS/IPS. My customers mostly don't have any internal IT staff, so who maintains it, who even looks at the alerts etc?
So I am running it in alerting-only and try to see what happens in their subnets. And I learn things :-) thanks
-
@sgw said in Alerts received on incorrect interface:
My customers mostly don't have any internal IT staff, so who maintains it, who even looks at the alerts etc?
In that circumstance, I would be extremely reluctant to deploy an IDS/IPS in blocking mode. Any IDS/IPS is subject to generate false positives, and if no qualified admin is there to immediately assess the alert and disable or otherwise remediate the resulting block, the users will be very frustrated.
And there is a warning for simple alerting mode as well. If the firewall is a device without a spinning hard disk (or a very good quality server-grade SSD), the amount of logged data from an IDS/IPS can result in quickly exhausting the write cycles of the disk. Prime example would be a smallish eMMC type disk.
-
@bmeeks said in Alerts received on incorrect interface:
In that circumstance, I would be extremely reluctant to deploy an IDS/IPS in blocking mode. Any IDS/IPS is subject to generate false positives, and if no qualified admin is there to immediately assess the alert and disable or otherwise remediate the resulting block, the users will be very frustrated.
I agree and see that danger.
And there is a warning for simple alerting mode as well. If the firewall is a device without a spinning hard disk (or a very good quality server-grade SSD), the amount of logged data from an IDS/IPS can result in quickly exhausting the write cycles of the disk. Prime example would be a smallish eMMC type disk.
Had that today ;-) suricata filled ~4 GB of storage since yesterday.
Although the information text says that per default it should only use 25(?)% of the free storage it filled up the root filesystem completely.I set a limit of 1 GB now for a start, but sure, that won't give much history to browse through etc. / It's a test anyway, we'll see if we keep it at that site. At least in that place there's a admin on site most of the time.
-
@sgw said in Alerts received on incorrect interface:
I set a limit of 1 GB now for a start, but sure, that won't give much history to browse through etc.
The recommended approach is to export logging data off to a dedicated third-party SIEM platform. There are several open-source varieties to choose from, especially for Suricata when using its EVE JSON logging options.
-
I see.
The set limit didn't work, by the way. Full rootfs again today.
I reduced the limit again and edited some time spans down. Plus: disabled one interface in suricata, looking at the DMZ only now. -
@sgw said in Alerts received on incorrect interface:
I see.
The set limit didn't work, by the way. Full rootfs again today.
I reduced the limit again and edited some time spans down. Plus: disabled one interface in suricata, looking at the DMZ only now.Forcing the Suricata binary to rotate logs is very difficult. There is no built-in ability within the binary's code for rotating logs based on their size. The only log rotation option available in the binary is a time interval one where the log file is rotated on a fixed interval regardless of the file's size. And even that limited option is not available for every type of log file.
The GUI package for Suricata attempts to force size-based log rotation by checking log sizes using a crontab task that runs every 5 minutes. That job checks the current log file and directory sizes against user-set limits. For files that meet the rotation critiera, it rotates them by changing the file name, "touches" a new empty file with the original name, and then sends a SIGHUP signal to the binary. The SIGHUP is supposed to cause the binary to restart logging, but that is not always honored by the binary in my experience. When the SIGHUP is not honored, the binary will continue to write to the old log file and thus can fill up the disk.
I really wish the upstream Suricata team would add a size limit rotation feature to all of the log files.
-
@bmeeks thanks for the explanation. Since yesterday the limit seems to work, just checked.