Why app-stats.log doesn't appear in Alterts Tab?
-
Hi there
I have recently configured snort pretty much as is described in the official pfsense documentation. At the moment I want to use snort mainly for layer 7 application detection.
For whatever reason, i can see the detected applications on the "Snort Interfaces" > "WAN Logs" > app-stats.log but there's only https alerts showing (class misc activity).
I also have checked the variable $HOME_NET which should be ok.
Any suggestions how to get the app detection into the alters? Otherwise you can't do any filter actions, right?
Many thanks!
-
Do you have the corresponding OpenAppID text rules enabled for download AND enabled for alerting on the CATEGORIES tab?
Double-check that $HOME_NET is correct. This is especially true if you have anything besides a default network with WAN and LAN only. Most instances with some rules failing to trigger can be traced to $HOME_NET or $EXTERNAL_NET issues.
-
I did the following, and still no luck:
-
Installed the OpenApp ID Detector Rules and download
-
Installed AppID Text Rules and download
-
Activated all appid rules
-
Enabled AppID detection and AppStats Logging
-
Manually created an alias "alllocalnetworks" including all local subnets
-
Created a pass list and assigned the alias "alllocalnetwork" while uncheck all ohter options und "Auto generated IP Addresses"
-
In the interface options "WAN Settings" > Home Net: i put the pass list created before. When hitting the Home Net > View List button, only the local networks are shown
-
On the "View List" for external, the list ist just inverted using ! operator
No clue what else i cloud do... I'm happy for every hint
-
-
You will not see the data on the ALERTS tab that you see in the
app-stats.log. That's not the way the process works. You will only see data on the ALERTS tab from rules that fire and actually generate alerts.The data in the
app-stats.logcomes from a different process. All kinds of traffic is logged there, not just OpenAppID stuff. -
Hm.. but in the ALERTS i don't have anything from AppID as you can see in the screenshot. Shouldn't the recognized apps not fire alerts as well?
app-stats.log shows:
statTime="1601402700",appName="DNS",txBytes="11016",rxBytes="0"
statTime="1601402700",appName="iTunes",txBytes="0",rxBytes="7597"
statTime="1601402700",appName="Netflix",txBytes="0",rxBytes="5299"
statTime="1601402700",appName="HTTPS",txBytes="0",rxBytes="340515"
statTime="1601402700",appName="SSL client",txBytes="0",rxBytes="237250"
statTime="1601402700",appName="Microsoft",txBytes="0",rxBytes="7666"
statTime="1601402700",appName="Exchange Online",txBytes="0",rxBytes="185796"
statTime="1601402700",appName="Office 365",txBytes="0",rxBytes="7108"
statTime="1601402700",appName="Azure cloud portal",txBytes="0",rxBytes="23784"
statTime="1601402700",appName="__unknown",txBytes="3255",rxBytes="110926"Alerts window:
-
You will only get alerts from OpenAppID text rules that are enabled (not commented out, for example, in the rules file) and that match traffic.
I don't use the OpenAppID rules on my personal firewall, so I don't have a handy method at the moment to research the various rules files. The
app-stats.logis populated from the action of the Snort stub-detectors, it is not populated by the OpenAppID text rules. It takes both pieces to get alerts. The stub-detector must be in place to actually sense the traffic, and a corresponding text rule must be present and enabled in order to trigger an alert on the ALERTS tab. Perhaps you are missing that latter part ???As I said, I don't use OpenAppID, but others here do and so far as I know it is working for them.
-
I decided to test this on my personal firewall, and everything worked fine. After enabling the OpenAppID rules, preprocessor, and configuring some rules, I immediately started getting alerts from my wife's iPhone while she was using Facebook. See below --
Here are all the steps you need to do:
-
On the GLOBAL SETTINGS tab, enable the download of both the OpenAppID stub detectors and the OpenAppID text rules themselves. That means checking two checkboxes on that tab. Click Save.
-
Go to the UPDATES tab and click Update to pull down the OpenAppID rule files and install them on the firewall.
-
Go to the PREPROCESSORS tab for the interface you want to use OpenAppID on and click the checkbox to enable the Application ID preprocessor. Scroll to the bottom and click Save.
-
Go to the CATEGORIES tab and enable the OpenAppID rules categories you want to use. Click Save. (Note that a previous screenshot of yours seems to show you've done this step).
-
Go to the INTERFACES tab and restart Snort on the interface you just edited.
You should now get OpenAppID alerts when matching traffic is detected. I immediately disabled the Social Networking rules on my system, though, as I was getting constant alerts. Since Social Networking is normal and widespread in home networks, the utility of OpenAppID there is questionable. It is really more suited to a corporate or other business environment where you want to monitor compliance with employee workplace policies around acceptable use of the company's network and/or productive use of employee working hours.
-
-
Thanks @bmeeks much appriciated! I reconfigerd snort on the LAN interface, and now i get some alerts. I noticed now, that it only catches when destination port is 80. I have not one alert wit dst port 443...
-
@Liceo said in Why app-stats.log doesn't appear in Alterts Tab?:
Thanks @bmeeks much appriciated! I reconfigerd snort on the LAN interface, and now i get some alerts. I noticed now, that it only catches when destination port is 80. I have not one alert wit dst port 443...
What are you expecting to show up on 443 that is only showing on 80? Notice that all of the Facebook traffic captured by my firewall's OpenAppID rules was on port 443. Different services can use different destination ports. If you look at the alerts I posted, the Facebook application on my wife's iPhone was basically in a constant conversation with the Facebook server farm. Her phone is sourcing traffic on a random port (that's normal) and directing it to a Facebook IP at the destination port of 443 (the SSL port).
-
Yes, i'm absolutely aware of this. Most applications use SSL as the destination port. I know i have several application in the network which should be appear in the alerts like outlook, netflix, instagram, office365 like in your example, but i only see a couple of dest port 80 apps like "chrome".
-
Well, whether or not they appear in the alerts depends on exactly what the rule is looking for (in other words, what triggers it). Remember the text rules that generate the alerts were created by volunteer non-paid staff at a University in Brazil and contributed to pfSense users by them. I don't think the actual text rules content has been updated for quite some time, but the Snort OpenAppID stub detectors have changed over time. So maybe the rules no longer work correctly for some applications.
Also remember that the rules only have a very few things they can actually see when it comes to an SSL session. Basically they see the initial header exchanges before the SSL key is negotiated, and they use that to decide whether or not to "trigger". Once the secured SSL session begins with encrypted packet payloads, the IDS/IPS sees nothing but random data due to the encryption. This is true for everything analyzed by Snort or Suricata when it comes to encrypted traffic. The only way the IDS/IPS can do real DPI (deep packet inspection) is when you configure SSL termination (MITM) on the firewall.
-
I understand that some rules may be outdated and layer 7 app detection is probably not 100% but i don't think that this is the problem. At least some well known applications (like facebook in your screenshot) should be alerted => but NOTHING on my box. When i filter the alerts for destination port other than 80 (!80) i have no alert at all. You screenshot shows, that your pfsense alerting on facebook (DPort 443), but it does not appear on my pfsense (I did also test with facebook).
That's why i had the assumption, that snort don't inspect other communication than http but i don't know why...
-
Snort will not limit its inspection to just HTTP unless that were the only types of rules you had enabled.
My inclination at this point is your $HOME_NET and/or $EXTERNAL_NET variables are not correct. You mentioned in a much earlier post about modifying them by assigning a custom list (you said you created a Pass List and used that). In the vast majority of installations there is never a need to modify the $HOME_NET and $EXTERNAL_NET variables from their default values. I would go back to the INTERFACE SETTINGS tab and reset both of those drop-downs to "default". Then click Save and return to the INTERFACES tab and restart Snort on that interface.
I also see from looking back at one of your screenshots that you appear to be running on a Hyper-V installation (I see the hn0 NIC) listed. Are you 100% positive that your virtual switch is correctly configured such that pfSense is actually seeing the traffic you want to get alerts from? You blacked out all the IP addresses in your screenshot, so I can't tell if there are RFC1918 addresses in use or not. If they are RFC1918 addresses, then there is really no point in blacking them out. It would be really useful if you would post up a screenshot of the contents of your $HOME_NET and $EXTERNAL_NET variables along with another screenshot of those alerts with the IP addresses visible (assuming they are all either RFC1918 or external Internet addresses).
-
Yes, pfsense is running on Hyper-V 2019. The NIC config is pretty default. Just one interface is configured as a VLAN trunk.
$HOME_NET is at follows:
10.242.5.0/24 62.2.17.60 62.2.17.61 62.2.24.158 62.2.24.162 77.59.160.1 77.59.160.79 127.0.0.1 172.16.10.0/24 172.16.10.7 172.16.10.8 172.16.22.0/24 172.16.98.0/24 ::1 fd45:c46d:6ae2:a::1 fe80::215:5dff:fe0a:f00 fe80::215:5dff:fe0a:f01 fe80::215:5dff:fe0a:f02 -
VLANs can be problematic for IDS/IPS setups. You will need to be sure all NIC offloading is disabled. Since you are running in a Hyper-V setup, I would think perhaps that is already done, but you want to make sure. There are settings within pfSense and then likely settings in Hyper-V or the Hyper-V host someplace. I don't use Hyper-V, so can't help there. I do use ESXi, but I do not run my firewall as a VM.
Are you just running Snort on the parent interface?
I guess I'm still not clear on what you are expecting to see that you are not seeing. So have you tried to duplicate my Facebook results? Enable the Social Media rules and then open Facebook on a smartphone that is running on one of the networks listed above in your HOME_NET. So for example, make sure the phone is not using Cellular Data or otherwise bypassing the networks listed in HOME_NET.
-
Regarding Hyper-V adapter settings: I have to investigate in this, will take a bit more time for testing the various settings.
"Are you just running Snort on the parent interface?" > I'm running snort neither on an interface configured as VLAN trunk nor the interface has a VLAN configured on the VM configuration.
I tested exactly the way you have described (devices are located in local WLAN/LAN of course) and additionally i have tested with Netflix. I also expect other services like FaceTime, Youtube, ect..), because i know there is traffic towards those services.
-
@Liceo said in Why app-stats.log doesn't appear in Alterts Tab?:
I'm running snort neither on an interface configured as VLAN trunk nor the interface has a VLAN configured on the VM configuration.
So you are running Snort on the WAN? Does all of the traffic you are expecting to see actually traverse that interface? I would assume so, but since it is a VM and has a virtual switch, there are ways for things to bypass. Also remember that Snort sits outside of the firewall directly in front of the NIC. So on your WAN, it will see all LAN traffic post-NAT rules applied (meaning it will not see any individual LAN hosts, but only the WAN's public IP). The recommended place to run the IDS/IPS is on the LAN so that it can see inbound traffic on the interface before NAT is applied, and outbound traffic from the interface after any NAT is undone.
I run my main Snort instances on my LAN and DMZ interfaces. I do run an instance on my WAN with just the ET-CIARMY rules, but that instance is solely to generate alerts that I use when testing new GUI package updates.
-
No i'm running Snort on LAN. I have 3 interfaces: WAN, LAN and LAN/VLAN-Trunk.
BTW, this are the system logs from Status > System Logs > System > General when restarting the Snort enabled LAN interface, maybe it helps debugging...
Oct 2 08:16:02 kernel hn1: promiscuous mode enabled Oct 2 08:16:01 snort 44143 AppInfo: AppId 2314 is UNKNOWN Oct 2 08:16:01 snort 44143 Invalid direct service AppId, 2314, for 0x80b7b4f90 0x804d9a440 Oct 2 08:16:01 snort 44143 AppInfo: AppId 2312 is UNKNOWN Oct 2 08:16:01 snort 44143 Invalid direct service AppId, 2312, for 0x80b7b4f90 0x804d9a440 Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 php [Snort] Snort START for LAN10(hn1)... Oct 2 08:16:01 php [Snort] Building new sid-msg.map file for LAN10... Oct 2 08:16:01 php [Snort] Enabling any flowbit-required rules for: LAN10... Oct 2 08:16:01 php [Snort] Enabling any flowbit-required rules for: LAN10... Oct 2 08:16:00 php [Snort] Updating rules configuration for: LAN10 ... Oct 2 08:15:59 kernel hn1: promiscuous mode disabled Oct 2 08:15:59 snort 74522 *** Caught Term-Signal Oct 2 08:15:58 php-fpm [Snort] Snort STOP for LAN10(hn1)... Oct 2 08:15:58 php-fpm Restarting Snort on LAN10(hn1) per user request... -
@Liceo said in Why app-stats.log doesn't appear in Alterts Tab?:
No i'm running Snort on LAN. I have 3 interfaces: WAN, LAN and LAN/VLAN-Trunk.
BTW, this are the system logs from Status > System Logs > System > General when restarting the Snort enabled LAN interface, maybe it helps debugging...
Oct 2 08:16:02 kernel hn1: promiscuous mode enabled Oct 2 08:16:01 snort 44143 AppInfo: AppId 2314 is UNKNOWN Oct 2 08:16:01 snort 44143 Invalid direct service AppId, 2314, for 0x80b7b4f90 0x804d9a440 Oct 2 08:16:01 snort 44143 AppInfo: AppId 2312 is UNKNOWN Oct 2 08:16:01 snort 44143 Invalid direct service AppId, 2312, for 0x80b7b4f90 0x804d9a440 Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 snort 44143 AppId Oct 2 08:16:01 php [Snort] Snort START for LAN10(hn1)... Oct 2 08:16:01 php [Snort] Building new sid-msg.map file for LAN10... Oct 2 08:16:01 php [Snort] Enabling any flowbit-required rules for: LAN10... Oct 2 08:16:01 php [Snort] Enabling any flowbit-required rules for: LAN10... Oct 2 08:16:00 php [Snort] Updating rules configuration for: LAN10 ... Oct 2 08:15:59 kernel hn1: promiscuous mode disabled Oct 2 08:15:59 snort 74522 *** Caught Term-Signal Oct 2 08:15:58 php-fpm [Snort] Snort STOP for LAN10(hn1)... Oct 2 08:15:58 php-fpm Restarting Snort on LAN10(hn1) per user request...Those two "Invalid" lines are from typos in the OpenAppID text rules. The authors did not complete the spelling of "direct-tv" (or something similar to that phrase, don't remember now but I once researched it). Those are present on my system as well and have no bearing on OpenAppID working.
I'm still confused. You say you have three interfaces but run Snort on the LAN. Yet your earlier screenshots show "WAN" in the interface name. So do you mean you have Snort running on all three interfaces, two of the interfaces, or just one of them?
Also, if you have (or had in the past) multiple interfaces configured with Snort, you will have multiple sub-directories in
/var/log/snort. Each Snort instance has its own private logging system in one of those sub-directories. So now I'm not clear from which interface theapp-stats.logyou posted a snippet of actually came from.I can tell you the OpenAppID subsystem works for me and many others. You appear to have a configuration problem on your end.
-
Sorry for the confusion, i had configured Snort on WAN first, but now it's only on the LAN Interface with same results. If have no doubts, that snort is running on many installations, i just want to make it work on my box ;-).
I think i will deploy a clean pfsense install and see if it's working. I will let you know.
But anyway, thanks for your help so far!
