appid metadata - unknown
-
While creating email alerts for snort, i noticed there is a great deal of 'unknown' messages in the system logs for snort. Is this a cause for concern? Is the assumption that Snort cannot identify these applications?
Same below
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2190) appid metadata "python_urllib" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2194) appid metadata "ebay_bid" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2195) appid metadata "ebay_search" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2196) appid metadata "ebay_watch" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2198) appid metadata "google_apis" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2199) appid metadata "google_app_engi" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2200) appid metadata "google_translat" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2202) appid metadata "dicks_sporting" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2206) appid metadata "sender_rewritin" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2212) appid metadata "barnes_n_noble" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2213) appid metadata "capital_one" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2214) appid metadata "barneys_ny" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2215) appid metadata "car_and_driver" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2220) appid metadata "black_decker" unknown.
Mar 28 16:47:40 snort 26727 WARNING: /usr/local/etc/snort/snort_34702_igb3/rules/snort.rules(2223) appid metadata "best_buy" unknown. -
You are likely seeing the result of non-maintained OpenAppID text rules. OpenAppID in Snort requires two different pieces of information in order to function correctly.
The first piece comes from the Snort VRT (Vulnerability Research Team) and contains the AppID stubs. These are special configuration files that contain the various application traffic types Snort OpenAppID can identify. These AppID rule stubs get updated periodically by the Snort VRT. These updates may add new applications, remove outdated ones, or simply edit the names of existing application IDs. The AppID rules stubs get updated via the automated rules update task in the Snort package.
The second, and probably more critical, piece of OpenAppID is the actual text rules that define the alert triggers. These rules reference the AppID names contained in the stub rules mentioned above, and they are responsible for actually generating an alert via OpenAppID.
The text rules are meant to be created by the firewall admin. But way back when I first put OpenAppID in the Snort package on pfSense, a user at a University in Brazil volunteered to share the results of a research project done at that school with OpenAppID. They wrote a bunch of the required AppID text rules for OpenAppID and shared them on the University website for all the pfSense users to download. However, the college website used GeoIP blocking that resulted in pfSense users in some areas of the world not being able to access the rules archive for download. The Netgate team offered to host the rules instead, and have been doing so for several years. But the Netgate team does not MAINTAIN the rules. They simply host the MD5 and zipped text rules files. Over the years, the University folks in Brazil have ceased updating the rules. So now the text rules for OpenAppID are dated. They likely are referring to AppID names in the AppID stubs that have since been either deleted by the Snort VRT or edited to have a different name. That will cause the errors you are seeing in the logs.
You can fix this issue by editing your existing OpenAppID text rules to correct the errors. Then uncheck the option on the GLOBAL SETTINGS tab to download the OpenAppID text rules. Keep the setting to download the AppID stubs, but do not check the option to download the text rules. That way you can maintain your own custom AppID rules.
-
@bmeeks said in appid metadata - unknown:
ep the setting to download the AppID stubs, but do not check the option to d
Understood. So basically, rely on the signatures provided by Snort as they update them and uncheck the custom signatures provided by the Brazil University.
Follow up questions
- Will there/should there ever be a time where that AppID Open Text Rules option is removed from the GUI. Seeing how its no longer maintained.
- Is there a way to get further context on what an application is? For example, If i block "google" but keep "gmail" what does that mean? For now im simply monitoring but to get granular in the future i would need to understand what each application contains. All my googling for this information doesnt come up with much.
-
@michmoor said in appid metadata - unknown:
@bmeeks said in appid metadata - unknown:
ep the setting to download the AppID stubs, but do not check the option to d
Understood. So basically, rely on the signatures provided by Snort as they update them and uncheck the custom signatures provided by the Brazil University.
Follow up questions
- Will there/should there ever be a time where that AppID Open Text Rules option is removed from the GUI. Seeing how its no longer maintained.
- Is there a way to get further context on what an application is? For example, If i block "google" but keep "gmail" what does that mean? For now im simply monitoring but to get granular in the future i would need to understand what each application contains. All my googling for this information doesnt come up with much.
Be careful not to confuse the AppID stubs with the AppID text rules. Not the same thing at all, but each requires the other to function as OpenAppID (the feature).
So yes, you would continue to let the Snort VRT update the AppID stubs (those contain the application names that OpenAppID currently recognizes, and also the "how to" of detecting them). The AppID text rules contain the instructions for Snort to detect the particular applications you care about (via enabling them on the CATEGORIES tab, for example). It takes both pieces for OpenAppID to work.
As for removing the older text rules from the GUI, I think not for now. Without them OpenAppID would not work at all for anyone setting it up for the first time. As it is, it does work, but just does not detect everything possible. The expectation is the admin wanting to use the feature gets more familiar with it and writes/edits the text rules themselves. So the included set of text rules is meant to be a "starter set".
If you feel adventurous and generous, you can "fix up" the problem rules and submit the changes to the Netgate team for possibly updating the AppID text rules they currently host.
-
@bmeeks I am feeling adventerous and i want to contribute. That said, just finding out where the files where located in the filesystem was a task, figured it would be /var/lib perhaps but nope. /usr/local/etc/snort is where its at.
That being said for fixing the issue per-se, i will add it to my lists. -
@michmoor said in appid metadata - unknown:
@bmeeks I am feeling adventerous and i want to contribute. That said, just finding out where the files where located in the filesystem was a task, figured it would be /var/lib perhaps but nope. /usr/local/etc/snort is where its at.
That being said for fixing the issue per-se, i will add it to my lists.The default location for Snort rules is
/usr/local/etc/snort/rules/. The rules archives are downloaded to a subdirectory created under/tmp, checked and unpacked, copied to the/usr/local/etc/snort/rules/location, and then the subdirectory created under/tmpis removed.If you are interested in fixing and "modernizing" the text rules for OpenAppID, that would be very much appreciated. I think there are actually just some misspellings in some of them. I remember finding a few a long time back.
-
@bmeeks said in appid metadata - unknown:
he text rules are meant to be created by the firewall admin. But way back when I first put OpenAppID in the Snort package on pfSense, a user at a University in Brazil volunteered to share the results of a research project done at that school with OpenAppID. They wrote a bunch of the required AppID text rules for OpenAppID and shared them on the University website for all the pfSense
@bmeeks where are the AppID stubs config stored as well? I want to clean up a few text rules over the weekend. I need to match up the stub application with the rules identification..if that makes sense.
I understand the task its just more manual labor at this point. -
@michmoor said in appid metadata - unknown:
@bmeeks said in appid metadata - unknown:
he text rules are meant to be created by the firewall admin. But way back when I first put OpenAppID in the Snort package on pfSense, a user at a University in Brazil volunteered to share the results of a research project done at that school with OpenAppID. They wrote a bunch of the required AppID text rules for OpenAppID and shared them on the University website for all the pfSense
@bmeeks where are the AppID stubs config stored as well? I want to clean up a few text rules over the weekend. I need to match up the stub application with the rules identification..if that makes sense.
I understand the task its just more manual labor at this point.Should be in
/usr/local/etc/snort/appid/. The data in this directory is what gets updated periodically when the Snort VRT makes a change.The text rules that are the admin's responsibility live in
/usr/local/etc/snort/rules/. Those rules are overwritten each time you download and unpack the OpenAppID Text Rules that Netgate hosts. -
@bmeeks
i managed to test out some new updates. for the social networking items i can get them recognized as i see the alerts show up in the alerts tab.
Disney_Plus is one that doesn't come up but I suspiciously see a lot of 'itunes' around the same time I turn on the app. Cant prove it but somehow I think its miscatorizing it.Any advice on choosing SID numbers? There doesn't seem to be a reason as to why the SIDs were chosen for each category.
social networking
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"tiktok";flow:from_client;appid:tiktok; sid:7276621; classtype:misc-activity; rev:1;)
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"yelp";flow:from_client;appid:yelp; sid:7276622; classtype:misc-activity; rev:1;)streaming_media.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"disney_plus";flow:from_client;appid:disney_plus; sid:7276624 ; classtype:misc-activity; rev:1;)
-
@michmoor said in appid metadata - unknown:
@bmeeks
i managed to test out some new updates. for the social networking items i can get them recognized as i see the alerts show up in the alerts tab.
Disney_Plus is one that doesn't come up but I suspiciously see a lot of 'itunes' around the same time I turn on the app. Cant prove it but somehow I think its miscatorizing it.Any advice on choosing SID numbers? There doesn't seem to be a reason as to why the SIDs were chosen for each category.
social networking
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"tiktok";flow:from_client;appid:tiktok; sid:7276621; classtype:misc-activity; rev:1;)
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"yelp";flow:from_client;appid:yelp; sid:7276622; classtype:misc-activity; rev:1;)streaming_media.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"disney_plus";flow:from_client;appid:disney_plus; sid:7276624 ; classtype:misc-activity; rev:1;)
The only rule with SIDs is they cannot be duplicated, other than that one, there are no rules. Looks like the original author started with something up in the 7 million range (7000000) for his SIDs.
-
@bmeeks I think i found the source of an issue. It seems that if there is the first match on any appID that is where snort stops reviewing other appID rules. This is a bit of an issue.
For example, appID xbox_live and/or xbox_live_sites. The very first match for these apps seems to be "Microsoft" which is correct but not wholly accurate.Its the same for appID disney_plus. Snort will match on appID "iTunes" for this.
I thought I was conflicting with SIDs but even picking an unusually large number such as 99991, doesn't matter as it never gets matched.
The reason Yelp or TikTok were successful entries is that there was no previous match for those in the rules I believe
-
@michmoor said in appid metadata - unknown:
@bmeeks I think i found the source of an issue. It seems that if there is the first match on any appID that is where snort stops reviewing other appID rules. This is a bit of an issue.
For example, appID xbox_live and/or xbox_live_sites. The very first match for these apps seems to be "Microsoft" which is correct but not wholly accurate.Its the same for appID disney_plus. Snort will match on appID "iTunes" for this.
I thought I was conflicting with SIDs but even picking an unusually large number such as 99991, doesn't matter as it never gets matched.
The reason Yelp or TikTok were successful entries is that there was no previous match for those in the rules I believe
Perhaps the original rules were not well optimized in the sense they were not granular enough. By that I mean identifying the app as "Microsoft" when something like "XBox" would be more granular.
-
J JonathanLee referenced this topic on
