You have at least three different packages installed on your firewall that all can, to varying degrees, result in blocked network traffic (SquidGuard, Snort, and pfBlockerNG). Do you know how to check which of these packages are producing alerts/blocks and what IP addresses they might be impacting? If so, then do the research in your alert logs for the installed packages and find out which package (or packages) is doing the blocking and which IP is being blocked. You very likely are the victim of false positives. Once you trace the true source of the block, you can make a determination if it is a false positive (or not) and proceed accordingly.

@steveits Yeah but its running in a loaded datacenter with a shitload of terminalservers accessing all kinds of sites. You cant keep up and even if you witelist it, it blocks.

@enesas said in Snort Application and site blocking Problem!: @bmeeks I reviewed what you said. However, I couldn't quite figure it out. For example, is there an example list of blocked drops for Youtube or another site? Also, when you add this list to SID MGMT, will it work directly, is a different setting required? When the communication problem due to language is added, it becomes a little difficult to understand. I'm sorry for making you tired. An IPS Policy is a pre-defined collection of rules designed to provide a given base level of security. There are four defined policies, but only three of them are useful in a production setup. The fourth policy (max-detect) is an extreme security policy designed primarily for testing only. It will block all manner of likely desired traffic. Here are the three IPS Policies (ordered by increasing protection): IPS Policy Connectivity IPS Policy Balanced IPS Policy Security These are created by the Snort rule authors (also known as the Snort Vulnerability Research Team, or VRT). The policies exist via metadata tags included within each Snort text rule (but excluding OpenAppID; those rules are NOT part of any IPS Policy directly). When creating rules, the Snort VRT will tag each rule with one or more IPS Policy tags. That allows an automated rule selection algorithm to pick the rules tagged with a chosen policy tag. For IDS beginners, it is best to start with the IPS Policy Connectivity policy as that one provides reasonable protection from common threats without generating too many false positives. I strongly recommend you never go higher than IPS Policy Balanced unless you are protecting military secrets or something. The "Security" policy will block a ton of desirable stuff -- meaning lots of "normal" network traffic will get blocked and cause you headaches and frustration. However, if new to an IDS/IPS, you should start with NO blocking enabled. Choose rules but do NOT enable blocking at first. You need to let your choice of rules run in your network environment for several days or even weeks. Check the ALERTS tab often in Snort to see what alerts have triggered. Research them and determine if they might actually be false positives in your network. That is highly likely these days due to the way modern web sites work to serve adds and due to the encryption of lots of other traffic. For false-positive triggering rules, you should probably disable them. Now lets talk about OpenAppID. For that to work you must have two different kinds of rules downloaded (or else custom created by you, the admin). One requirement is the OpenAppID rule stubs that come from the Snort VRT. That set of stubs defines the applications that Snort OpenAppID can detect and gives Snort the internal "how-to" instructions for detection. The other required piece of OpenAppID is a set of text rules that leverage those detector stubs to actually scan traffic and produce alerts. This latter set of text rules tells Snort which applications you want to look for. You must either write these text rules yourself, or you can take advantage of a starter-set of OpenAppID text rules created by a group at a University in Brazil. You can enable the download of these starter rules on the GLOBAL SETTINGS tab of Snort where you enable the download of the OpenAppID stub rules. Then on the CATEGORIES tab you can enable one or more categories of OpenAppID rules. But be aware these categories were created by a team of volunteers (University students, actually), so they may not be complete. Additionally, they have not been updated in several years. Thus many more modern applications are missing detection rules and thus won't be detected by Snort using these OpenAppID starter rules. So for any missing applications in the starter rules package, you would need to create your own Custom Rules containing the necessary syntax. You can find out more about writing OpenAppID rules here: https://blog.snort.org/2014/04/openappid-application-rules.html You can search on Google for other OpenAppID tutorials. Be aware that most of the recent effort with OpenAppID has been targeted for Snort3 and not legacy Snort as used on pfSense. The pfSense package is based on Snort 2.9.x and is NOT compatible with Snort3 rules! Do not attempt to use any Snort3 rules on pfSense. Doing so will totally break the Snort package on pfSense.

@spacey said in Suricata V3.0 Inline Mode: @bmeeks Get go* the sample.conf files shouldn't have #comments and should just be enabled The comments are there to be instructive so the admin can see how to customize their rules. The whole purpose of that tab is to allow customization of rules depending upon the network environment. I am still not understanding what you are asking for. With an IDS/IPS, there is no one-size-fits-all setup. That's why a lot of experience and knowledge about threats and exposures is required in order to be a qualified IDS/IPS administrator. You pick and choose the rules using knowledge of the specific exposures/vulnerabilities present in your individual network.

You may be hitting this bug I found posted over on the upstream Suricata Redmine site: https://redmine.openinfosecfoundation.org/issues/5247. Suricata is a multithreaded application, and thus has some special logic for handling flows in a multithreaded environment. Sometimes that special logic fails, though, at assigning a flow to the correct thread. So if the logic gets confused and assigns part of the flow conversation to one thread, but the other part of the flow conversation to another thread, you could then see this error. But remember those applayer rules are just informational. One triggering does not automatically mean "malware" is present. They are finding and alerting on abnormalities in traffic flow. However, the rules can misfire or may even be buggy sometimes.

@steveits Thank you so much for you advice!! I tracked the location of that IP and just emailed the company, explaining what's happening. The most intriguing part is that the IP is from the agency that registers and maintains all".br" websites here in Brazil, maybe they were victims of some kind of attack.

@dobby_ Yes, it appears, if they can get it out of user space...

Actually just registered to reply to this and say thank you. While some people think its pointless to preserve the snort2c table, i find it highly useful.

@michmoor said in Suricata and having a Oinkmaster code: @bmeeks so I’m trying to understand the logic as it’s shown In the gui. If I do not select no Snort rule categories ,does snort VRT rules run under the hood? It seems that way implicitly. I don't fully understand your question here. If you do not choose any Snort category files under CATEGORIES, and you don't enable an IPS Policy, then no Snort VRT rules will be used at all (ignoring OpenAppID here as those rules are totally different). The rules may still be downloaded and updated if you have the download option enabled, but they will not be selected for use on any interface unless the appropriate category is selected on that interface. Note that SID MGMT, if used, can also automatically enable categories including Snort VRT rules. But if you enable an IPS Policy, that will automatically choose a collection of Snort VRT rules and enable them. Remember I posted somewhere up above that the Snort VRT rules (and ONLY the Snort VRT rules) contain a special metadata tag that associates a rule with one or more pre-defined IPS policies. Also, for each assigned policy, the metadata contains a suggested action (ALERT or DROP) for each rule. So when you check the box in the GUI to enable an IPS Policy, the PHP code will automatically scan the Snort VRT rules and pull in all the rules tagged with the IPS policy metadata matching the policy you chose on the CATEGORIES tab. You can easily see these metadata tags by opening some of the Snort VRT rules and searching for the string "policy {policy_name}-ips", {policy_name} is replaced by "connectivity, balanced, etc.". When IPS Policy is enabled, manual selection of Snort VRT categories on the CATEGORIES tab is disabled. The logic there is the user is handing over the Snort VRT rule selections to the enabled IPS policy logic. However, if desired, you can still use the SID MGMT features to add additional Snort VRT rules.

@bmeeks ok so using two IDS in none blocking mode maybe work but not as an IPS. That makes sense. Was curious if snort just for OpenAppID use case could still be used but doesn’t seem like it. Ok fair enough

@michmoor said in appid metadata - unknown: @bmeeks I think i found the source of an issue. It seems that if there is the first match on any appID that is where snort stops reviewing other appID rules. This is a bit of an issue. For example, appID xbox_live and/or xbox_live_sites. The very first match for these apps seems to be "Microsoft" which is correct but not wholly accurate. Its the same for appID disney_plus. Snort will match on appID "iTunes" for this. I thought I was conflicting with SIDs but even picking an unusually large number such as 99991, doesn't matter as it never gets matched. The reason Yelp or TikTok were successful entries is that there was no previous match for those in the rules I believe Perhaps the original rules were not well optimized in the sense they were not granular enough. By that I mean identifying the app as "Microsoft" when something like "XBox" would be more granular.

@bmeeks I don't have a proxy set up yet because there's a lot about its configuration that i'm unsure about. I run suricata in inline mode on the lan interface. I also run unbound with full DoT to cloudflare 1.1.1.2 for dns. i have nat rules and blocks set up so that, unless a client is running a vpn from there device, dns is handled by unbound. i'm not accessing anything yet from the outside, so.. (still learning this stuff.. babysteps) My goal is to be able to perform a full-on mitm setup, however i'm still confused with the whole cert thing, wpad, transparent proxy thing.. (the more i read up, the more questions i have). i thought it was supposed to work as follows: there are basically 2 certs (for simplicity sake i'll refer to the certs as follows): -A WAN-side cert, so pfsense can de-encrypt/re-encrypt traffic as it receives/sends client data with the outside world, and -A LAN-side cert so that it could de-encrypt/re-encrypt traffic as it receives/sends data with the clients on the lan-side for https traffic. that way, the moment my data hits pfsense on the lan-side, it is decrypted.. pfsense in all it's glory can process all packets to its fullest capability, and then before sending the data out to the internet, re-encrypts it.. and reverse (obviously) when it receives data. Normally, i'd leave it as is, however as i fill my head with all this 'stuff', i realize that these packages really aren't doing much any more because of the encryption. and since it's just me and my family on my home network, i'd like to be able to use these facets.. also, this stuff is kinda fun! frustrating at times.. but fun.. Also i realize that email is the means of most attack vectors, and it's all basically double-encrypted, such that even with a full-on mitm proxy, email and certain apps can't be de-encrypted, and im fine with that. however regular web traffic i'd like to have scanned and processed. yeah, it's a lot, and those who know would say it's complete overkill and not serving a purpose, however it's all learning to me.. experience that i can take with me where ever i go..

@gertjan It is. Topping out at around 1gbit/s. Pipe is 10gbit/s. Without Suricata running the FW fares well and load sit below 25% on CPU and 4% RAM. When SC is running then it dies instantly. Both legacy and inline mode.

@daddygo hi, sorry, I come back to this topic again. Actually, i am also using expressvpn too. What i don't understand. vpn is not a wan interface outsiders on the same vpn access point, can probe the pfsense no? especially as bogon/ private networks are not blocked. don't we want to have visibility to attacks from th expressvpn network? Isn't that what Suricata helps out with?

@michmoor said in OpenAppID - what is the application?: @bmeeks Thanks Bill. Now you are just showing off your search skills :) Thanks for this. Im understanding the structure here of how the app writing takes place. Not to difficult to piece together. You can only detect what you can see. I am not looking forward to TLS1.4 and more from a AppID perspective. ha ha. Yeah, not too complicated once you dig into it a little. What I call the AppID stubs that download regularly from the Snort VRT contain the metadata and detection pieces for identifying specific apps. The AppID text rules then reference that metadata and application names in the AppID stubs to generate alerts when specific app traffic comes through. It takes both to make the whole. And the text rules are usually the responsibility of the firewall admin, but that starter list included in the Snort package helps you get a basic setup working out of the box. But as mentioned up above, that starter package is a bit dated now as the maintainer is no longer updating it.

@bmeeks and bookmarked

@bmeeks Just chiming in to note this has been implemented via feature request #10809, working just fine. Received one alert via Telegram on Dec 11 last year, all good. Suricata new rule categories are available: - Emerging Threats Open rules: threatview_CS_c2.rules

@bmeeks This looks quite cool... I'll try some things out tomorrow during office hours - so I can have a look on the results more or less in livetime. Keep you informed

@bmeeks Thanks a lot for the detailed answer. I will go ahead with Suricata IPS in pfSense with Legacy-mode blocking on, then send those alerts to system log and then send it to the Security Onion over syslog. Security Onion can parse pfSense logs out of the box and then have custom Kibana Dashboards.