Snort fatal error after emerging.rules update
-
@mzawolski for me no love
-
I was able to get it to load by disabling the following ET Pro rules released last night.
Edit: bmeeks post about disabling rules in bulk, https://forum.netgate.com/topic/81700/mass-disable-snort-rules
2046273 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M1 (malware.rules)
2046274 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M2 (malware.rules)
2046275 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M3 (malware.rules)
2046276 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M4 (malware.rules)
2046277 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M5 (malware.rules)
2046278 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M6 (malware.rules)
2046279 - ET MALWARE [Mandiant] UNC4841 SEASPY Backdoor Activity M7 (malware.rules)The release notes for the rules have this comment, it looks like they might have missed part of the range of rules that don't work with Snort 2.9.
Hey folks, I thought I’d add a little bit more context for tonight’s rule release:
Rules
2046275 - 2046279 (SEASPY) are only available for Suricata 5+. This is due to detection engine limitations – neither Suricata 4.x nor Snort 2.9 have the tcp.hdr option to perform content matching in the TCP header portion of network traffic.Please refer to: Barracuda ESG Zero-Day Vulnerability (CVE-2023-2868) Exploited Globally by Aggressive and Skilled Actor, Suspected Links to China | Mandiant 2
It appears that a number of the SEASPY detection rules rely on analyzing specific TCP OPTIONS.
Take a look at the documentation for tcp.hdr for more information: 5.3. IP Keywords — Suricata unknown documentation 2
-
@zoomequipd have you seen this bug?
-
@bmeeks you should make a Redmine request about this package with your findings, that way it will get fixed.
-
@JonathanLee said in Snort fatal error after emerging.rules update:
@bmeeks you should make a Redmine request about this package with your findings, that way it will get fixed.
I am the package developer/maintainer for both Snort and Suricata on pfSense. I maintain both packages, not Netgate. A Redmine ticket makes no sense for this issue.
This is not a "bug" in the package. It is an error in the Emerging Threats rule package produced by other parties (in this case Proofpoint, who bought Emerging Threats a few years ago). The creators of the rules package will fix this problem. This is not the first time an error has been introduced by a rules package update from a vendor.
-
@bmeeks said in Snort fatal error after emerging.rules update:
@JonathanLee said in Snort fatal error after emerging.rules update:
@bmeeks you should make a Redmine request about this package with your findings, that way it will get fixed.
I am the package developer/maintainer for both Snort and Suricata on pfSense. I maintain both packages, not Netgate. A Redmine ticket makes no sense for this issue.
This is not a "bug" in the package. It is an error in the Emerging Threats rule package produced by other parties (in this case Proofpoint, who bought Emerging Threats a few years ago). The creators of the rules package will fix this problem. This is not the first time an error has been introduced by a rules package update from a vendor.
Thx for the update!
I hope we will know when it will be resolved...
-
Emerging threats released out of band rules update to resolve.
https://community.emergingthreats.net/t/ruleset-update-summary-2023-06-15-v10349/656/4
This problem was due to two rules using the syntax flow:stateless,to_server for the snort version of two of the SEASPY rules. Snort does not like having flow:stateless combined with other flow options and throws an error. The error isn’t formatted like any of the other errors Snort typically throws regarding rule syntax errors, and our QA systems missed it. Our QA system has been updated to account for this error, and we’ve released an emergency out of band update that is available now to fix this problem.
We apologize for any inconvenience this has caused you or any other netgate customers, and have made necessary precautions to prevent it from happening in the future. If there is anything else I can do for you, please let me know.
-
@valete3 said in Snort fatal error after emerging.rules update:
Emerging threats released out of band rules update to resolve.
https://community.emergingthreats.net/t/ruleset-update-summary-2023-06-15-v10349/656/4
This problem was due to two rules using the syntax flow:stateless,to_server for the snort version of two of the SEASPY rules. Snort does not like having flow:stateless combined with other flow options and throws an error. The error isn’t formatted like any of the other errors Snort typically throws regarding rule syntax errors, and our QA systems missed it. Our QA system has been updated to account for this error, and we’ve released an emergency out of band update that is available now to fix this problem.
We apologize for any inconvenience this has caused you or any other netgate customers, and have made necessary precautions to prevent it from happening in the future. If there is anything else I can do for you, please let me know.
Thanks @valete3 for the status update and getting the issue resolved
. -
@bmeeks Hello thanks for the reply, Sorry I should have been more clear with this. I did not mean the Snort package, I was talking about the Watchdog package, you seemed to have a lot to say about that package as well as knowledge on a direction on what needs to be done to fix it/improve it. Back to Snort, it seems has no code written for error conditions such as this. I assumed that if such errors occured in the past and still continues to occur once and a while that some type of code/function could be implemented to catch this error. That way it could at least bring Snort into an active state again. Right now as it sits we can agree that this error condition has the ability to offline and degrade a firewalls whole IPS/IDS security system. This error flat took it offline and disabled it. Therefore the Snort pfSense IPS/IDS package's uptime took a massive hit. It degraded performance that drastically within the firewall. Cisco ASA and Firepower, Palo Alto IPS/IDS, Juniper, Ciena, Riverbed, and even smaller Meraki systems would see this as a major security concern as a couple rules in this set did have the ability to offline the security system.
It's the weak point in the wall.
We need a backup plan like tar and feathers type code.. .Make sure to upvote
-
-
@JonathanLee said in Snort fatal error after emerging.rules update:
I assumed that if such errors occured in the past and still continues to occur once and a while that some type of code/function could be implemented to catch this error.
And exactly how should the Snort GUI package PHP code identify what's wrong and fix it? How in the world can the PHP code know what's wrong within the third-party Snort binary?
All it knows is the binary threw an error starting up and aborted. The PHP code can do nothing at that point.
If uptime is such a huge concern, then the very first rule of reliability is never, ever let a system perform automatic unattended updates. You can never know what might get pushed out. For high reliability installs, you should personally manage all updates (including for IDS rules) in person at the firewall so you are there to take remedial action if necessary. And most every large outfit I've been involved with has a dedicated test system that is a duplicate of production, and they thoroughly test all updates in that test system prior to rolling them out to production.
-
@bmeeks What about it keeps the old ruleset if it fails with such a condition reload the old rules and stop for a period of a sometime like 8 hours and try again. Log the errors so everyone can see with a log that says loading prior rule set with a date and time? It pointed to a rule after it updated, so it should in theory be able to catch the error and do an if statement.
You like the castle don't you
-
Main issue: Snort fails completely open within this situation. Snort does not function at all during this error.
https://redmine.pfsense.org/issues/14480
If anyone else wants to provide background I made a redmine about it because it causes a fail open condition. Logs or anything would be great. Thanks @bmeeks for the help fixing this and with the background on what causes this issue.
The timing of this issue and relationships to what CISA reported today is of concern. I can't imagine a worse time to to have this fail open.
https://www.reuters.com/world/us/us-government-agencies-hit-global-cyber-attack-cnn-2023-06-15/ -
@chudak I also got the update and it works again, make sure to also normalize the 2 rules that were worked on so they detect again. I had them disabled and had them turned on for a couple hours now no issues.
-
@bmeeks Thanks for the clarification about line numbers vs rule ID's.
-
@valete3 I've forced an update and re-enabled the rule set I had disabled as a workaround and all seems well now.
-
@InstanceExtension Greetings - and apologies all for the disruption this caused last week. As identified within this thread, two rules (SIDs 2046273 and 2046274) were released to the live ruleset with syntax errors. The rules had the "flow: stateless" option set with "to_server" also set which causes a Fatal Error within Snort. Upon investigation it was found that due to a text parsing issue in our QA infrastructure these errors were missed and the rules were released into production.
Going forward, we have made adjustments to our QA process to ensure this will not recur and errors of this sort will be caught within our QA process and mended. The next morning we released an out-of-band update to address.
Feel free to reach out here via DM, on twitter (@et_labs) or on our Discourse.
