URLHaus - Anyone have a mod already?
-
Has anyone already found a way to use the Suricata rules from URLHaus in the pfSense package?
Or Bill, if this is trivial for you to implement, have you seen the available ruleset described below?
https://urlhaus.abuse.ch/about/
https://urlhaus.abuse.ch/downloads/ids/ -
You can paste that list in as a Custom Rule on the RULES tab. That's going to be the only way to use those rules unless you want to go outside of the GUI entirely and run Suricata from a command-line prompt.
For now I only intend to support the Emerging Threats and Snort Subscriber Rules downloads in the GUI package. Those rules are packaged in a similar format and are widely respected and used. Outside of them you get more varied results and changed packaging conditions. For example, the list you mention does not appear to come in the tarball format the Emerging Threats and Snort rules do. The GUI rules update code expects (and today, needs) the tarball format in order to work.
-
@bmeeks said in URLHaus - Anyone have a mod already?:
For example, the list you mention does not appear to come in the tarball format the Emerging Threats and Snort rules do. The GUI rules update code expects (and today, needs) the tarball format in order to work.
Point taken on the consistency and your intent. I would just note that they do make a tarball version available here: https://urlhaus.abuse.ch/downloads/urlhaus.tar.gz
abuse.ch is also one of the anchors in the pfBlocker package as its other services are used in the firehol block lists. But it's not as professional as Talos or Proofpoint.
The custom rules tab isn't a great option because the list changes quite a bit as it only contains and blocks active malware urls.
Anyway, figured I'd ask! Thanks again!
-
You can do this semi-automatically on your firewall if you are willing to write you own download script and then modify one of the Suricata PHP source files. Bear in mind that if you do this, you will need to repeat the modification of the Suricata source file each time you update the Suricata package.
Create your own shell script to download the rules file or files you want and write the contents to a directory on your firewall.
Open the file /usr/local/pkg/suricata/suricata_yaml_template.inc and search for this section (it should start on line #266):
default-rule-path: {$suricatacfgdir}/rules rule-files: - {$rules_files} classification-file: {$suricatacfgdir}/classification.config reference-config-file: {$suricatacfgdir}/reference.configDon't change what is already there, but you can add additional lines underneath this one to indicate to Suricata which additional rules files to use. Note this is a global change, so all configured Suricata interfaces will use any added rules files!
- {$rules_files}So you might have something like this, for example:
- {$rules_files} - /root/my_rules.rulesNote that the indenting is critical. Keep the lines all lined up or the YAML parser will complain and Suricata will fail to start. Be safe and make a backup of the file before you modify it.
You were also asking about a custom reference.config file in another thread, your solution to that can be found here as well. See the lines following the rules files parameters.
-
Sounds great, I'll give this a shot.
Many thanks!
-
Created directory /usr/local/etc/suricata/rules.local/
Created shell script /usr/local/bin/urlhaus_suricata_rules_update.sh with 744:
#!/bin/sh fetch -o /usr/local/etc/suricata/rules.local/urlhaus.tar.gz https://urlhaus.abuse.ch/downloads/urlhaus.tar.gz tar xvfz /usr/local/etc/suricata/rules.local/urlhaus.tar.gz -C /usr/local/etc/suricata/rules.local/ rm /usr/local/etc/suricata/rules.local/urlhaus.tar.gz chown root:wheel /usr/local/etc/suricata/rules.local/urlhaus.rules chmod 644 /usr/local/etc/suricata/rules.local/urlhaus.rulesAdded the following to /etc/crontab (3 minutes prior to normal Suricata rules update):
20 0,6,12,18 * * * root /usr/local/bin/urlhaus_suricata_rules_update.shModified /usr/local/pkg/suricata/suricata_yaml_template.inc as shown above to include
- /usr/local/etc/rules.local/urlhaus.rulesDouble-checked YAML format was correct.
Generated YAML file /usr/local/etc/suricata/suricata_43414_igb0/suricata.yaml appears to contain what I wanted it to
default-rule-path: /usr/local/etc/suricata/suricata_43414_igb0/rules rule-files: - suricata.rules - flowbit-required.rules - /usr/local/etc/suricata/rules.local/urlhaus.rulessuricata.log indicates the rules are processed properly (included one rule error to show it's processing urlhaus.rules):
-> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/url=http://sietepuntocero.com.ar/en_us/messages/112018|26|data=02|01|kbesic@pella.com|17810e138c1d413ab8a108d64a6df3be|a66b0f6bd9534f0995b75213bd230c18|0|0|636778233436312957|26|sdata=bdjpihczaitno2gt/kt/9owjxappq2frvcm5id4tppe=|26|reserved=0"; http_uri; depth:243; isdataat:!1,relative; content:"na01.safelinks.protection.outlook.com"; http_host; depth:37; isdataat:!1,relative; metadata:created_at 2018_11_14; reference:url, urlhaus.abuse.ch/url/80452/; classtype:trojan-activity;sid:80943552; rev" from file /usr/local/etc/suricata/rules.local/urlhaus.rules at line 4244 25/11/2018 -- 14:26:56 - <Info> -- 3 rule files processed. 46668 rules successfully loaded, 2159 rules failed 25/11/2018 -- 14:26:56 - <Info> -- Threshold config parsed: 5 rule(s) found 25/11/2018 -- 14:26:57 - <Info> -- 46672 signatures processed. 1119 are IP-only rules, 6740 are inspecting packet payload, 33983 inspect application layer, 103 are decoder event onlyEverything looks good.
5 seconds later in system.log:
Nov 25 14:27:02 rawr kernel: pid 86298 (suricata), uid 0: exited on signal 6 (core dumped)No additional information in suricata.log or system.log that I can see. But apparently Suricata is aborting on something?
I understand this isn't supported, so if you're done offering advice that makes sense -- I'll tinker with it -- but if anything jumps out at you I'm all ears.
-
Ah, disabling hyperscan (AC Pattern Match) allows Suricata to start and function fully.
As per a thread (at that-which-shall-not-be-named), I'm thinking this is a hyperscan bug or a bad pattern likely confirming your view that the quality of the rules isn't there.
I'll see if I can track down the particular rule(s).
Otherwise, this seems to work.
-
@boobletins said in URLHaus - Anyone have a mod already?:
Ah, disabling hyperscan (AC Pattern Match) allows Suricata to start and function fully.
As per a thread (at that-which-shall-not-be-named), I'm thinking this is a hyperscan bug or a bad pattern likely confirming your view that the quality of the rules isn't there.
I'll see if I can track down the particular rule(s).
Otherwise, this seems to work.
Yes, I agree that something in the new rule set is likely triggering a bug in the hyperscan code. That is yet another independent library Suricata uses (the binary, not my GUI package). That's one of my pet peaves about a lot of Unix packages these days, they are loading up all kinds of third-party libraries and it is very easy to have compatibility issues with different versions of the libraries. Reminds me very much of the "DLL Hell" that Linux users teased Windows users about. Now the Unix/Linux/FreeBSD users have their own custom version of "DLL Hell" in the form of a bunch of shared libraries. I know it's not memory efficient and is considered out-of-date, but boy am I a fan of static-linked runtime code where the versions of third-party libraries are fixed to a particular binary.
You can try reporting this bug upstream to the Suricata team. Just tell them it's an issue with that rule (very nice if you can figure out which one) and the hyperscan feature. I would not even necessarily tell them you are using pfSense. If you do, they might want to circle you back to here. But I believe that is either a bug in their hyperscan implementation or else in the FreeBSD libary. Just tell them you are running Suricata on FreeBSD 11.2 for an OS.
-
Thanks for the advice, I'll be sure to do that -- it also won't be far from the truth.
I just built hyperscan v 5.0 on a VM of FreeBSD 11.2 to see if they had already fixed the bug. After some file renaming shenanigans, I was able to run Suricata in Inline mode with hyperscan 5.0 just fine until I added back in the urlhaus rules. So no love from version 5, I'll see if the debug compile version will give me any hints and then take your advice.
In my libs I also added in AVX2 support as I'm using an i5. Do you know if that is that enabled in the version for FreeBSD? I wouldn't think so since we can't use the "fat" binaries? There would have to be multiple versions and some detection scheme, and it doesn't look like there is one?
-
I don't know about the AVX2 support. That's not in my area of expertise.
So the Signal 6 happens after all the rules have been parsed and actual traffic inspection begins? That may be a hard problem to find unless you compile debug versions of the Suricata binary and the hyperscan library. The trick for troubleshooting is finding which exact rule is giving the problem.
Since you know how to run Suricata using a pure command-line setup, see if you can duplicate the issue on a Linux virtual machine. I suspect it does not matter if you use inline mode or just plain libpcap alerting mode, either should trigger the bug if it's where we think (in the hyperscan code). That code is loaded no matter if blocking is enabled or not.
In my view, even a poorly written rule should never result in a core dump. So I would say the problem should be looked at by either the Suricata or hyperscan teams. If it crashes on Linux, that would point the finger more towards the Suricata code. If it runs fine on Linux but crashes only on FreeBSD, well that of course points the finger at the FreeBSD port of hyperscan.
-
So this is fun:
I temporarily stopped working on getting the debug version of hyperscan 5.0 to compile (issues with their make file and headers, I believe) and instead did a kind of manual binary search.
I started with the entire rule set, use split -l [half of the lines] and kept trying to load both sets of rules to see which crashed.
I figured in a few iterations I'd have the rule identified. How wildly optimistic of me.
I have attached a file containing 16 rules from my binary sort. If I attempt to add that file to my rules, hyperscan (or similar) crashes.
So, I continued my process and split that into the 2 attached files below (8.1 and 8.2 ---- .0's added so they can be uploaded)
1_1543200208633_8.2.0 0_1543200208633_8.1.0
The annoying thing is that 8.1 and 8.2 load just fine separately. It's something about the rules combined into the file of 16 -- some interaction between the rules is causing the problem.
I double checked with a diff checker and the split files, when combined, are exactly the same as the whole 16.
That's... I mean come on! :(
I'll try to replicate on Linux and see what happens. If anyone else wants to test these to confirm I'm not crazy that'd be welcome.
-
Compiled the latest Hyperscan and Suricata versions with debug enabled.
Findings are detailed here:
https://redmine.openinfosecfoundation.org/issues/2714Essentially, there's a check to try to weed out duplicate signatures/patterns, but either the process by which Suricata is hashing the signatures or some truncation somewhere, 2 long and similar (but not identical) patterns are taken to be equivalent. An assertion checks to make sure they are, in fact, the same and fails because they aren't.
Pretty sure this is a Suricata problem. The patterns in question are interesting -- extremely long URLs -- I wonder if there's an unknown limit somewhere.
I suspect if I remove patterns above a certain length the hashing will work again.
-
@boobletins said in URLHaus - Anyone have a mod already?:
Compiled the latest Hyperscan and Suricata versions with debug enabled.
Findings are detailed here:
https://redmine.openinfosecfoundation.org/issues/2714Essentially, there's a check to try to weed out duplicate signatures/patterns, but either the process by which Suricata is hashing the signatures or some truncation somewhere, 2 long and similar (but not identical) patterns are taken to be equivalent. An assertion checks to make sure they are, in fact, the same and fails because they aren't.
Pretty sure this is a Suricata problem. The patterns in question are interesting -- extremely long URLs -- I wonder if there's an unknown limit somewhere.
I suspect if I remove patterns above a certain length the hashing will work again.
Excellent detective work. That report should help them narrow down the issue. Maybe it's just something as simple as a memory buffer overflow or something. Of course the potential for a hash collision is also there, although you generally expect that to be a very rare occurrence. The typical method for handling a hash collision is to then compare the two strings byte-by-byte to see if they are really different or not.
-
@boobletins said in URLHaus - Anyone have a mod already?:
Created directory /usr/local/etc/suricata/rules.local/
Created shell script /usr/local/bin/urlhaus_suricata_rules_update.sh with 744:
#!/bin/sh fetch -o /usr/local/etc/suricata/rules.local/urlhaus.tar.gz https://urlhaus.abuse.ch/downloads/urlhaus.tar.gz tar xvfz /usr/local/etc/suricata/rules.local/urlhaus.tar.gz -C /usr/local/etc/suricata/rules.local/ rm /usr/local/etc/suricata/rules.local/urlhaus.tar.gz chown root:wheel /usr/local/etc/suricata/rules.local/urlhaus.rules chmod 644 /usr/local/etc/suricata/rules.local/urlhaus.rulesAdded the following to /etc/crontab (3 minutes prior to normal Suricata rules update):
20 0,6,12,18 * * * root /usr/local/bin/urlhaus_suricata_rules_update.shModified /usr/local/pkg/suricata/suricata_yaml_template.inc as shown above to include
- /usr/local/etc/rules.local/urlhaus.rulesDouble-checked YAML format was correct.
Generated YAML file /usr/local/etc/suricata/suricata_43414_igb0/suricata.yaml appears to contain what I wanted it to
default-rule-path: /usr/local/etc/suricata/suricata_43414_igb0/rules rule-files: - suricata.rules - flowbit-required.rules - /usr/local/etc/suricata/rules.local/urlhaus.rulessuricata.log indicates the rules are processed properly (included one rule error to show it's processing urlhaus.rules):
-> $EXTERNAL_NET any (msg:"URLhaus Known malware download URL detected"; flow:established,from_client; content:"GET"; http_method; content:"/url=http://sietepuntocero.com.ar/en_us/messages/112018|26|data=02|01|kbesic@pella.com|17810e138c1d413ab8a108d64a6df3be|a66b0f6bd9534f0995b75213bd230c18|0|0|636778233436312957|26|sdata=bdjpihczaitno2gt/kt/9owjxappq2frvcm5id4tppe=|26|reserved=0"; http_uri; depth:243; isdataat:!1,relative; content:"na01.safelinks.protection.outlook.com"; http_host; depth:37; isdataat:!1,relative; metadata:created_at 2018_11_14; reference:url, urlhaus.abuse.ch/url/80452/; classtype:trojan-activity;sid:80943552; rev" from file /usr/local/etc/suricata/rules.local/urlhaus.rules at line 4244 25/11/2018 -- 14:26:56 - <Info> -- 3 rule files processed. 46668 rules successfully loaded, 2159 rules failed 25/11/2018 -- 14:26:56 - <Info> -- Threshold config parsed: 5 rule(s) found 25/11/2018 -- 14:26:57 - <Info> -- 46672 signatures processed. 1119 are IP-only rules, 6740 are inspecting packet payload, 33983 inspect application layer, 103 are decoder event onlyEverything looks good.
5 seconds later in system.log:
Nov 25 14:27:02 rawr kernel: pid 86298 (suricata), uid 0: exited on signal 6 (core dumped)No additional information in suricata.log or system.log that I can see. But apparently Suricata is aborting on something?
I understand this isn't supported, so if you're done offering advice that makes sense -- I'll tinker with it -- but if anything jumps out at you I'm all ears.
Hello to all,
can anyone confirm that is working now?i want to use the suricata rules of urlhaus - https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz
is this the only way as described above ?BR
-
for me it does not work at all according to the instructions linked above.
just i overwrite the custom.rules with the following script:
#!/bin/sh fetch -o /usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz https://urlhaus.abuse.ch/downloads/urlhaus_suricat a.tar.gz tar xvfz //usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz -C /usr/local/etc/suricata/rules.local/ rm /usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz rm /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rules mv /usr/local/etc/suricata/rules.local/urlhaus_suricata.rules /usr/local/etc/suricata/suricata_32296_igb1.300/rules/cus tom.rules chown root:wheel /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rules chmod 644 /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rulesunder cutsom.rules was nothing to see and after the refresh the custom.rules had shrunk to 0 kb :/
anybody still some tips for me?
BR -
@p54 said in URLHaus - Anyone have a mod already?:
for me it does not work at all according to the instructions linked above.
just i overwrite the custom.rules with the following script:
#!/bin/sh fetch -o /usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz https://urlhaus.abuse.ch/downloads/urlhaus_suricat a.tar.gz tar xvfz //usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz -C /usr/local/etc/suricata/rules.local/ rm /usr/local/etc/suricata/rules.local/urlhaus_suricata.tar.gz rm /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rules mv /usr/local/etc/suricata/rules.local/urlhaus_suricata.rules /usr/local/etc/suricata/suricata_32296_igb1.300/rules/cus tom.rules chown root:wheel /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rules chmod 644 /usr/local/etc/suricata/suricata_32296_igb1.300/rules/custom.rulesunder cutsom.rules was nothing to see and after the refresh the custom.rules had shrunk to 0 kb :/
anybody still some tips for me?
BRGuys, Snort and Suricata on pfSense are GUI applications! That means everything is configured within the GUI, stored in the
config.xmlof the firewall, and then written out to the pertinent configuration and rules files each time Suricata or Snort is started. So all those changes you make via a CLI session are immediately overwritten the instant you go back into the GUI and start Suricata or Snort.Custom rules must be entered from the RULES tab in the GUI. They are then stored in
config.xmlas Base64-encoded text. That text is read back out and written to thecustom.rulesfile each time the GUI is used to start Suricata or Snort.If you absolutely must have those rules, then you have to make an edit to the PHP source code files of the package. And be aware that change will be overwritten the next time you update or otherwise reinstall the package. You will need to manually edit the file
/usr/local/pkg/suricata/suricata_yaml_template.incand include a path to your URLHaus rules file in that template. The template is used to create thesuricata.yamlfile for each interface. Be warned that if you are not familiar with PHP coding syntax, you can very easily break your Suricata installation.
