Suricata-4.1.7 Package Update - Release Notes

[bmeeks]

@kiokoman:
The pfSense team is currently working on some behind the scenes "magic" for the pkg utility to work out installing the new Suricata 5.x branch on pfSense-2.4.5-RC installs that are running on AMD64 or aarch64 hardware. Look for that to show up soon.

Same thing will happen with pfSense-2.5-DEVEL except it is waiting on 2.5-DEVEL to move to FreeBSD-12/STABLE. So Suricata 5.x will take longer to show up in the 2.5-DEVEL branch.

The problems with different versions on various releases is all due to the upstream decision by the Suricata development team to make the Rust language mandatory for compiling Suricata 5.x. That greatly complicated the Suricata package on pfSense because of the need to support ARM 32-bit hardware such as the SG-1000 and SG-3100 boxes which can't run Rust currently as well as aarch64 ARM hardware and AMD64 (Intel) hardware which can run Rust. So any device that can't run Rust must run only the Suricata 4.x binary and associated GUI package.

[kiokoman]

sorry maybe i was not clear, i want to know if Suricata-4.1.7 will be available later on for 2.4.5 and 2.5.0 because you said that it's only for pfSense-2.4.4_p3 RELEASE

[bmeeks]

@kiokoman said in Suricata-4.1.7 Package Update - Release Notes:

sorry maybe i was not clear, i want to know if Suricata-4.1.7 will be available later on for 2.4.5 and 2.5.0 because you said that it's only for pfSense-2.4.4_p3 RELEASE

Oh, yeah I was not clear on that. Yes, the Suricata 4.1.7 package should be available but it will get a new name with "4" in it. Probably something like "Suricata4", for example. So it will take a conscious effort on the user's part to install the 4.x version.

Exactly how this all works is being determined by the pfSense pkg guru.

[NollipfSense]

Bravo Bill...now running Suricata 5.0.2 on pfSense 2.5-dev...everything seems okay; however, I got this below and wanted to know is there anything needs tweaking!

4/3/2020 -- 12:14:34 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
4/3/2020 -- 12:14:34 - <Info> -- CPUs/cores online: 8
4/3/2020 -- 12:14:34 - <Info> -- HTTP memcap: 67108864
4/3/2020 -- 12:14:34 - <Notice> -- using flow hash instead of active packets
4/3/2020 -- 12:14:34 - <Info> -- Netmap: Setting IPS mode
4/3/2020 -- 12:14:34 - <Info> -- fast output device (regular) initialized: alerts.log
4/3/2020 -- 12:14:34 - <Info> -- http-log output device (regular) initialized: http.log
4/3/2020 -- 12:14:34 - <Info> -- stats output device (regular) initialized: stats.log
4/3/2020 -- 12:14:37 - <Info> -- Rule with ID 2026440 is bidirectional, but source and destination are the same, treating the rule as unidirectional
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/suricata_18986_igb0/rules/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract MD5"; filemd5:fileextraction-chksum.list; filestore; sid:28; rev:1;)" from file /usr/local/etc/suricata/suricata_18986_igb0/rules at line 20533
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/suricata_18986_igb0/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract SHA1"; filesha1:fileextraction-chksum.list; filestore; sid:29; rev:1;)" from file /usr/local/etc/suricata/suricata_18986_igb0 at line 20534
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract SHA256"; filesha256:fileextraction-chksum.list; filestore; sid:30; rev:1;)" from file /usr/local/etc/suricata at line 20535
4/3/2020 -- 12:14:44 - <Info> -- 2 rule files processed. 20645 rules successfully loaded, 3 rules failed
4/3/2020 -- 12:14:44 - <Info> -- Threshold config parsed: 0 rule(s) found
4/3/2020 -- 12:14:44 - <Info> -- 20655 signatures processed. 21 are IP-only rules, 6018 are inspecting packet payload, 14357 inspect application layer, 99 are decoder event only
4/3/2020 -- 12:14:44 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.ButterflyJoin' is checked but not set. Checked in 2011296 and 0 other sigs
4/3/2020 -- 12:14:44 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.Netwire.HB.1' is checked but not set. Checked in 2018282 and 0 other sigs
4/3/2020 -- 12:14:44 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs
4/3/2020 -- 12:15:15 - <Info> -- Going to use 1 thread(s)
4/3/2020 -- 12:15:15 - <Notice> -- opened netmap:igb0/R from igb0: 0x81213c000
4/3/2020 -- 12:15:15 - <Notice> -- opened netmap:igb0^ from igb0^: 0x81213c300
4/3/2020 -- 12:15:15 - <Info> -- using magic-file /usr/share/misc/magic
4/3/2020 -- 12:15:15 - <Info> -- Going to use 1 thread(s)
4/3/2020 -- 12:15:15 - <Notice> -- opened netmap:igb0^ from igb0^: 0x81401c000
4/3/2020 -- 12:15:15 - <Notice> -- opened netmap:igb0/T from igb0: 0x81401c300
4/3/2020 -- 12:15:15 - <Info> -- using magic-file /usr/share/misc/magic
4/3/2020 -- 12:15:15 - <Notice> -- all 2 packet processing threads, 2 management threads initialized, engine started.

[bmeeks]

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

fileextraction-chksum.list

I will need to look into this particular error some more. It appears, at first glance, to be some new configurable parameter for 5.0. But I don't see anything referencing it in the documentation within the Suricata 5.x source tarball.

For now the errors mean those rules are not being loaded by Suricata. The flowbit errors are harmless.

[bmeeks]

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

Bravo Bill...now running Suricata 5.0.2 on pfSense 2.5-dev...everything seems okay; however, I got this below and wanted to know is there anything needs tweaking!

4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/suricata_18986_igb0/rules/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract MD5"; filemd5:fileextraction-chksum.list; filestore; sid:28; rev:1;)" from file /usr/local/etc/suricata/suricata_18986_igb0/rules at line 20533
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/suricata_18986_igb0/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract SHA1"; filesha1:fileextraction-chksum.list; filestore; sid:29; rev:1;)" from file /usr/local/etc/suricata/suricata_18986_igb0 at line 20534
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_OPENING_RULE_FILE(41)] - opening hash file /usr/local/etc/suricata/fileextraction-chksum.list: No such file or directory
4/3/2020 -- 12:14:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert http any any -> any any (msg:"Black list checksum match and extract SHA256"; filesha256:fileextraction-chksum.list; filestore; sid:30; rev:1;)" from file /usr/local/etc/suricata at line 20535

Okay, after some research it appears your file extraction errors are self-inflicted. Those rules come from the default file.rules file iincluded with the Suricata source code. However, all of those rules are disabled by default by the Suricata team. Did you enable them yourself on the RULES tab? You must have done that because I did not get this error in my testing with the 5.0.2 binary. They are default disabled for a reason.

Do not enable things that are default disabled unless you are sure of what you are doing. In this case you would need to create your own text file of MD5 checksums and copy it to the correct location.

[NollipfSense]

@bmeeks said in Suricata-4.1.7 Package Update - Release Notes:

They are default disabled for a reason.
Do not enable things that are default disabled unless you are sure of what you are doing.

So, these rules (files.rules category) were default disabled? I don't remember enabling or disabling any...they're from pfSense 2.4days...all seems self-inflicted as the code suggests...none showing default color code. Changed all to default which resulted in all default disabled...thank you, Bill.

[NollipfSense]

@bmeeks What about this SC_Warn_Flowbits...anything needs tweaking?

4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.ButterflyJoin' is checked but not set. Checked in 2011296 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.Netwire.HB.1' is checked but not set. Checked in 2018282 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs

[NollipfSense]

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

@bmeeks What about this SC_Warn_Flowbits...anything needs tweaking?

4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.ButterflyJoin' is checked but not set. Checked in 2011296 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.Netwire.HB.1' is checked but not set. Checked in 2018282 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs

Found the flowbits...just not sure how to go about setting what is checked...according to the read me: If these dependent flowbits are not set, then some of your chosen rules may not fire. Enabling all the rules that set these dependent flowbits ensures your chosen rules fire as intended.

[bmeeks]

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

@bmeeks What about this SC_Warn_Flowbits...anything needs tweaking?

4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.ButterflyJoin' is checked but not set. Checked in 2011296 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.Netwire.HB.1' is checked but not set. Checked in 2018282 and 0 other sigs
4/3/2020 -- 23:02:24 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs

Found the flowbits...just not sure how to go about setting what is checked...according to the read me: If these dependent flowbits are not set, then some of your chosen rules may not fire. Enabling all the rules that set these dependent flowbits ensures your chosen rules fire as intended.

You need to go do some Google research on what flowbits are and how they are used in Suricata and Snort. Once you do that, you will be able to answer your question, and you will have learned more about instrusion detection systems ... .

BTW, I did answer your question in my first reply earlier in this thread, but if you want to learn why those errors are likely harmless like I said, then go do the suggested Google research.

[NollipfSense]

@bmeeks said in Suricata-4.1.7 Package Update - Release Notes:

BTW, I did answer your question in my first reply earlier in this thread, but if you want to learn why those errors are likely harmless like I said, then go do the suggested Google research.

Just noticed that...thank you; however, I have gone down the path of curiosity...despite their harmlessness...learning never stops!

[NollipfSense]

@bmeeks said in Suricata-4.1.7 Package Update - Release Notes:

You need to go do some Google research on what flowbits are and how they are used in Suricata and Snort. Once you do that, you will be able to answer your question, and you will have learned more about instrusion detection systems ...

Ah...Bill, I found the answer: *We recommend using suricata-update for downloading and updating the rules. It will automatically resolve flowbits issues.

The warning tells you a rule can never match. So you can disable it at no loss, but it's probably better to enable the rule(s) that set the flowbit. Again, suricata-update automates this for you*...https://redmine.openinfosecfoundation.org/issues/2702

All auto-flowbits rules were enabled by default.

[bmeeks]

@NollipfSense said in Suricata-4.1.7 Package Update - Release Notes:

@bmeeks said in Suricata-4.1.7 Package Update - Release Notes:

You need to go do some Google research on what flowbits are and how they are used in Suricata and Snort. Once you do that, you will be able to answer your question, and you will have learned more about instrusion detection systems ...

Ah...Bill, I found the answer: *We recommend using suricata-update for downloading and updating the rules. It will automatically resolve flowbits issues.

The warning tells you a rule can never match. So you can disable it at no loss, but it's probably better to enable the rule(s) that set the flowbit. Again, suricata-update automates this for you*...https://redmine.openinfosecfoundation.org/issues/2702

All auto-flowbits rules were enabled by default.

The pfSense Suricata package has the same logic (more or less) as the suricata-update program and the older PulledPork for Snort. The most likely issue in your case is that the rule set itself is missing the "flowbits:set" option in some rule. To see, you can do a grep on all of your downloaded rules looking for any that contain either of those two flowbit names (et.MCOFF or et.DocVBAProject). Your downloaded Suricata rules set is in /usr/local/share/suricata/rules. Look and see if you find one or more rules that "set" those flowbit names.

[NollipfSense]

@bmeeks BTW Bill, it did sort itself out...I didn't needed to do anything.