Emerging Threats Pro rules file download failed. Bad MD5 checksum.
-
I'm experiencing the same issue. I have an ET pro ruleset subscription. I have a single instance of pfSense that runs both Snort and Suricata. Snort is able to successfully download and use the ET Pro ruleset. Suricata gives an MD5 checksum error when downloading the ET Pro ruleset:
Emerging Threats Pro rules file download failed. Bad MD5 checksum.
Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
Expected Emerging Threats Pro rules file MD5:
Emerging Threats Pro rules file download failed. Emerging Threats Pro rules will not be updated.Since Snort is working, I suspect the issue is not with the way the files are being served. I verified that I have the same ET PRO ruleset code configured for Snort and Suricata (copy/paste after all). Any suggestions?
I'm running the latest version of pfSense, 2.4.2.
Thanks,
SteveI will need to test in a Suricata virtual machine. I use Snort for my personal home network (just because that's what I started with before I created the Suricata package). Is this something that just started happening? If so, can you give me a time table for when it began?
Bill
-
For me this is a new install. New hardware and a fresh install of pfSense 2.4.2. This is the first time I've ever had an ET Pro subscription. Unfortunately, I can't tell you if this is a new problem or was preexisting, its a new issue to me. That said, its working today. Sometime between midnight and 8am (PST) it started working:
Starting rules update… Time: 2017-12-01 00:30:00
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Emerging Threats Pro rules file download failed. Bad MD5 checksum.
Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
Expected Emerging Threats Pro rules file MD5:
Emerging Threats Pro rules file download failed. Emerging Threats Pro rules will not be updated.
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
The Rules update has finished. Time: 2017-12-01 00:30:07Starting rules update... Time: 2017-12-01 08:18:15
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Extracting and installing Emerging Threats Pro rules...
Installation of Emerging Threats Pro rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
Updating rules configuration for: LAN ...
Live-Reload of updated rules is enabled...
Live swap of updated rules requested for WAN.
Live swap of updated rules requested for LAN.
Live-Reload of the updated rules is complete.
The Rules update has finished. Time: 2017-12-01 08:18:29I guess we'll have to blame Proofpoint in this case. Thanks for the help.
-
It sucess download yesterday. :D
-
Thanks for the feedback. Sometimes the rules vendors have hiccups in the distribution networks.
Bill
-
Bill,
I have been seeing the same issue since I installed pfSense 2.4.2 on 23 November. Until now, I haven't had any significant periods of failed updates in the three years I have been running pfSense and Suricata. I haven't made any changes to configuration in months.
Starting rules update... Time: 2017-11-23 12:30:00 Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5... Checking Emerging Threats Pro rules md5 file... There is a new set of Emerging Threats Pro rules posted. Downloading file 'etpro.rules.tar.gz'... Done downloading rules file. Emerging Threats Pro rules file download failed. Bad MD5 checksum. Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158 Expected Emerging Threats Pro rules file MD5: Emerging Threats Pro rules file download failed. Emerging Threats Pro rules will not be updated. The Rules update has finished. Time: 2017-11-23 12:30:08
A few things to note:
The MD5 checksum for the rules file quoted in that error message does not change ie it is the same today as it was on the November 23 (and also seems to match the other log entries posted here). The expected MD5 checksum is never populated. Is it even being read correctly? If the MD5 of the downloaded rules isn't changing it is either not downloading the rules correctly or it is downloading the same unchanging file each time.
Around the same time, Emerging Threats enabled a new ruleset for Suricata 4.0. I don't know if they modified their folder structure for older engines or not. I can't see exactly where the updater is looking for the files so its hard for me to troubleshoot this by downloading the original ET rules files and manually computing the MD5 ie is it trying to pull 2.0 rules or 3.2 rules or 4.0 rules?
https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html
I uninstalled and then reinstalled Suricata (keeping settings) over the weekend. Immediately after install I saw the following successful log entry:
Starting rules update... Time: 2017-12-02 06:30:00 Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5... Checking Emerging Threats Pro rules md5 file... Emerging Threats Pro rules are up to date. The Rules update has finished. Time: 2017-12-02 06:30:06
However, subsequent updates continue to fail with exactly the same MD5 checksum error as above:
Starting rules update... Time: 2017-12-04 06:30:00 Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5... Checking Emerging Threats Pro rules md5 file... There is a new set of Emerging Threats Pro rules posted. Downloading file 'etpro.rules.tar.gz'... Done downloading rules file. Emerging Threats Pro rules file download failed. Bad MD5 checksum. Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158 Expected Emerging Threats Pro rules file MD5: Emerging Threats Pro rules file download failed. Emerging Threats Pro rules will not be updated. The Rules update has finished. Time: 2017-12-04 06:30:06
I'd like to do some more troubleshooting. I'd like to check the timestamps and MD5 of the etpro.rules.tar.gz and the contents of etpro.rules.tar.gz.md5 and see what is going on. Can you tell me where the rules files are downloaded to on pfSense? Can you also confirm the download URL that I should be looking at for Suricata 4.0.1?
Greg
-
Bill,
I have been seeing the same issue since I installed pfSense 2.4.2 on 23 November. Until now, I haven't had any significant periods of failed updates in the three years I have been running pfSense and Suricata. I haven't made any changes to configuration in months.
I'd like to do some more troubleshooting. I'd like to check the timestamps and MD5 of the etpro.rules.tar.gz and the contents of etpro.rules.tar.gz.md5 and see what is going on. Can you tell me where the rules files are downloaded to on pfSense? Can you also confirm the download URL that I should be looking at for Suricata 4.0.1?
Greg
I have not touched the ET-Pro rules download URL since the Suricata package was created. I once was given an ET-Pro code for testing. I will need to see if it still works and use the access to check the directory structure. Could be some changes have happened with the new owners.
Bill
-
Bill,
Further troubleshooting at my end…
I found the relevant code that defines the URL for downloading files in the pfSense Suricata package.
The base download URL is defined in the variable ETPRO_BASE_DNLD_URL in the default config file as "https://rules.emergingthreatspro.com/".
In suricata_check_for_rule_updates.php this base URL is extended on line 71 by appending "{$etproid}/suricata/"
/* Set up Emerging Threats rules filenames and URL */ if ($etpro == "on") { $emergingthreats_filename = ETPRO_DNLD_FILENAME; $emergingthreats_filename_md5 = ETPRO_DNLD_FILENAME . ".md5"; $emergingthreats_url = ETPRO_BASE_DNLD_URL; ---> $emergingthreats_url .= "{$etproid}/suricata/"; $et_name = "Emerging Threats Pro"; $et_md5_remove = ET_DNLD_FILENAME . ".md5"; unlink_if_exists("{$suricatadir}{$et_md5_remove}"); } else {
For me, that complete URL https://rules.emergingthreatspro.com/{$etproid}/suricata/ where I substitute my Subscription code for {$etproid} is now an error page:
Emerging Threats Pro suricata-1.3-enhanced Sorry, I wasn't able to find your subscription to this service. Please contact support@emergingthreats.net for help. Name Last Modified Size
If I change line 71 of suricata_check_for_rule_updates.php to append "{$etproid}/suricata-4.0/" then everything starts updating correctly:
Starting rules update... Time: 2017-12-05 08:01:37 Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5... Checking Emerging Threats Pro rules md5 file... There is a new set of Emerging Threats Pro rules posted. Downloading file 'etpro.rules.tar.gz'... Done downloading rules file. Extracting and installing Emerging Threats Pro rules... Installation of Emerging Threats Pro rules completed. Copying new config and map files... Updating rules configuration for: LAN ... Live-Reload of updated rules is enabled... Live swap of updated rules requested for LAN. Live-Reload of the updated rules is complete. The Rules update has finished. Time: 2017-12-05 08:02:17
It looks like the root cause of the update problems is Emerging Threats reorganising their folder structure in the third week of November. The link I previously posted now states that updates should be retrieved using the following URL format:```
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/Hope that helps. Greg
-
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/
Yes, I can download again. :D
-
Bill,
Further troubleshooting at my end…
I found the relevant code that defines the URL for downloading files in the pfSense Suricata package.
The base download URL is defined in the variable ETPRO_BASE_DNLD_URL in the default config file as "https://rules.emergingthreatspro.com/".
In suricata_check_for_rule_updates.php this base URL is extended on line 71 by appending "{$etproid}/suricata/"
/* Set up Emerging Threats rules filenames and URL */ if ($etpro == "on") { $emergingthreats_filename = ETPRO_DNLD_FILENAME; $emergingthreats_filename_md5 = ETPRO_DNLD_FILENAME . ".md5"; $emergingthreats_url = ETPRO_BASE_DNLD_URL; ---> $emergingthreats_url .= "{$etproid}/suricata/"; $et_name = "Emerging Threats Pro"; $et_md5_remove = ET_DNLD_FILENAME . ".md5"; unlink_if_exists("{$suricatadir}{$et_md5_remove}"); } else {
For me, that complete URL https://rules.emergingthreatspro.com/{$etproid}/suricata/ where I substitute my Subscription code for {$etproid} is now an error page:
Emerging Threats Pro suricata-1.3-enhanced Sorry, I wasn't able to find your subscription to this service. Please contact support@emergingthreats.net for help. Name Last Modified Size
If I change line 71 of suricata_check_for_rule_updates.php to append "{$etproid}/suricata-4.0/" then everything starts updating correctly:
Starting rules update... Time: 2017-12-05 08:01:37 Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5... Checking Emerging Threats Pro rules md5 file... There is a new set of Emerging Threats Pro rules posted. Downloading file 'etpro.rules.tar.gz'... Done downloading rules file. Extracting and installing Emerging Threats Pro rules... Installation of Emerging Threats Pro rules completed. Copying new config and map files... Updating rules configuration for: LAN ... Live-Reload of updated rules is enabled... Live swap of updated rules requested for LAN. Live-Reload of the updated rules is complete. The Rules update has finished. Time: 2017-12-05 08:02:17
It looks like the root cause of the update problems is Emerging Threats reorganising their folder structure in the third week of November. The link I previously posted now states that updates should be retrieved using the following URL format:```
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/Hope that helps. Greg
Thanks for the research into this. I will need to update the URL used in the package.
Edit: I have submitted the patch for this to the pfSense for review and merging into production. Here is a link to the Github Pull Request: https://github.com/pfsense/FreeBSD-ports/pull/486. Look for a new Suricata package version to appear in the next few days. The only change in the new package is this ET-Pro and ET-Open rules URL update.
Bill
-
Bill, Thankyou for the quick response.
While appending suricata-4.0 seems to work, on closer inspection of the ET Mailing list entry I think it would be better to base the Rules URL on the full Suricata version number. They give the following examples:
Suricata 4.0: https://rules.emergingthreatspro.com/$oinkcode/suricata-4.0.0/ Suricata 3.2.3: https://rules.emergingthreatspro.com/$oinkcode/suricata-3.2.3/ Suricata 2.0.11: https://rules.emergingthreatspro.com/$oinkcode/suricata-2.0.11/
Reference: https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html
Perhaps a longer term fix is to append the current package version number to the URL?
Greg
-
Bill, Thankyou for the quick response.
While appending suricata-4.0 seems to work, on closer inspection of the ET Mailing list entry I think it would be better to base the Rules URL on the full Suricata version number. They give the following examples:
Suricata 4.0: https://rules.emergingthreatspro.com/$oinkcode/suricata-4.0.0/ Suricata 3.2.3: https://rules.emergingthreatspro.com/$oinkcode/suricata-3.2.3/ Suricata 2.0.11: https://rules.emergingthreatspro.com/$oinkcode/suricata-2.0.11/
Reference: https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html
Perhaps a longer term fix is to append the current package version number to the URL?
Greg
Thank you for the update and the link to the mailing list. I will look into this. For now, the issue should be fixed with the new package update released today.
Bill