Emerging Threats Open rules file download failed. Bad MD5 checksum.

[fellymar]

Dear All,
searching on internet I found some post regarding bad MD5 checksum but I still cannot solve.

Few months ago I tried to install snort on a new pfSense (vmware) but I had problems downloading Emergency Threats Open rules.

I'm trying again now but same problem, I tried on the old pfSense and on a new installation (also vmware)

Please see below log:

Starting rules update... Time: 2020-07-03 11:19:21
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29160.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29160.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID RULES detectors md5 file appid_rules.tar.gz.md5...
Checking Snort OpenAppID RULES detectors md5 file...
There is a new set of Snort OpenAppID RULES detectors posted.
Downloading file 'appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Emerging Threats Open rules file download failed. Bad MD5 checksum.
Downloaded Emerging Threats Open rules file MD5: 3262041dd41539e8e904b2d664db10d4
Expected Emerging Threats Open rules file MD5: 5e2eb6ae63574f6c258431196b71beb9
Emerging Threats Open rules file download failed. Emerging Threats Open rules will not be updated.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Extracting and installing Snort OpenAppID detectors...
Installation of Snort OpenAppID detectors completed.
Extracting and installing Snort OpenAppID detectors...
Installation of Snort OpenAppID detectors completed.
Extracting and installing Snort GPLv2 Community Rules...
Installation of Snort GPLv2 Community Rules completed.
Copying new config and map files...
Warning: No interfaces configured for Snort were found...
The Rules update has finished. Time: 2020-07-03 11:21:13

I'm not using RAM drive, I tried with auto partition and manual partition during the installation but same problem.

Hope someone can help me.

Many thanks.

Kind regards,
Mario

[DaddyGo]

@fellymar said in Emerging Threats Open rules file download failed. Bad MD5 checksum.:

Emerging Threats Open rules

hi,
download issues... hmmm???
did you read this:
https://forum.netgate.com/topic/123682/emerging-threats-pro-rules-file-download-failed-bad-md5-checksum

(it's PRO, but there's no difference)
if I'm on the wrong path @bmeeks
Bill can definitely provide more up-to-date information

[fellymar]

Hi,
download issue?? Could be but connection is working, I updated pfsense, packages and all other snort updates are downloaded without problem.

I downloaded from my pc the file from https://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz and MD5 is 0B3F464A8D091E2C928BC29B54FB7058
I downloaded also the file https://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz.md5 to compare and MD5 string inside is same 0b3f464a8d091e2c928bc29b54fb7058

From pfsense snort log md5 are completely different:
Downloaded Emerging Threats Open rules file MD5: 3262041dd41539e8e904b2d664db10d4
Expected Emerging Threats Open rules file MD5: 5e2eb6ae63574f6c258431196b71beb9

[DaddyGo]

@fellymar said in Emerging Threats Open rules file download failed. Bad MD5 checksum.:

https://rules.emergingthreats.net/open/snort-2.9.0/

with me , that's what it looks like (made now):

Starting rules update... Time: 2020-07-03 13:21:44
Downloading Emerging Threats Open rules md5 file...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Downloading Snort VRT rules md5 file...
Checking Snort VRT rules md5 file...
There is a new set of Snort rules posted.
Downloading file 'snortrules-snapshot-29160.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Extracting and installing Snort rules...
Installation of Snort rules completed.
Extracting and installing Snort GPLv2 Community Rules...
Installation of Snort GPLv2 Community Rules completed.
Copying new config and map files...
Updating rules configuration for: VPNPT ...
Updating rules configuration for: LAN ...
Live-Reload of updated rules is enabled...
Live swap of updated rules requested for VPNPT.
Live swap of updated rules requested for LAN.
Live-Reload of the updated rules is complete.
The Rules update has finished. Time: 2020-07-03 13:22:29

[bmeeks]

@fellymar said in Emerging Threats Open rules file download failed. Bad MD5 checksum.:

Hi,
download issue?? Could be but connection is working, I updated pfsense, packages and all other snort updates are downloaded without problem.

I downloaded from my pc the file from https://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz and MD5 is 0B3F464A8D091E2C928BC29B54FB7058
I downloaded also the file https://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz.md5 to compare and MD5 string inside is same 0b3f464a8d091e2c928bc29b54fb7058

From pfsense snort log md5 are completely different:
Downloaded Emerging Threats Open rules file MD5: 3262041dd41539e8e904b2d664db10d4
Expected Emerging Threats Open rules file MD5: 5e2eb6ae63574f6c258431196b71beb9

Do you have RAM disks configured for /tmp? If so, it is possible the /tmp directory is not large enough. The IDS/IPS packages need at least 256 MB of free space in /tmp to download and extract the rules packages. And the free disk space you see on the pfSense Dashboard is misleading because after a rules download Snort and Suricata both clean up behind themselves and erase what they downloaded, even when the rules update fails. So the free disk showing is BEFORE the rules download occurs. You would need to look in the pfSense system log to see any actual out-of-space errors.

The most common cause of a problem like you are seeing is a disk space issue (usually caused by having a RAM disk defined). The Emerging Threats file is the last one downloaded, so a disk space issue is most likely to hit it because the downloaded and extracted Snort and Snort GPLv2 files have already eaten up some space.

And by the way, you don't need to use the Snort GPLv2 Community Rules when you have an Oinkcode and are downloading the Snort Subscriber Rules. All of the Community Rules are already in the Snort Subscriber Rules. The GPLv2 Community Rules are for users who do not have an Oinkcode. They are a subset of the Snort Subscriber Rules.

[fellymar]

No, I do not have RAM disk, checking system log:

Jul 3 15:44:23 php-cgi snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules file update downloaded successfully
Jul 3 15:44:24 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Jul 3 15:44:24 php-cgi snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Jul 3 15:44:24 php-cgi snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file download failed. Bad MD5 checksum...
Jul 3 15:44:24 php-cgi snort_check_for_rule_updates.php: [Snort] Downloaded File MD5: 3262041dd41539e8e904b2d664db10d4
Jul 3 15:44:24 php-cgi snort_check_for_rule_updates.php: [Snort] Expected File MD5: 5e2eb6ae63574f6c258431196b71beb9
Jul 3 15:44:32 php-cgi snort_check_for_rule_updates.php: [Snort] The Rules update has finished.

I tried also to enable only Snort Subscriber Ruleset and Emerging Threats Open Rules but same error, below log:

Starting rules update... Time: 2020-07-03 15:43:12
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29160.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29160.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Emerging Threats Open rules file download failed. Bad MD5 checksum.
Downloaded Emerging Threats Open rules file MD5: 3262041dd41539e8e904b2d664db10d4
Expected Emerging Threats Open rules file MD5: 5e2eb6ae63574f6c258431196b71beb9
Emerging Threats Open rules file download failed. Emerging Threats Open rules will not be updated.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Copying new config and map files...
Warning: No interfaces configured for Snort were found...
The Rules update has finished. Time: 2020-07-03 15:44:32

[bmeeks]

@fellymar :

The Snort Community GPLv2 rules I mentioned have zero to do with your issue. I simply mentioned that at the end of my post to let you know you do not need them when you have the Snort rules enabled.

Do you have any other packages installed on your firewall? Something like Squid maybe? I'm wondering if something is sending Snort a cached version of a corrupt file.

That error message means the file that Snort received (the ET Open rules archive tarball) did not generate the same MD5 checksum when one was calculated locally after downloading as the one in the MD5 file. The logic works this way-

1. The code downloads the MD5 file from the remote site. This is a short text file containing the MD5 checksum hash of the matching rules tarball (the much larger gzipped file).

2. Next the code downloads the actual rules tarball gzip file.

3. Now it uses the local md5sum utility in FreeBSD to calculate the MD5 hash of the tarball gzip it just downloaded. It then compares the calculated hash to the one it reads from the previously downloaded MD5 file. If they do not match, it assumes the gzip tarball is invalid so it does not use it. Your error message in the log shows the MD5 values are not matching, and so the code is properly abandoning updating those rules because one of the downloaded files is wrong (either the MD5 checksum file or the actual gzip rules tarball archive).

So in your case, it is one of two possibilities. Either something in your firewall setup has cached an older copy of the MD5 or an older copy of the tarball gzip archive and sending that to pfSense -- or I guess it is possible that the ET rules files you are downloading do not match up at the source. This could happen if the ET-Open rules were hosted on some type of CDN and one of the nodes someplace had a bad copy of one of the files ... ???

How large is your /tmp directory?

[kiwibenis]

@bmeeks
Thanks! I had 100M /tmp tmpfs and run exactly into this issue.
After increasing /tmp to 400M it worked!