[solved] Snort Registered User rules download fails
-
I have snort running and it updates community rules, etc but the registered user rules keep failing
Starting rules update... Time: 2019-02-16 00:05:00
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29120.tar.gz.md5...
Snort Subscriber rules md5 download failed.
Server returned error code 500.
Server error message was: 500 Internal Server Error
Snort Subscriber rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID RULES detectors md5 file appid_rules.tar.gz.md5...
Checking Snort OpenAppID RULES detectors md5 file...
There is a new set of Snort OpenAppID RULES detectors posted.
Downloading file 'appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file. -
Works fine here, give it another go.
Server returned error code 500.
Server error message was: 500 Internal Server Error
Snort Subscriber rules will not be updated.Starting rules update... Time: 2019-02-16 08:47:31
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29120.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29120.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID RULES detectors md5 file appid_rules.tar.gz.md5...
Checking Snort OpenAppID RULES detectors md5 file...
There is a new set of Snort OpenAppID RULES detectors posted.
Downloading file 'appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Extracting and installing Snort OpenAppID detectors...
Installation of Snort OpenAppID detectors completed.
Extracting and installing Snort OpenAppID detectors...
Installation of Snort OpenAppID detectors completed.
Extracting and installing Snort GPLv2 Community Rules...
Installation of Snort GPLv2 Community Rules completed.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
Updating rules configuration for: LAN ...
Updating rules configuration for: USER ...
Updating rules configuration for: GUEST ...
Updating rules configuration for: IOT ...
Updating rules configuration for: DMZ ...
Updating rules configuration for: VOICE ...
Updating rules configuration for: TEST ...
Updating rules configuration for: NORD ...
Updating rules configuration for: SECURE ...
Restarting Snort to activate the new set of rules...
Snort has restarted with your new set of rules.
The Rules update has finished. Time: 2019-02-16 08:49:21 -
@gwaitsi said in Snort Registered User rules download fails:
I have snort running and it updates community rules, etc but the registered user rules keep failing
Starting rules update... Time: 2019-02-16 00:05:00
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29120.tar.gz.md5...
Snort Subscriber rules md5 download failed.
Server returned error code 500.
Server error message was: 500 Internal Server Error
Snort Subscriber rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID RULES detectors md5 file appid_rules.tar.gz.md5...
Checking Snort OpenAppID RULES detectors md5 file...
There is a new set of Snort OpenAppID RULES detectors posted.
Downloading file 'appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file.This error happens for some users from time to time. Trying again will pretty much always succeed. It is not a problem with the pfSense package. It has something to do with the AWS site where the Snort team hosts their rules packages. My personal suspicion is that it is probably a timing/replication issue when they update the files. The MD5 file is a calculated checksum of the posted rules archive. That means two files have to get posted and then replicated to all of the servers that make up whatever kind of CDN they are using. Those two files are the actual gzip archive of new rules and the MD5 checksum file used to validate the integrity of the new rules archive. I'm thinking there is sometimes a delay in propagating the new MD5 file as that is the file that most often triggers the "500" error.
Try moving your rules update time to something a bit later than midnight. For example, I used 0130 U.S. Eastern Time and have never seen this error since changing my rules update time.
-
Afraid i am still getting bad checksum, but only on the subscriber rules. others are working fine
Starting rules update... Time: 2019-02-20 00:05:00
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29120.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29120.tar.gz'...
Done downloading rules file.
Snort Subscriber rules file download failed. Bad MD5 checksum.
Downloaded Snort Subscriber rules file MD5: ef8d4ba392d098f1e37a34b95d68d143
Expected Snort Subscriber rules file MD5: 68c0c20030c213ef6b3c95ffd3d95e0a
Snort Subscriber rules file download failed. Snort Subscriber rules will -
@gwaitsi
I know this is not what you would like to hear, but the problem has to be on your end of the connection. This is working for everyone else so far as I know. My own rules are updating just fine. Where are you located? Your ISP and geolocation may be resulting in you getting routed to an AWS server with a bad copy of that rules file. Snort rules are hosted on Amazon Web Services infrastructure.Can you go to the Snort.org web site and manually download the rules from there to your PC successfully?
The rules download code is very simple. It first downloads the MD5 checksum file. That file is a very small text file whose content is the MD5 checksum hash of the larger Snort Subscriber Rules tarball. It them compares that MD5 value to the value contained in the last MD5 file your firewall downloaded (in other words, the copy sitting in /usr/local/etc/snort). If the MD5 values do not match, then it downloads the newly posted Snort Subscriber Rules tarball. After downloading that tarball to your firewall in the /tmp directory, it calculates the MD5 checksum of the file it just downloaded. If the calculated value from the downloaded rules tarball does not match the posted MD5 value (downloaded from the Snort site), the code assumes the downloaded tarball is corrupt and thus it prints the error and skips updating those rules.
So in your case either the download of the tarball is actually getting corrupted, or you have some other issue whereby your IP is getting pointed to an older copy of the MD5 checkum file. Are you using any kind of proxy or caching system? If so, make sure any cache is cleaned out.
-
This post is deleted! -
@bmeeks so this is weird, if i look in the system logs i am seeing
Feb 22 00:59:12 kernel pid 18885 (pfctl), uid 0 inumber 30 on /tmp: filesystem full
Feb 22 00:59:19 php-cgi snort_check_for_rule_updates.php: [Snort] Rules download error: Failed writing body (0 != 1122)
Feb 22 00:59:19 kernel pid 66153 (php-cgi), uid 0 inumber 4233 on /tmp: filesystem fullhowever, system information shows me
/tmp 2% of 38MiB - ufs in RAM -
https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#ram-disk-sizes
-
@gwaitsi
@gwaitsi said in Snort Registered User rules download fails:
@bmeeks so this is weird, if i look in the system logs i am seeing
Feb 22 00:59:12 kernel pid 18885 (pfctl), uid 0 inumber 30 on /tmp: filesystem full
Feb 22 00:59:19 php-cgi snort_check_for_rule_updates.php: [Snort] Rules download error: Failed writing body (0 != 1122)
Feb 22 00:59:19 kernel pid 66153 (php-cgi), uid 0 inumber 4233 on /tmp: filesystem fullhowever, system information shows me
/tmp 2% of 38MiB - ufs in RAMThis would have been a very valuable piece of information to include with your original post! I've posted on this forum more times than I can count for folks to NEVER use a RAM disk with Snort or Suricata. It usually causes nothing but problems. Your's is the number one problem -- running out of disk space during a rules download. I should have asked about the RAM disk first off. It came to my mind, but I said maybe folks are finally starting to stop using RAM disks what with today's highly reliable SSDs; so I failed to ask.
Either ditch the RAM disk entirely (highly recommended) or else bump up the size for /tmp to at least 256 MB and potentially even 512 MB. Snort needs lots of space on /tmp to download and extract the rules tarball. When it finishes, it cleans up behind itself. That's why the Dashboard is not showing the space used. Snort cleaned up after the failure.
-
sorry dude, only just noticed the error. Set the ram size to 512mg running like a charm. thanks so much
-
@gwaitsi said in Snort Registered User rules download fails:
sorry dude, only just noticed the error. Set the ram size to 512mg running like a charm. thanks so much
I still suggest you ditch the RAM disk. That technology was useful back with NanoBSD and early Flash Memory cards. Today's Solid State Disks are plenty reliable. You will run out of other space as well at some point (like logging, potentially). Your RAM would serve you much better if it's available for use by the Snort process and other parts of pfSense as memory and not disk space.
-
@bmeeks thanks anything, but i will probably see how it goes first. I 512Mb for /tmp and 256Mb for /var at 13% with a 4G system and 64Gb SSD. As it is only for a small home network of a few PCs, mobiles and media boxes, the remaining 3Gb should be more than enough i guess.
-
I've been battling this as well. Be sure the Oinkcode is correct and without a leading space. Rookie mistake but it happens, drove me crazy for a week. Good luck!
