Snort VRT Updates Stop Part Way Through
-
Weird…just checked mine here in the USA in the Eastern Time Zone. Here are the update log results--
Starting rules update... Time: 2015-09-13 22:09:21 Downloading Snort VRT rules md5 file snortrules-snapshot-2975.tar.gz.md5... Checking Snort VRT rules md5 file... Snort VRT rules are up to date. Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5... Checking Emerging Threats Open rules md5 file... Emerging Threats Open rules are up to date. The Rules update has finished. Time: 2015-09-13 22:09:22I did see something on Friday I think it was about an undersea fiber cable cut connecting the US with Europe. Saw that on a Yahoo mail outage notice I believe it was. You would think that would long since be bypassed and sorted out, though.
Bill
-
Thanks for the feedback, is there any way to download and install them manually? I couldn't find any references on how to do it. I took less than 20 seconds to download with my oink code using https.
The internal code is also using https, so that should not make a difference. The package tarballs are downloaded to a temporary subdirectory in the /tmp directory. Do you have at least 100 MB of free space there? That could be one issue causing the downloads to stall and fail. Also, do you have another package like pfBlocker that may be blocking the download server by accident (false positive maybe)?
I just checked my box here in the USA, and my Snort VRT and ET rules are downloading just fine.
Bill
I just checked and definitely not a space issue. It's mounted to the root and 19G available. the largest directory right now is /tmp/suricata_rules_up at 16M
I have two different installations doing the same thing. I setup one on KVM just to see if the results were any different.
Where are you pulling the detailed logs from?
-
Shouldn't I be able to do a curl or wget for the https snort vrt url?
I'm getting a 422 error
[2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode"
–2015-09-13 22:47:53-- https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode
Resolving www.snort.org (www.snort.org)... 104.20.21.171, 104.20.19.171, 104.20.17.171, ...
Connecting to www.snort.org (www.snort.org)|104.20.21.171|:443... connected.
HTTP request sent, awaiting response... 422 Unprocessable Entity
2015-09-13 22:47:54 ERROR 422: Unprocessable Entity.[2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:
Funny thing is that URL clearly works in a web browser
when I try the same thing for the community url[2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget https://www.snort.org/rules/community
–2015-09-13 22:36:49-- https://www.snort.org/rules/community
Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.18.171, 104.20.20.171, ...
Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0%3D [following]
–2015-09-13 22:36:49-- https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0=
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.16.88
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.16.88|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 263839 (258K) [application/x-tar]
Saving to: 'community.1'community.1 100%[=====================>] 257.66K 381KB/s in 0.7s
2015-09-13 22:36:51 (381 KB/s) - 'community.1' saved [263839/263839]
[2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:
-
Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?
You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…
Edit the file: /usr/local/www/suricata/suricata_check_for_rule_updates.php
Line numbers 200 - 202
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);and change them to the following:
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.
-
I'm sitting here scratching my head. Now I get this, which is similar to what I see in the pfsense gui. It starts the download and then just stops part way through for no reason. I've tried this from multiple browsers and operating systems from the same network. This is starting to look like a FreeBSD issue?
[2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/tmp/vrt: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx"
–2015-09-13 23:46:09-- https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
Resolving www.snort.org (www.snort.org)... 104.20.20.171, 104.20.18.171, 104.20.19.171, ...
Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68%2BI6xX26S1ruTGc%3D [following]
–2015-09-13 23:46:10-- https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68+I6xX26S1ruTGc=
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.14.240
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2015-09-13 23:46:10 ERROR 403: Forbidden.--2015-09-13 23:46:10-- https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S%2FVF9vDFXzRanI%3D [following]
–2015-09-13 23:46:11-- https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34081451 (33M) [application/octet-stream]
Saving to: 'snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx'ar.gz?oinkcode=178e7359c 11%[===> ] 3.87M 283KB/s eta 82s
when I try to download this direct url from s3 using a browser from the same ip pat, it works
https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=
when I try wget from pfsense it stops part way through
same thing from a Ubuntu 14.04 on the same network downloads just fine
agilani@ubuntu:~/Desktop$ wget -4 "https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX"
–2015-09-13 22:51:03-- https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX
Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.20.171, 104.20.21.171, ...
Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D [following]
–2015-09-13 22:51:03-- https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33654635 (32M) [application/octet-stream]
Saving to: ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’100%[======================================>] 33,654,635 6.94MB/s in 7.8s
2015-09-13 22:51:12 (4.11 MB/s) - ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’ saved [33654635/33654635]
agilani@ubuntu:~/Desktop$
-
Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?
You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…
Edit the file: /usr/local/www/suricata/suricata_check_for_rule_updates.php
Line numbers 200 - 202
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);and change them to the following:
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.
looks like the file is in /usr/local/pkg/suricata instead - but not the droid I'm looking for. The changes didn't make any difference.
-
@AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.
Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?
Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.
-
@AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.
Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?
Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.
-
Just realized you guys having the issue are using the pfSense 2.2.5 snapshot. I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems. As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots. That may be the root of all the Snort/Suricata rule download problems. One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.
Bill
-
Yeah, I'm pretty much convinced it's not package or Snort website fault at all… Was working for some week or so, though. Then, it pretty much started to behave like if all the root CAs have expired or what. Gitsync couldn't fix it either, needed a new snapshot. Uh. :o ???
-
Just realized you guys having the issue are using the pfSense 2.2.5 snapshot. I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems. As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots. That may be the root of all the Snort/Suricata rule download problems. One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.
Bill
I can reproduce the problem pretty reliably on 2.2.4 and 2.2.5 - I only upgraded form 2.2.4 to 2.2.5 to see if it behaved any differently. I didn't have a lot of time to look at the packet capture I uploaded, but from the initial look - it appears the pfsense box just stops sending tcp aknowledgements, then sends a whole bunch of duplicate acknowledgements and then doesnt' respond again and just drops the conversation altogether after a tcp rst.
the funny thing is I can't reproduce this on my Ubuntu or windows boxes on the same network. They all work fine and download it without any problems.
I'm starting to wonder if the snort site is checking for a client agent and checking it for a valid signature….but then everyone else should be having the same problem.
-
Anybody have any luck getting this to work lately? I still can't dload VRT rules. Tried the SSLv3 fix and a new oinkcode but no luck. All the other rulesets work.
-
-
Thanks…my issue is either pfblockerng or one of the suricata blocking rules apparently.
