Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Snort VRT Updates Stop Part Way Through

    IDS/IPS
    5
    21
    2.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      doktornotor Banned
      last edited by

      Noone will fix the downloads here; as noted in the GUI - wait and try later.

      1 Reply Last reply Reply Quote 0
      • A
        AliBaba0101
        last edited by

        Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          No, there is really no way to manually install them.

          Bill

          1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks
            last edited by

            @AliBaba0101:

            Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

            The internal code is also using https, so that should not make a difference.  The package tarballs are downloaded to a temporary subdirectory in the /tmp directory.  Do you have at least 100 MB of free space there?  That could be one issue causing the downloads to stall and fail.  Also, do you have another package like pfBlocker that may be blocking the download server by accident (false positive maybe)?

            I just checked my box here in the USA, and my Snort VRT and ET rules are downloading just fine.

            Bill

            1 Reply Last reply Reply Quote 0
            • D
              doktornotor Banned
              last edited by

              I see it failing as well in suricata…

              
Sep 14 00:46:01	php: suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules md5 download failed...
Sep 14 00:46:01	php: suricata_check_for_rule_updates.php: File 'snortrules-snapshot-2975.tar.gz.md5' download attempts: 4 ...
Sep 14 00:45:46	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:46	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:31	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:31	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:16	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:16	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:01	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:01	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none

              Also have certificate issues with packages.pfsense.org. WTF is going on…

              1 Reply Last reply Reply Quote 0
              • bmeeksB
                bmeeks
                last edited by

                Weird…just checked mine here in the USA in the Eastern Time Zone.  Here are the update log results--

                
Starting rules update...  Time: 2015-09-13 22:09:21
	Downloading Snort VRT rules md5 file snortrules-snapshot-2975.tar.gz.md5...
	Checking Snort VRT rules md5 file...
	Snort VRT rules are up to date.
	Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
	Checking Emerging Threats Open rules md5 file...
	Emerging Threats Open rules are up to date.
The Rules update has finished.  Time: 2015-09-13 22:09:22

                I did see something on Friday I think it was about an undersea fiber cable cut connecting the US with Europe.  Saw that on a Yahoo mail outage notice I believe it was.  You would think that would long since be bypassed and sorted out, though.

                Bill

                1 Reply Last reply Reply Quote 0
                • A
                  AliBaba0101
                  last edited by

                  @bmeeks:

                  @AliBaba0101:

                  Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

                  The internal code is also using https, so that should not make a difference.  The package tarballs are downloaded to a temporary subdirectory in the /tmp directory.  Do you have at least 100 MB of free space there?  That could be one issue causing the downloads to stall and fail.  Also, do you have another package like pfBlocker that may be blocking the download server by accident (false positive maybe)?

                  I just checked my box here in the USA, and my Snort VRT and ET rules are downloading just fine.

                  Bill

                  I just checked and definitely not a space issue.  It's mounted to the root and 19G available.  the largest directory right now is /tmp/suricata_rules_up at 16M

                  I have two different installations doing the same thing.  I setup one on KVM just to see if the results were any different.

                  Where are you pulling the detailed logs from?

                  1 Reply Last reply Reply Quote 0
                  • A
                    AliBaba0101
                    last edited by

                    Shouldn't I be able to do a curl or wget for the https snort vrt url?

                    I'm getting a 422 error

                    [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode"
                    –2015-09-13 22:47:53--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode
                    Resolving www.snort.org (www.snort.org)... 104.20.21.171, 104.20.19.171, 104.20.17.171, ...
                    Connecting to www.snort.org (www.snort.org)|104.20.21.171|:443... connected.
                    HTTP request sent, awaiting response... 422 Unprocessable Entity
                    2015-09-13 22:47:54 ERROR 422: Unprocessable Entity.

                    [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:

                    Funny thing is that URL clearly works in a web browser
                    when I try the same thing for the community url

                    [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget https://www.snort.org/rules/community
                    –2015-09-13 22:36:49--  https://www.snort.org/rules/community
                    Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.18.171, 104.20.20.171, ...
                    Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
                    HTTP request sent, awaiting response... 302 Found
                    Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0%3D [following]
                    –2015-09-13 22:36:49--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0=
                    Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.16.88
                    Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.16.88|:443... connected.
                    HTTP request sent, awaiting response... 200 OK
                    Length: 263839 (258K) [application/x-tar]
                    Saving to: 'community.1'

                    community.1        100%[=====================>] 257.66K  381KB/s  in 0.7s

                    2015-09-13 22:36:51 (381 KB/s) - 'community.1' saved [263839/263839]

                    [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:

                    1 Reply Last reply Reply Quote 0
                    • BBcan177B
                      BBcan177 Moderator
                      last edited by

                      Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                      You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…

                      Edit the file:  /usr/local/www/suricata/suricata_check_for_rule_updates.php

                      Line numbers 200 - 202

                      curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
                      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
                      curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);

                      and change them to the following:

                      curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
                      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                      curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

                      Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.

                      "Experience is something you don't get until just after you need it."

                      Website: http://pfBlockerNG.com
                      Twitter: @BBcan177  #pfBlockerNG
                      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                      1 Reply Last reply Reply Quote 0
                      • A
                        AliBaba0101
                        last edited by

                        I'm sitting here scratching my head. Now I get this, which is similar to what I see in the pfsense gui.  It starts the download and then just stops part way through for no reason.  I've tried this from multiple browsers and operating systems from the same network.  This is starting to look like a FreeBSD issue?

                        [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/tmp/vrt: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx"
                        –2015-09-13 23:46:09--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
                        Resolving www.snort.org (www.snort.org)... 104.20.20.171, 104.20.18.171, 104.20.19.171, ...
                        Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
                        HTTP request sent, awaiting response... 302 Found
                        Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68%2BI6xX26S1ruTGc%3D [following]
                        –2015-09-13 23:46:10--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68+I6xX26S1ruTGc=
                        Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.14.240
                        Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
                        HTTP request sent, awaiting response... 403 Forbidden
                        2015-09-13 23:46:10 ERROR 403: Forbidden.

                        --2015-09-13 23:46:10--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
                        Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
                        HTTP request sent, awaiting response... 302 Found
                        Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S%2FVF9vDFXzRanI%3D [following]
                        –2015-09-13 23:46:11--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=
                        Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
                        HTTP request sent, awaiting response... 200 OK
                        Length: 34081451 (33M) [application/octet-stream]
                        Saving to: 'snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx'

                        ar.gz?oinkcode=178e7359c  11%[===>                                ]  3.87M  283KB/s  eta 82s

                        when I try to download this direct url from s3 using a browser from the same ip pat, it works

                        https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=

                        when I try wget from pfsense it stops part way through

                        same thing from a Ubuntu 14.04 on the same network downloads just fine

                        agilani@ubuntu:~/Desktop$ wget -4 "https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX"
                        –2015-09-13 22:51:03--  https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX
                        Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.20.171, 104.20.21.171, ...
                        Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
                        HTTP request sent, awaiting response... 302 Found
                        Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D [following]
                        –2015-09-13 22:51:03--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D
                        Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
                        Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:443... connected.
                        HTTP request sent, awaiting response... 200 OK
                        Length: 33654635 (32M) [application/octet-stream]
                        Saving to: ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’

                        100%[======================================>] 33,654,635  6.94MB/s  in 7.8s

                        2015-09-13 22:51:12 (4.11 MB/s) - ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’ saved [33654635/33654635]

                        agilani@ubuntu:~/Desktop$

                        1 Reply Last reply Reply Quote 0
                        • A
                          AliBaba0101
                          last edited by

                          @BBcan177:

                          Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                          You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…

                          Edit the file:  /usr/local/www/suricata/suricata_check_for_rule_updates.php

                          Line numbers 200 - 202

                          curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
                          curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
                          curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);

                          and change them to the following:

                          curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
                          curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                          curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

                          Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.

                          looks like the file is in /usr/local/pkg/suricata instead - but not the droid I'm looking for.  The changes didn't make any difference.

                          1 Reply Last reply Reply Quote 0
                          • D
                            doktornotor Banned
                            last edited by

                            @AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.

                            @BBcan177:

                            Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                            Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.

                            1 Reply Last reply Reply Quote 0
                            • A
                              AliBaba0101
                              last edited by

                              @doktornotor:

                              @AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.

                              @BBcan177:

                              Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                              Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.

                              packetcapture.zip

                              1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks
                                last edited by

                                Just realized you guys having the issue are using the pfSense 2.2.5 snapshot.  I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems.  As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots.  That may be the root of all the Snort/Suricata rule download problems.  One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.

                                Bill

                                1 Reply Last reply Reply Quote 0
                                • D
                                  doktornotor Banned
                                  last edited by

                                  Yeah, I'm pretty much convinced it's not package or Snort website fault at all… Was working for some week or so, though. Then, it pretty much started to behave like if all the root CAs have expired or what. Gitsync couldn't fix it either, needed a new snapshot. Uh.  :o ???

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    AliBaba0101
                                    last edited by

                                    @bmeeks:

                                    Just realized you guys having the issue are using the pfSense 2.2.5 snapshot.  I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems.  As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots.  That may be the root of all the Snort/Suricata rule download problems.  One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.

                                    Bill

                                    I can reproduce the problem pretty reliably on 2.2.4 and 2.2.5 - I only upgraded form 2.2.4 to 2.2.5 to see if it behaved any differently.  I didn't have a lot of time to look at the packet capture I uploaded, but from the initial look - it appears the pfsense  box just stops sending tcp aknowledgements, then sends a whole bunch of duplicate acknowledgements and then doesnt' respond again and just drops the conversation altogether after a tcp rst.

                                    the funny thing is I can't reproduce this on my Ubuntu or windows boxes on the same network.  They all work fine and download it without any problems.

                                    I'm starting to wonder if the snort site is checking for a client agent and checking it for a valid signature….but then everyone else should be having the same problem.

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      dprince
                                      last edited by

                                      Anybody have any luck getting this to work lately?  I still can't dload VRT rules.  Tried the SSLv3 fix and a new oinkcode but no luck.  All the other rulesets work.

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        doktornotor Banned
                                        last edited by

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          dprince
                                          last edited by

                                          Thanks…my issue is either pfblockerng or one of the suricata blocking rules apparently.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post