Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Snort VRT Updates Stop Part Way Through

    Scheduled Pinned Locked Moved IDS/IPS
    21 Posts 5 Posters 3.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      AliBaba0101
      last edited by

      Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by

        No, there is really no way to manually install them.

        Bill

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          @AliBaba0101:

          Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

          The internal code is also using https, so that should not make a difference.  The package tarballs are downloaded to a temporary subdirectory in the /tmp directory.  Do you have at least 100 MB of free space there?  That could be one issue causing the downloads to stall and fail.  Also, do you have another package like pfBlocker that may be blocking the download server by accident (false positive maybe)?

          I just checked my box here in the USA, and my Snort VRT and ET rules are downloading just fine.

          Bill

          1 Reply Last reply Reply Quote 0
          • D
            doktornotor Banned
            last edited by

            I see it failing as well in suricata…

            
Sep 14 00:46:01	php: suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules md5 download failed...
Sep 14 00:46:01	php: suricata_check_for_rule_updates.php: File 'snortrules-snapshot-2975.tar.gz.md5' download attempts: 4 ...
Sep 14 00:45:46	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:46	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:31	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:31	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:16	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:16	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none
Sep 14 00:45:01	php: suricata_check_for_rule_updates.php: [Suricata] Will retry in 15 seconds...
Sep 14 00:45:01	php: suricata_check_for_rule_updates.php: [Suricata] Rules download error: error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none

            Also have certificate issues with packages.pfsense.org. WTF is going on…

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by

              Weird…just checked mine here in the USA in the Eastern Time Zone.  Here are the update log results--

              
Starting rules update...  Time: 2015-09-13 22:09:21
	Downloading Snort VRT rules md5 file snortrules-snapshot-2975.tar.gz.md5...
	Checking Snort VRT rules md5 file...
	Snort VRT rules are up to date.
	Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
	Checking Emerging Threats Open rules md5 file...
	Emerging Threats Open rules are up to date.
The Rules update has finished.  Time: 2015-09-13 22:09:22

              I did see something on Friday I think it was about an undersea fiber cable cut connecting the US with Europe.  Saw that on a Yahoo mail outage notice I believe it was.  You would think that would long since be bypassed and sorted out, though.

              Bill

              1 Reply Last reply Reply Quote 0
              • A
                AliBaba0101
                last edited by

                @bmeeks:

                @AliBaba0101:

                Thanks for the feedback, is there any way to download and install them manually?  I couldn't find any references on how to do it.  I took less than 20 seconds to download with my oink code using https.

                The internal code is also using https, so that should not make a difference.  The package tarballs are downloaded to a temporary subdirectory in the /tmp directory.  Do you have at least 100 MB of free space there?  That could be one issue causing the downloads to stall and fail.  Also, do you have another package like pfBlocker that may be blocking the download server by accident (false positive maybe)?

                I just checked my box here in the USA, and my Snort VRT and ET rules are downloading just fine.

                Bill

                I just checked and definitely not a space issue.  It's mounted to the root and 19G available.  the largest directory right now is /tmp/suricata_rules_up at 16M

                I have two different installations doing the same thing.  I setup one on KVM just to see if the results were any different.

                Where are you pulling the detailed logs from?

                1 Reply Last reply Reply Quote 0
                • A
                  AliBaba0101
                  last edited by

                  Shouldn't I be able to do a curl or wget for the https snort vrt url?

                  I'm getting a 422 error

                  [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode"
                  –2015-09-13 22:47:53--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=myoinkcode
                  Resolving www.snort.org (www.snort.org)... 104.20.21.171, 104.20.19.171, 104.20.17.171, ...
                  Connecting to www.snort.org (www.snort.org)|104.20.21.171|:443... connected.
                  HTTP request sent, awaiting response... 422 Unprocessable Entity
                  2015-09-13 22:47:54 ERROR 422: Unprocessable Entity.

                  [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:

                  Funny thing is that URL clearly works in a web browser
                  when I try the same thing for the community url

                  [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root: wget https://www.snort.org/rules/community
                  –2015-09-13 22:36:49--  https://www.snort.org/rules/community
                  Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.18.171, 104.20.20.171, ...
                  Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
                  HTTP request sent, awaiting response... 302 Found
                  Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0%3D [following]
                  –2015-09-13 22:36:49--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/455/original/community-rules.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442205409&Signature=2F9wV2KvD6BwZLlbkaLaJqLNcX0=
                  Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.16.88
                  Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.16.88|:443... connected.
                  HTTP request sent, awaiting response... 200 OK
                  Length: 263839 (258K) [application/x-tar]
                  Saving to: 'community.1'

                  community.1        100%[=====================>] 257.66K  381KB/s  in 0.7s

                  2015-09-13 22:36:51 (381 KB/s) - 'community.1' saved [263839/263839]

                  [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/root:

                  1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator
                    last edited by

                    Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                    You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…

                    Edit the file:  /usr/local/www/suricata/suricata_check_for_rule_updates.php

                    Line numbers 200 - 202

                    curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
                    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
                    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);

                    and change them to the following:

                    curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
                    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

                    Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    1 Reply Last reply Reply Quote 0
                    • A
                      AliBaba0101
                      last edited by

                      I'm sitting here scratching my head. Now I get this, which is similar to what I see in the pfsense gui.  It starts the download and then just stops part way through for no reason.  I've tried this from multiple browsers and operating systems from the same network.  This is starting to look like a FreeBSD issue?

                      [2.2.5-DEVELOPMENT][admin@pfSense.localdomain]/tmp/vrt: wget "https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx"
                      –2015-09-13 23:46:09--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
                      Resolving www.snort.org (www.snort.org)... 104.20.20.171, 104.20.18.171, 104.20.19.171, ...
                      Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
                      HTTP request sent, awaiting response... 302 Found
                      Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68%2BI6xX26S1ruTGc%3D [following]
                      –2015-09-13 23:46:10--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209570&Signature=bLs08YpUJhB68+I6xX26S1ruTGc=
                      Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.14.240
                      Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
                      HTTP request sent, awaiting response... 403 Forbidden
                      2015-09-13 23:46:10 ERROR 403: Forbidden.

                      --2015-09-13 23:46:10--  https://www.snort.org/rules/snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx
                      Connecting to www.snort.org (www.snort.org)|104.20.20.171|:443... connected.
                      HTTP request sent, awaiting response... 302 Found
                      Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S%2FVF9vDFXzRanI%3D [following]
                      –2015-09-13 23:46:11--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=
                      Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.14.240|:443... connected.
                      HTTP request sent, awaiting response... 200 OK
                      Length: 34081451 (33M) [application/octet-stream]
                      Saving to: 'snortrules-snapshot-2975.tar.gz?oinkcode=xxxxx'

                      ar.gz?oinkcode=178e7359c  11%[===>                                ]  3.87M  283KB/s  eta 82s

                      when I try to download this direct url from s3 using a browser from the same ip pat, it works

                      https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/445/original/snortrules-snapshot-2975.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442209571&Signature=WdJQ2RTN19Jj0S/VF9vDFXzRanI=

                      when I try wget from pfsense it stops part way through

                      same thing from a Ubuntu 14.04 on the same network downloads just fine

                      agilani@ubuntu:~/Desktop$ wget -4 "https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX"
                      –2015-09-13 22:51:03--  https://www.snort.org/rules/snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX
                      Resolving www.snort.org (www.snort.org)... 104.20.17.171, 104.20.20.171, 104.20.21.171, ...
                      Connecting to www.snort.org (www.snort.org)|104.20.17.171|:443... connected.
                      HTTP request sent, awaiting response... 302 Found
                      Location: https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D [following]
                      –2015-09-13 22:51:03--  https://s3.amazonaws.com/snort-org-site/production/release_files/files/000/002/443/original/snortrules-snapshot-2962.tar.gz?AWSAccessKeyId=AKIAIXACIED2SPMSC7GA&Expires=1442213463&Signature=GzfIOg0wkc4ODvW4JEmEIGa%2FbPc%3D
                      Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
                      Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:443... connected.
                      HTTP request sent, awaiting response... 200 OK
                      Length: 33654635 (32M) [application/octet-stream]
                      Saving to: ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’

                      100%[======================================>] 33,654,635  6.94MB/s  in 7.8s

                      2015-09-13 22:51:12 (4.11 MB/s) - ‘snortrules-snapshot-2962.tar.gz?oinkcode=XXXXXX’ saved [33654635/33654635]

                      agilani@ubuntu:~/Desktop$

                      1 Reply Last reply Reply Quote 0
                      • A
                        AliBaba0101
                        last edited by

                        @BBcan177:

                        Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                        You can try to manually lower the cURL settings to see if that is the issue. Then you can try to resolve the underlying issue…

                        Edit the file:  /usr/local/www/suricata/suricata_check_for_rule_updates.php

                        Line numbers 200 - 202

                        curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1");
                        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
                        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, true);

                        and change them to the following:

                        curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "TLSv1.2, TLSv1, SSLv3");
                        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

                        Then try the rules update and see if that works.. You could also try changing one line at a time and see which setting could be the issue. You definetly do not want to leave it with lower SSL settings for long.

                        looks like the file is in /usr/local/pkg/suricata instead - but not the droid I'm looking for.  The changes didn't make any difference.

                        1 Reply Last reply Reply Quote 0
                        • D
                          doktornotor Banned
                          last edited by

                          @AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.

                          @BBcan177:

                          Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                          Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.

                          1 Reply Last reply Reply Quote 0
                          • A
                            AliBaba0101
                            last edited by

                            @doktornotor:

                            @AliBaba0101: Sounds like you managed to get temporarily banned with your hammering of the servers.

                            @BBcan177:

                            Snort/Suricata is now using more secure cURL SSL options to download the Rule Updates. Maybe something is failing with your SSL connection?

                            Yeah, something was failing the CA certs bundle, or dunno… I reinstalled latest 2.2.5 snapshot and it works. WTF.

                            packetcapture.zip

                            1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks
                              last edited by

                              Just realized you guys having the issue are using the pfSense 2.2.5 snapshot.  I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems.  As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots.  That may be the root of all the Snort/Suricata rule download problems.  One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.

                              Bill

                              1 Reply Last reply Reply Quote 0
                              • D
                                doktornotor Banned
                                last edited by

                                Yeah, I'm pretty much convinced it's not package or Snort website fault at all… Was working for some week or so, though. Then, it pretty much started to behave like if all the root CAs have expired or what. Gitsync couldn't fix it either, needed a new snapshot. Uh.  :o ???

                                1 Reply Last reply Reply Quote 0
                                • A
                                  AliBaba0101
                                  last edited by

                                  @bmeeks:

                                  Just realized you guys having the issue are using the pfSense 2.2.5 snapshot.  I am still on 2.2.4 for my box, so that's likely why I'm not seeing problems.  As @doktornotor noted, there did appear to be some kind CA problem in one of the 2.2.5 snapshots.  That may be the root of all the Snort/Suricata rule download problems.  One of the recent updates for both Snort and Suricata upgraded the rules download to be more secure by using HTTPS and tightening up on the options supplied to cURL.

                                  Bill

                                  I can reproduce the problem pretty reliably on 2.2.4 and 2.2.5 - I only upgraded form 2.2.4 to 2.2.5 to see if it behaved any differently.  I didn't have a lot of time to look at the packet capture I uploaded, but from the initial look - it appears the pfsense  box just stops sending tcp aknowledgements, then sends a whole bunch of duplicate acknowledgements and then doesnt' respond again and just drops the conversation altogether after a tcp rst.

                                  the funny thing is I can't reproduce this on my Ubuntu or windows boxes on the same network.  They all work fine and download it without any problems.

                                  I'm starting to wonder if the snort site is checking for a client agent and checking it for a valid signature….but then everyone else should be having the same problem.

                                  1 Reply Last reply Reply Quote 0
                                  • D
                                    dprince
                                    last edited by

                                    Anybody have any luck getting this to work lately?  I still can't dload VRT rules.  Tried the SSLv3 fix and a new oinkcode but no luck.  All the other rulesets work.

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      doktornotor Banned
                                      last edited by

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        dprince
                                        last edited by

                                        Thanks…my issue is either pfblockerng or one of the suricata blocking rules apparently.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.