Error 302: Snort xyz file download failed... server returned error '302'

demux

Hi,

I'm on the latest versions of all and I'm seeing the following errors:

Downloading Snort Subscriber rules md5 file... done.
Checking Snort Subscriber rules md5 file... done.
There is a new set of Snort Subscriber rules posted.
Downloading snortrules-snapshot-29161.tar.gz... FAILED!
Snort Subscriber rules file download failed... server returned error '302'.
Snort Subscriber rules will not be updated.
Downloading Snort OpenAppID detectors md5 file... done.
Checking Snort OpenAppID detectors md5 file... done.
There is a new set of Snort OpenAppID detectors posted.
Downloading snort-openappid.tar.gz... FAILED!
Snort OpenAppID detectors file download failed... server returned error '302'.
Snort OpenAppID detectors will not be updated.
Downloading Snort AppID Open Text Rules md5 file... done.
Checking Snort AppID Open Text Rules md5 file... done.
There is a new set of Snort AppID Open Text Rules posted.
Downloading appid_rules.tar.gz... done.
Downloading Snort GPLv2 Community Rules md5 file... done.
Checking Snort GPLv2 Community Rules md5 file... done.
There is a new set of Snort GPLv2 Community Rules posted.
Downloading community-rules.tar.gz... FAILED!
Snort GPLv2 Community Rules file download failed... server returned error '302'.
Snort GPLv2 Community Rules will not be updated.

The same with Suricata.
What I saw is:

Nov 5 19:50:32 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:50:16 php [Snort] Will retry in 15 seconds...
Nov 5 19:50:16 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:50:01 php [Snort] Will retry in 15 seconds...
Nov 5 19:50:01 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:49:46 php [Snort] Will retry in 15 seconds...
Nov 5 19:49:46 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:49:46 php [Snort] There is a new set of Snort OpenAppID detectors posted. Downloading snort-openappid.tar.gz...
Nov 5 19:49:45 php [Snort] Snort Subscriber rules file download failed... server returned error '302'...
Nov 5 19:49:45 php File 'snortrules-snapshot-29161.tar.gz' download attempts: 4 ...
Nov 5 19:49:30 php [Snort] Will retry in 15 seconds...
Nov 5 19:49:30 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:49:15 php [Snort] Will retry in 15 seconds...
Nov 5 19:49:15 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:48:59 php [Snort] Will retry in 15 seconds...
Nov 5 19:48:59 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:48:44 php [Snort] Will retry in 15 seconds...
Nov 5 19:48:44 php [Snort] Rules download error: SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
Nov 5 19:48:43 php [Snort] There is a new set of Snort Subscriber rules posted. Downloading snortrules-snapshot-29161.tar.gz...
Nov 5 19:48:20 php [Snort] Downloading and updating configured rule sets.
Nov 5 19:48:20 php [Snort] Settings successfully migrated to new configuration format...
Nov 5 19:48:20 php [Snort] Checking configuration settings version...

ET rules are ok. Only snort rules are my problem.

Before that happened I did a package update of freeradius and saw that it did something with snort. I thought that it was a mistake what I saw...
Then I did update snort. Errors I have have seen:

Nov 5 19:53:12 snort 28098 FATAL ERROR: Failed to load /usr/local/etc/snort/snort_22683_pppoe0/snort_dynamicpreprocessor/libsf_appid_preproc.so: Shared object "libluajit-5.1.so.2" not found, required by "libsf_appid_preproc.so"

I did reinstall that. Everything seems to work now except the download of the rules.
And I did reinstall freeradius and snort. Nothing changed.

I already read that there may be a problem with curl. But what are the libs that I may have to reinstall?

What has happened? Any thoughts, ideas or solutions?

Please tell me what you need to narrow the problem and to help.

I thank you very much.

-demux

kiokoman

system clock ?
today is not -> Nov 5
the certificate for that url is ok, maybe it was a temporary problem

demux

You're right. I did that on thursday.

demux

I did it on the pfSense command line using curl.
Seem to be two errors in one.
A certificate problem for the site I will be redirected to (302) and at the end a file not found (404)

curl -o foo.txt -L https://www.snort.org/downloads/community/community-rules.tar.gz
curl: (60) SSL: no alternative certificate subject name matches target host name 'snort-org-site.s3.amazonaws.com'
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The URL I will be redirected to is:
'https://snort-org-site.s3.amazonaws.com/production/release_files/files/000/015/505/original/community-rules.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIXACIED2SPMSC7GA%2F20201108%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201108T101513Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc0bab0115d40669bf8d979e5e676e43c6d296c034bde8b60791177614770cb1'

In foo.txt I read (when adding -k to curl command line):
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

This happens when using curl on the pfSense box.
When I use my good old Chrome on Windows, both URLs end in a happily downloaded rules file.

demux

I just added

curl_setopt($ch, CURLOPT_SSL_**VERIFYHOST**, $config['installedpackages']['snortglobal']['curl_no_verify_ssl_peer'] == "on" ? false : true);

after

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, $config['installedpackages']['snortglobal']['curl_no_verify_ssl_peer'] == "on" ? false : true);

to snort_check_for_rule_updates.php in function snort_download_file_url.

Now I got beyond the certificate errors and was able to get the 404 errors in the download logs.

demux

Maybe someone can try the two links on the command line using curl and tell me what happens.
I am unsure if it's curl or snort's web site.

kiokoman

[2.4.5-RELEASE][root@pfSense.trmultiservice.lab]/root: curl -o foo.txt -L https://www.snort.org/downloads/community/community-rules.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   467    0   467    0     0    531      0 --:--:-- --:--:-- --:--:--   531
100  324k  100  324k    0     0  93743      0  0:00:03  0:00:03 --:--:--  167k


[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: curl -o foo.txt -L https://www.snort.org/downloads/community/community-rules.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   467    0   467    0     0    368      0 --:--:--  0:00:01 --:--:--   368
100  324k  100  324k    0     0  90309      0  0:00:03  0:00:03 --:--:--  190k

[2.4.5-RELEASE][root@pfSense.trmultiservice.lab]/root: curl -V
curl 7.67.0 (amd64-portbld-freebsd11.3) libcurl/7.67.0 OpenSSL/1.0.2u zlib/1.2.11 nghttp2/1.40.0
Release-Date: 2019-11-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

maybe reinstall curl
pkg install -f curl-7.68.0 php72-curl-7.2.29

demux

@kiokoman said in Error 302: Snort xyz file download failed... server returned error '302':

pkg install -f curl-7.68.0 php72-curl-7.2.29

No. That did not work.

Any other ideas?

demux

Are there any other packages where reinstallation may be helpful?

demux

curl --version gives:
curl 7.68.0 (amd64-portbld-freebsd11.3) libcurl/7.68.0 OpenSSL/1.0.2u zlib/1.2.11 nghttp2/1.40.0
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

Do you see the same? (OpenSSL seems pretty old).

bmeeks

Do you have any other packages installed on this firewall? Especially packages such as Squid, Squidguard or pfBlockerNG? If so, I would concentrate my efforts there. You mentioned FreeRADIUS, but there is absolutely no link at all between that package and Snort.

A little over 24,000 Snort and Suricata users on pfSense visit the Snort Rules servers every day to check for rules updates. I got that number directly from the Snort team. In fact they asked me quite some time back to create some "randomness" in the rules update time because the big load at the old default update times was killing their servers.

So why do I mention that there are 24,000 rule downloads per day? To illustrate that this 100% has to be a problem on your end. What that is, I can't say other than to offer the troubleshooting tips in the first paragraph above. If this were a problem with the Snort package, there would be a ton of posts here describing the same issue.

If you don't have any of the packages I mentioned installed, or you can't find any issues with them, the next thing I suggest is you backup your configuration and reinstall pfSense from scratch. A "wipe and reload" sequence.

Just FYI. The most common cause of Snort rule download failures of late has been due to Squid or Squidguard. In the more distant past, it was sometimes silly and poorly maintained lists of IP addresses fed to pfBlockerNG that contained the AWS IP space.

demux

@bmeeks

Do you have any other packages installed on this firewall? Especially packages such as Squid, Squidguard or pfBlockerNG? If so, I would concentrate my efforts there. You mentioned FreeRADIUS, but there is absolutely no link at all between that package and Snort.

Yes, I do have them installed. But they are all disabled. And on the LAN they do not make any problems. I can download from another machine using curl.

And all of your thoughts and statements are exactly mine. I am sure that I am "the only one" having these problems. Otherwise there would be some sort of storm.

BUT

Please see the logs. I did only look at the logs immediately after manual downloads of rules. I did not see what it did while I was away and doing other things.

Starting rules update... Time: 2020-11-07 18:07:18
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29161.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29161.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Done downloading rules file.
Downloading Snort AppID Open Text Rules md5 file appid_rules.tar.gz.md5...
Checking Snort AppID Open Text Rules md5 file...
There is a new set of Snort AppID Open Text Rules posted.
Downloading file 'appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Extracting and installing Snort OpenAppID detectors...
Installation of Snort OpenAppID detectors completed.
Extracting and installing Snort AppID Open Text Rules...
Installation of Snort AppID Open Text Rules completed.
Extracting and installing Snort GPLv2 Community Rules...
Installation of Snort GPLv2 Community Rules completed.
Copying new config and map files...
Updating rules configuration for: WAN1 ...
Updating rules configuration for: LAN ...
Updating rules configuration for: WAN2 ...
The Rules update has finished. Time: 2020-11-07 18:08:48

Starting rules update... Time: 2020-11-08 10:03:41
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29161.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29161.tar.gz' from url 'https://www.snort.org/rules/snortrules-snapshot-29161.tar.gz?oinkcode=my_oinkcode'...
Snort Subscriber rules file download failed. Server returned error 404.
The error text was: 404 Not Found
Snort Subscriber rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz' from url 'https://www.snort.org/downloads/openappid/snort-openappid.tar.gz'...
Snort OpenAppID detectors file download failed. Server returned error 404.
The error text was: 404 Not Found
Snort OpenAppID detectors will not be updated.
Downloading Snort AppID Open Text Rules md5 file appid_rules.tar.gz.md5...
Checking Snort AppID Open Text Rules md5 file...
There is a new set of Snort AppID Open Text Rules posted.
Downloading file 'appid_rules.tar.gz' from url 'https://files.pfsense.org/openappid/appid_rules.tar.gz'...
Done downloading rules file.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz' from url 'https://www.snort.org/downloads/community/community-rules.tar.gz'...
Snort GPLv2 Community Rules file download failed. Server returned error 404.
The error text was: 404 Not Found
Snort GPLv2 Community Rules will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
Extracting and installing Snort AppID Open Text Rules...
Installation of Snort AppID Open Text Rules completed.
The Rules update has finished. Time: 2020-11-08 10:03:44

There were many unsuccessful downloads (manual downloads). I did not see that there was an automatic successful download - inbetween. After this download I cleared all and started again with my testings (did not see the successful download). At the beginning I was unable to even download the appid files - that I can download now.

It's pretty erratic. And why are the ET always downloaded successfully? Why can I download from files.pfsense.org? They also use curl.

Puhhhhh....

PS: I changed the source to see a little bit more. This is why it looks different. I also added the line I mentioned above.

bmeeks

Snort rules are hosted on Amazon Web Services infrastructure. They also use redirects from one URL to another in order to eventually reference the AWS site. The ET rules have a more straightforward URL setup. All of that is the responsibility of the rules provider. The package has no control over that.

For whatever reason, your system is not liking that setup. With some packages, disabling them is not the same as removing them. My bet is on Squid or Squidguard if you have either of those installed. The 302 error is a redirect problem. Google "http 302" for more info.

The Snort download URLs 100% work. That I can guarantee you. You need to direct your efforts at the other installed packages and figure out why one of them is interfering with the download.

demux

@bmeeks Ok, I will try that on Friday afternoon. Remove squid, squidguard and even pfBlockerNG.

If that does not work, then I will reinstall.

Question to be sure:
Everything is contained within the pfSense config file that can be downloaded? For all installed packages? (Except files that I myself have added or modified) Nothing else needs to be saved before restore?

bmeeks

For Snort, everything is stored within the config.xml file where all other pfSense configuration information is stored. So you when you remove and reinstall Snort all the settings come back.

For the other packages, I can't say as I don't maintain those.