Snort Rules Download Fail: "SSL certificate problem"

satisfieduser

Downloading Snort rules fails.

I use a free (but registered) Snort account. The following rulesets fail: Snort VRT Rules, Snort GPLv2 Community Rules, Snort OpenAppID Detectors. However, Emerging Threats Open Rules does download successfully.

I've tried to reinstall Snort, and I removed the settings beforehand. No joy. Added/removed the interface, too. I have no other certificates installed on this pfSense installation (which, btw, runs on a dedicated PC). pfSense and Snort have been working well up to recently. Not sure how to troubleshoot this? Is there another certificate in the chain that is installed without my knowledge? Where to look, where to start?

Thank you all, and especially thank you Bill Meeks for your hard work on this forum.

START LOG:
–------------------------------------------------

Starting rules update... Time: 2016-07-20 15:44:34
Downloading Snort VRT rules md5 file snortrules-snapshot-2983.tar.gz.md5...
Checking Snort VRT rules md5 file...
There is a new set of Snort VRT rules posted.
Downloading file 'snortrules-snapshot-2983.tar.gz'...
Snort VRT rules file download failed. Server returned error 0.
The error text was: SSL certificate problem: unable to get local issuer certificate
Snort VRT rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Snort OpenAppID detectors file download failed. Server returned error 0.
The error text was: SSL certificate problem: unable to get local issuer certificate
Snort OpenAppID detectors will not be updated.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5...
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Snort GPLv2 Community Rules file download failed. Server returned error 0.
The error text was: SSL certificate problem: unable to get local issuer certificate
Snort GPLv2 Community Rules will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
The Rules update has finished. Time: 2016-07-20 15:47:41

bmeeks

Go to the GLOBAL SETTINGS tab and try checking the new option under Rules Update Settings for not verifying SSL peers. That may help. If not, then you will need to investigate why the firewall is failing to find its local certs. Report back.

This advice assumes you are running the latest 3.2.9.1_14 version of the Snort package.

Bill

satisfieduser

Bill,

Yes, I'm running snort 3.2.9.1_14 and pfSense 2.3.1-RELEASE-p5.

I "checked" the new option to not verify SSL peers per your suggestion. Unfortunately, this did not resolve the issue. I subsequently tried to download the rules normally as well as "force" update the rules and neither was successful.

Is there was away to verify/reinstall the local certs? From the log below, it appears we have a cert issue…
The log is providing different information now… note the lines in bold:

Starting rules update… Time: 2016-07-20 19:37:37
Downloading Snort VRT rules md5 file snortrules-snapshot-2983.tar.gz.md5...
Checking Snort VRT rules md5 file...
There is a new set of Snort VRT rules posted.
Downloading file 'snortrules-snapshot-2983.tar.gz'...
Snort VRT rules file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort VRT rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5…
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Snort OpenAppID detectors file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort OpenAppID detectors will not be updated.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5…
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Snort GPLv2 Community Rules file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort GPLv2 Community Rules will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5…
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
The Rules update has finished. Time: 2016-07-20 19:40:43

kody

This happened to me.

I regenerated my Snort ID on the Snort website, added the new ID to the appropriate page in Snort in pfSense, retried and success.

bmeeks

@satisfieduser:

Bill,

Yes, I'm running snort 3.2.9.1_14 and pfSense 2.3.1-RELEASE-p5.

I "checked" the new option to not verify SSL peers per your suggestion. Unfortunately, this did not resolve the issue. I subsequently tried to download the rules normally as well as "force" update the rules and neither was successful.

Is there was away to verify/reinstall the local certs? From the log below, it appears we have a cert issue…
The log is providing different information now… note the lines in bold:

Starting rules update… Time: 2016-07-20 19:37:37
Downloading Snort VRT rules md5 file snortrules-snapshot-2983.tar.gz.md5...
Checking Snort VRT rules md5 file...
There is a new set of Snort VRT rules posted.
Downloading file 'snortrules-snapshot-2983.tar.gz'...
Snort VRT rules file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort VRT rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5…
Checking Snort OpenAppID detectors md5 file...
There is a new set of Snort OpenAppID detectors posted.
Downloading file 'snort-openappid.tar.gz'...
Snort OpenAppID detectors file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort OpenAppID detectors will not be updated.
Downloading Snort GPLv2 Community Rules md5 file community-rules.tar.gz.md5…
Checking Snort GPLv2 Community Rules md5 file...
There is a new set of Snort GPLv2 Community Rules posted.
Downloading file 'community-rules.tar.gz'...
Snort GPLv2 Community Rules file download failed. Server returned error 0.
The error text was: SSL: certificate subject name 'pfSense-54e15a9eb6fa5' does not match target host name 's3.amazonaws.com'
Snort GPLv2 Community Rules will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5…
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
The Rules update has finished. Time: 2016-07-20 19:40:43

There is something wrong with the cert process, but I am not enough of an expert in that area to tell you what the cause may be. Are you using any kind of proxy? If so, there could be issues there. Really all the package code does is use a native PHP implementation of cURL to download the rules files. That is failing because the native PHP function is getting confused by the cert handed to it by your pfSense implementation. The key is in those bold text error messages. "s3.amazonws.com" is the web site where the VRT rules live. For some reason when attempting to check the HTTPS cert from that site, your pfSense installation is instead presenting a cert from the firewall itself.

Bill

gogol

Maybe Squid with SSL intercept?

satisfieduser

Bill, Kody, Gogol,

I sincerely thank you for your time and for providing the thoughtful suggestions. Given the fact that Bill Meeks (demi-god on this forum) didn't know the answer right off the cuff, I decided to wipe and reload pfSense tonight, and restore the config from a backup. This, indeed, fixed the issue. Snort now downloads successfully and all packages (snort, pfblocker, etc.) function correctly.

When it comes to certificates, I, too, am not an expert. Therefore, the integrity of the installed certificates are a serious concern given that the purpose of pfSense and Snort is to provide security for your network against a wide threat model– the thought of a MITM attack, or more likely, a weakened environment due to corrupt or missing certificates was an impediment for me to attempt to resolve this issue without fully understanding the problem. Therefore I decided to wipe and reinstall. My apologies to those who desired to learn something from this experience. I, too, would like to have known more but in this case, I chose to be conservative.

For what it's worth, I am not using a proxy, although in the past I did have it connected to a VPN but I reverted back a few weeks ago to removing the VPN connection (it too often sporadically/randomly dropped the connection). My pfSense package is installed on a PC, and performs its function for a wired network behind a wireless router acting as a gateway to the internet. I've had a few issues with pfSense in the past (and posted some of them on this forum seeking assistance for which I am grateful), but nothing this odd or curious. Prior to this experience, pfSense and the packages had run for months without incident.

Bill, thank you for your time and effort. Without Snort, pfSense wouldn't serve its intended purpose. Period.

All the best to you all! ;D

mhertzfeld

I am having the same problem.

Regenerated a new Oink code and saved it into snort, but the update still fails.

I am running PFSENSE 2.2.6 and Snort 3.2.9.1

The last successful update was 7/15/2016.

This is the error.

Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] The Rules update has finished.
Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Removed 0 obsoleted rules category files.
Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Hide Deprecated Rules is enabled. Removing obsoleted rules categories.
Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Emerging Threats Open rules are up to date…
Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Snort VRT rules file download failed… server returned error '0'...
Jul 23 10:54:43 php-fpm[28794]: /snort/snort_download_rules.php: File 'snortrules-snapshot-2976.tar.gz' download attempts: 4 …
Jul 23 10:54:28 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Will retry in 15 seconds…
Jul 23 10:54:28 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Rules download error: SSL certificate problem: unable to get local issuer certificate
Jul 23 10:54:13 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Will retry in 15 seconds…
Jul 23 10:54:13 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Rules download error: SSL certificate problem: unable to get local issuer certificate
Jul 23 10:53:57 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Will retry in 15 seconds…
Jul 23 10:53:57 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Rules download error: SSL certificate problem: unable to get local issuer certificate
Jul 23 10:53:42 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Will retry in 15 seconds…
Jul 23 10:53:42 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] Rules download error: SSL certificate problem: unable to get local issuer certificate
Jul 23 10:53:42 php-fpm[28794]: /snort/snort_download_rules.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2976.tar.gz…

mhertzfeld

I think DNSBL is causing the issue.

Stopping the DNSBL service wasn't enough to get it working.

I had to disable DNSBL to get the rules to download properly.

BBcan177

@mhertzfeld:

I think DNSBL is causing the issue.

Stopping the DNSBL service wasn't enough to get it working.

I had to disable DNSBL to get the rules to download properly.

Did you review the DNSBL Alerts to see what Domain is getting blocked? I've not seen Snort Updates getting blocked via any of the common DNSBL feeds available. Are you using any other Feed(s) not listed in the DNSBL threads?

The DNSBL Service is the web server portion of the DNSBL feature, so disabling that will only stop the reporting of the blocked Domains. Its not recommended to shut that down as it will cause the browser to wait longer in the timeout on the blocked Domains.

satisfieduser

I initially wiped and reinstalled pfSense to resolve my problem, however I came back to report that the same issue has recurred as of today. I also removed Snort and installed Surricata instead in order to test this problem. Indeed, the same issue persists with Surricata.

I have verified what "mhertzfeld" indicated in a previous post on this thread: DNSBL (part of pfBlocker package) is causing this issue. Turning off pfBlocker and DNSBL will successfully permit rules downloads to proceed as normal.

–----------------------------------------------------

I am using the following lists in DNSBL:

[Advertisements]
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://adaway.org/hosts.txt
http://someonewhocares.org/hosts/hosts

[Malicious \ Malware]
http://mirror1.malwaredomains.com/files/justdomains
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://data.phishtank.com/data/online-valid.csv.gz
https://www.openphish.com/feed.txt

[DShield.org suspicious domains]
https://isc.sans.edu/suspicious_domains.html)
https://dshield.org/feeds/suspiciousdomains_Medium.txt
https://dshield.org/feeds/suspiciousdomains_High.txt

[Windows telemetry]
https://github.com/WindowsLies/BlockWindows)
https://raw.githubusercontent.com/WindowsLies/BlockWindows/master/hostslist

–----------------------------------------------------

More testing and results to follow...

mhertzfeld

@BBcan177:

@mhertzfeld:

I think DNSBL is causing the issue.

Stopping the DNSBL service wasn't enough to get it working.

I had to disable DNSBL to get the rules to download properly.

Did you review the DNSBL Alerts to see what Domain is getting blocked? I've not seen Snort Updates getting blocked via any of the common DNSBL feeds available. Are you using any other Feed(s) not listed in the DNSBL threads?

The DNSBL Service is the web server portion of the DNSBL feature, so disabling that will only stop the reporting of the blocked Domains. Its not recommended to shut that down as it will cause the browser to wait longer in the timeout on the blocked Domains.

I did not see any alerts during or shortly after the time I triggered manual update of the Snort rules when they were failing.

If I remember correctly, I only disabled DNSBL on the DNSBL tab, I don't think that I disabled pfblockerng. Could be remembering wrong though? I re-enabled shortly after the update succeeded.

Here is a list of feeds I am using in addition to the easy lists.

http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
https://adaway.org/hosts.txt
http://sysctl.org/cameleon/hosts
https://raw.githubusercontent.com/Dawsey21/Lists/master/adblock-list.txt
http://dshield.org/feeds/suspiciousdomains_Low.txt
http://dshield.org/feeds/suspiciousdomains_Medium.txt
http://dshield.org/feeds/suspiciousdomains_High.txt
http://someonewhocares.org/hosts/hosts
https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
https://malc0de.com/bl/BOOT
https://mirror1.malwaredomains.com/files/justdomains
http://winhelp2002.mvps.org/hosts.txt
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://adblock.gjtech.net/?format=unix-hosts
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
http://osint.bambenekconsulting.com/feeds/dga-feed.txt
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt
http://osint.bambenekconsulting.com/feeds/c2-masterlist.txt
https://raw.githubusercontent.com/WindowsLies/BlockWindows/master/hostslist
https://raw.githubusercontent.com/TambourineReindeer/BlockWindows/master/hostslist
http://hosts-file.net/ad_servers.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/exp.txt
http://hosts-file.net/fsa.txt
http://hosts-file.net/mmt.txt
http://hosts-file.net/pha.txt
http://hosts-file.net/psh.txt

BBcan177

Pls run the following commands to see if the Domain [ [b]s3.amazonws.com ] is listed by any of your Feeds:

This will report if this Domain is in the Original Feeds:

grep "s3.amazonws.com" /var/db/pfblockerng/dnsblorig/*

This will show what is in the Final Feeds (After any Whitelisting):

grep "s3.amazonws.com" /var/db/pfblockerng/dnsbl/*

Would recommend to add [ [b]s3.amazonws.com ] to your DNSBL Whitelist just in case…. When pfBlockedNG v2.1.1 is released, you will be able to prefix any Whitelisted Domain with a "Dot" to Whitelist all other Sub-Domains, however, do not prefix the Whitelited Domains until version 2.1.1, or it will not work correctly for v2.0.17…

satisfieduser

BBcan177,

Your white-listing suggestion seems to be working for the domain, "s3.amazonws.com" (which apparently hosts the Snort rules). Thank you for taking the time to provide this information!

;D

All the best,