Snort package reinstall stuck on download of snort rules-snapshot-2990.tar.gz
-
I tried to update to the latest packages today but that failed. After this, I noticed that snort disappeared from the services menu. In the list of backups, I saw an entry for today that said "(system): Snort pkg uninstall removed Dashboard widget." so I assumed this is why it went away.
I tried to re-install the snort package but I could see on the console it gets stuck on "Downloading snortrules-snapshot-2990.tar.gz". I was able to quickly download the file directly from the snort.org web site to my MacBook so i don't think there is a connectivity issue (assuming the re-install is getting it from the same site). After letting the file try to download for over an hour, I tried to revert to the pre-update configuration. This resulted in another reinstall of the snort package and messages that say "Packages are currently being reinstalled in the background. Do not make changes in the GUI until this is complete.".
Looking at the console, I can see the snort package again gets stuck on "Downloading snortrules-snapshot-2990.tar.gz". It has been sitting at this point for several hours.
How can I get the snort package to successfully re-install?
Thanks,
phil
-
This morning (after about 8.5 hours of "downloading") the re-install finally completed. Snort is back in the services menu. Unfortunately, I can't start it on any of the interfaces I use it on. When i try to start snort on an interface, I get the following error:
FATAL ERROR: /usr/local/etc/snort/snort_29637_igb0/snort.conf(169) => Unable to open the IIS Unicode Map file '/usr/local/etc/snort/unicode.map'.
How can this be fixed?
I saw a post suggesting to re-install snort to fix this problem. I just tried to do that again and it is stuck on the "Downloading snortrules-snapshot-2990.tar.gz"." step again. Should this take 8+ hours to complete? Installing snort has never taken this long before.
Thanks,
phil
-
I tried to start over with snort by manually removing the package and dependencies as well as the <snortglobal>configuration information from the command line.
After I did this, the snort package re-installed fine. The install window told me to set up snort in the GUI,. I configured the VRT and ET rules settings through the GUI. I then tried to "Update" the rules. The update got stuck on the VRT rules. I let this run for a long time before reconfiguring to disable the VRT rule set and use the community ruleset instead. This update worked just fine. The ET and community rules always load with no issue.
I then generated a new oinkcode, reconfigured to use the VRT ruleset again instead of the community ruleset, and tried to update rules. The VRT ruleset hung again.
It appears that only the VRT rules won't download.
Does anyone know what might be causing this problem?
I have not had any issues with the VRT rule set in the past.
Thanks,
phil</snortglobal>
-
Do you have any other packages installed? In particular some of the IP lists folks sometimes use with pfBlockerNG will cause this because they will block a portion of the AWS IP address space used by the server hosting the Snort VRT rules.
Is your Oinkcode valid? Look in the log file on the UPDATES tab to see what the exact error message is for the failing download.
Bill
-
I just ran the rule update process again and it worked as expected. I didn't change anything from the last time it didn't work a number of times in a row. I could download the file directly from the snort website with no issue to my MacBook pro. Are the AWS servers the update process us different than the web site? Maybe I couldn't get there for some reason.
I am using the new oikcode which didn't work earlier but is working now.
Now that the download seems to be working, would it make sense to re-load my old config but manually change to the new oinkcode in it prior to loading it?
The only packages I have installed are: avahi, FTP_Client_Proxy, mailreport, nmap, openvpn-client-export. Service_Watchdog, and snort.
I never received an error message. It would just never finish downloading the ruleset (or take 8+ hours to exit) but never seemed to actually succeed in completing the step of downloading the rules.
Here is a list of the messages when it tried to reinstall snort:
Upgrading pfSense-pkg-snort…
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):Installed packages to be REINSTALLED:
pfSense-pkg-snort-3.2.9.5_4 [pfSense]Number of packages to be reinstalled: 1
[1/1] Reinstalling pfSense-pkg-snort-3.2.9.5_4…
[1/1] Extracting pfSense-pkg-snort-3.2.9.5_4: …....... done
Removing snort components...
Menu items... done.
Services... done.
Loading package instructions...
Saving updated package information...
overwrite!
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...Saved settings detected.
Migrating settings to new configuration... done.
Downloading Snort VRT rules md5 file... done.
Checking Snort VRT rules md5 file... done.
There is a new set of Snort VRT rules posted.
Downloading snortrules-snapshot-2990.tar.gz... <<<<---it would say on this step for 8+ hours. Since I didn't watch it the whole time, I don't know what was reported after this step since.Thanks,
phil
-
With regards to the Service Watchdog package, make sure Snort is not managed by it! It does not understand how the Snort binary processes work and can cause multiple instances of Snort to be started on each configured interface.
Something was blocking your access to the AWS address space. Snort constructs a URL using your configured Oinkcode as follows:
https://www.snort.org/rules/snortrules-snapshot-2990.tar.gz?oinkcode=xxxxx
where the "xxxxx" is replaced by the Oinkcode you type into the box on the GLOBAL SETTINGS tab. Your problems with the installation were a result of being unable to download the file and thus the install process would not complete. I don't know what was preventing access to the rules download site, but it had to be somewhat unique to your setup.
Bill
-
Well I finally got things working again but I'm not really sure what was going wrong.
On the pfsense box, I was able to manually download from the rules from the URL you provided using curl from the command prompt. The initial download provided a redirect URL which I successfully loaded with a second curl command.
Despite this and a bunch of other things I tried, the package manager would always get stuck trying to download the "Downloading snortrules-snapshot-2990.tar.gz" file when trying to install the snort package.
Desperate, I saved off all my configs and reloaded a fresh image to the pfsense box and then tried to re-load my old config. This resulted in getting stuck the same place trying to download the file when I tried to install the snort package.
I reset to factory configuration and started over. I was able to install the snort package and configure snort but once again it got stuck trying to download the VRT rules despite the status screen showing that rules had never been downloaded. After waiting a significant time for this to complete, I tried to "force" the rules download. This got stuck at the same place. After waiting significant time for this to complete, I tried to "force" the rules download again. Once again it got stuck trying to download the files. I tried to force the download one more time and it worked! Not sure why but everything seems to be working now. I reconfigured the entire system from scratch which was a lot of work.
Can you think of what might have been causing this problem? I don't want to have to restart from scratch the next time there is a VRT rule update.
Thanks,
phil
-
Well I thought everything was working…after a system automatic snort rule update, almost all of the changes I made to the config to rebuild the system from the ground up was gone.
I looked at the snort logs and saw there was a checksum failure as well as other issues reported. I did a diff of the config before the automatic snort update and after it. It was clear that a large portion of the configuration changes I made were wiped out. I provided a bunch of details below.
On the snort Updates page, the last snort update was reported as 13:20 with a "success" status despite the failure. After the failure, most of my config was gone. A lot of config info was still there (e.g. interfaces, snort interfaces, dns servers, etc.) but a significant amount of config info was removed as shown below.
I restored to the config prior to the automatic snort update which succeeded. I verified that the items that I had previously configured were restored. I the hit the "Update Rules" button and the snort rules updated with no issue.
I am concerned that the next time snort fails it is going to wipe my configuration again. What could cause this to happen?
Where is there no "Starting rules update... Time:..." on the automatic rule update that failed?
Thanks,
phil
The saved configs show the following:
1/14/18 16:47:40 17.3 43 KiB (system): Snort pkg: updated status for updated rules package(s) check.
1/14/18 15:22:13 17.3 82 KiB admin@192.168.1.49: Updated OpenVPN server on WAN:1194 OpenVPNThe logs for the snort update are as follows:
Starting rules update... Time: 2018-01-14 13:20:33
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-14 13:20:34Done downloading rules file.
Snort VRT rules file download failed. Bad MD5 checksum.
Downloaded Snort VRT rules file MD5:
Expected Snort VRT rules file MD5: 7ceec7f1688a978314bff2eb955883ad
Snort VRT rules file download failed. Snort VRT rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Snort OpenAppID detectors md5 download failed.
Server returned error code .
Server error message was: Failed to create file /tmp/snort_rules_up/snort-openappid.tar.gz.md5
Snort OpenAppID detectors will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Emerging Threats Open rules md5 download failed.
Server returned error code .
Server error message was: Failed to create file /tmp/snort_rules_up/emerging.rules.tar.gz.md5
Emerging Threats Open rules will not be updated.
The Rules update has finished. Time: 2018-01-14 16:47:40Starting rules update... Time: 2018-01-14 18:38:22
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-14 18:38:24
After the 2018-01-14 16:47:40 snort rule update, the here is a summary of what changed from my config:
--- /conf/backup/config-1515961333.xml 2018-01-14 15:53:26.064759000 -0500
+++ /conf/backup/config-1515966460.xml 2018-01-14 17:00:05.504536000 -0500- <nextuid>2001</nextuid>
-
<nextuid>2000</nextuid> (because my VPN user was removed)
-
<disablenatreflection>yes</disablenatreflection>
-
<staticmap>... (all static IP address mapping was removed in the new config)
-
<rule>- <source> (all firewall rules were removed in the new config)
- ...
-
-</rule> -
<aliases>- ...
-</aliases>
- <aliases></aliases> (all aliases were removed in the new config)
- (all openvpn config info was removed in the new config)
- <time>1515961333</time>
- <username>admin@192.168.1.49</username>
- <time>1515966460</time>
- <username>(system)</username>
- (all dyndns info removed in the new config)
- <cert>- …. (all cert info removed in new config)
-</cert>
<dhcrelay6></dhcrelay6>
...-
<ips_policy_enable>off</ips_policy_enable>
- ... -
<arp_unicast_detection>off</arp_unicast_detection>
-
<last_rule_upd_status>success</last_rule_upd_status>
-
<last_rule_upd_time>1515954034</last_rule_upd_time>
- <last_rule_upd_status>failed</last_rule_upd_status>
- <last_rule_upd_time>1515966460</last_rule_upd_time> (rule download failed with checksum and other issues)
-
<ftpclientproxy> (not present in new config)
- ...
-</ftpclientproxy> -
<vpn_openvpn_export> (not present in new config)
- ...
-</vpn_openvpn_export> -
<notifications>(not present in new config)
- ... -
<notifications>- <ovpnserver> (not present in new config)
- ...
-</ovpnserver> -
<ca>- ...
-</ca></notifications></notifications></staticmap>
-
Snort will not remove things from your config.xml file outside of specific Snort-related settings. You have something else going on with that box, or else you have a terribly hosed config that you keep restoring from. I would try setting your box up totally from scratch and DO NOT import your old config from a backup. My guess is that config is corrupted. Just rebuild the firewall configuration completely from scratch and see how things work then.
A ton of users have the Snort rules updating regularly without failure (I'm one of them), so this is not an issue with the package. If it were, there would be lots of reports like yours.
These particular errors in your posted update log potentially indicate a serious problem with your file system:
Server error message was: Failed to create file /tmp/snort_rules_up/snort-openappid.tar.gz.md5 ... Server error message was: Failed to create file /tmp/snort_rules_up/emerging.rules.tar.gz.md5Are you perhaps trying to use a RAMDISK but with insufficient RAM? I would manually run fsck on your disk to see if there are errors.
Bill
-
Bill,
I don't doubt that the Snort package works. It worked for several years with no issue on this box until recently when I've had all the problems. I really had no significant issues until now and have been very happy with the pfSense distribution in general and the snort package in particular since it has been very effective a blocking traffic I don't want on my network. I'm starting to wonder if I have a hardware issue.
When I started over again last weekend, I installed with a brand new SSD (blank) and booted from and installed the latest build from memory stick. I did load load the old config and it couldn't download the VRT rules. After I determined this, I executed the "Factory Defaults Reset". I was under the impression that this would reset the "configuration" back to factory defaults but not necessarily changes the files system (e.g. packages loaded). I manually configured everything from scratch after the reset assuming that all the old config data would be gone and that I had good software since I just loaded from memory stick. I guess I should have re-installed from memory stick again prior to the manual reconfig from scratch. Do you think I need to re-isntall again?
The box I am running has 8 GB of RAM, a 250 GB SSD, and don't have RAM DISKs enabled. I did the default install and the bulk of the space went to /. I don't see why I would have run out of space on /tmp as indicated in the logs. Should the size of some of the other partitions be increased (e.g. /var/run)?
–
df -kFilesystem 1024-blocks Used Avail Capacity Mounted on
/dev/gptid/6b58c541-f945-11e7-a83c-90e2ba491b64 232575284 1087476 212881788 1% /
devfs 1 1 0 100% /dev
/dev/md0 3484 108 3100 3% /var/run
devfs 1 1 0 100% /var/dhcpd/devAs you suspected, there were fsck errors. I had to run fsck a number of time at boot up before it returned a clean report.
When I run fsck from the command line, however, it reports the error below (which doesn't show up during the book fsck):
fsck
** /dev/gptid/6b58c541-f945-11e7-a83c-90e2ba491b64 (NO WRITE)USE JOURNAL? no
** Skipping journal, falling through to full fsck
SETTING DIRTY FLAG IN READ_ONLY MODE
UNEXPECTED SOFT UPDATE INCONSISTENCY
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=19903491 OWNER=root MODE=100666
SIZE=0 MTIME=Jan 15 13:13 2018
CLEAR? no** Phase 5 - Check Cyl groups
20936 files, 271867 used, 57871954 free (2986 frags, 7233621 blocks, 0.0% fragmentation)Is there a way to fix this?
With my current config, I increased the update rate on snort rules to every 6 hours to accelerate the occurrence of errors if they would re-occur. As you can see below, despite a hash check failure, a 505 and 404 error from the server, snort gracefully moved on to the next ruleset as expected unlike what I was seeing in my previous posts:
Starting rules update... Time: 2018-01-16 00:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-16 00:05:58Starting rules update... Time: 2018-01-16 06:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-16 06:05:30Starting rules update... Time: 2018-01-16 12:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-16 12:05:10Starting rules update... Time: 2018-01-16 18:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
There is a new set of Snort VRT rules posted.
Downloading file 'snortrules-snapshot-2990.tar.gz'...
Done downloading rules file.
Snort VRT rules file download failed. Bad MD5 checksum.
Downloaded Snort VRT rules file MD5: ad7d02fd9fab9db1711129ebb6adcfca
Expected Snort VRT rules file MD5: 863c1933b8711b81317ef995d8381527
Snort VRT rules file download failed. Snort VRT rules will not be updated.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
Updating rules configuration for: LAN ...
Restarting Snort to activate the new set of rules...
Snort has restarted with your new set of rules.
The Rules update has finished. Time: 2018-01-16 20:22:00Starting rules update... Time: 2018-01-16 21:15:33
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
There is a new set of Snort VRT rules posted.
Downloading file 'snortrules-snapshot-2990.tar.gz'...
Done downloading rules file.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
Extracting and installing Snort VRT rules...
Using Snort VRT precompiled SO rules for FreeBSD-10-0 ...
Installation of Snort VRT rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
Updating rules configuration for: LAN ...
Restarting Snort to activate the new set of rules...
Snort has restarted with your new set of rules.
The Rules update has finished. Time: 2018-01-16 21:16:22Starting rules update... Time: 2018-01-17 00:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-17 00:05:28Starting rules update... Time: 2018-01-17 06:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Snort OpenAppID detectors md5 download failed.
Server returned error code 520.
Server error message was:
Snort OpenAppID detectors will not be updated.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-17 06:06:07Starting rules update... Time: 2018-01-17 12:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
Emerging Threats Open rules are up to date.
The Rules update has finished. Time: 2018-01-17 12:05:21Starting rules update... Time: 2018-01-17 18:05:00
Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
Checking Snort VRT rules md5 file...
Snort VRT rules are up to date.
Downloading Snort OpenAppID detectors md5 file snort-openappid.tar.gz.md5...
Checking Snort OpenAppID detectors md5 file...
Snort OpenAppID detectors are up to date.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Emerging Threats Open rules md5 download failed.
Server returned error code 404.
Server error message was: 404 Not Found
Emerging Threats Open rules will not be updated.
The Rules update has finished. Time: 2018-01-17 18:05:20Thanks,
phil
-
I recommend replacing that disk with a new one. My best guess is what's happening is that during the rules download (when the Snort GUI process is writing the downloaded blocks to disk in a temporary directory in /tmp), the disk write is randomly failing. When that happens, the OS is probably stuck trying repeatedly to make the write of the failing block. That would be why the process would seem to "freeze" at times when you are watching it (like during the package install when it downloads the rules). With disk failures, it is not uncommon for the low-level I/O driver to repeatedly attempt the write operation. It will eventually give up, but sometimes it takes a while depending on the particular operating system.
The fact you see whole parts of your configuration disappear lends further credence to bad hardware (in this case it appears to be the new SSD). I know in the past I've unfortunately purchased new conventional hard disks that were dead out of the box (DOA). The same thing can happen with an SSD. In your case it appears as more "flakey" that just simply DOA.
Bill
