Snort fatal error on start
-
My guess is that during the automatic package removal and re-install operation that occurs during a pfSense upgrade, one or more of the Snort binary files in the PBI got corrupted upon the re-install. The file it is complaining about is one of the binary shared-object rule files. It is true this file is updated with each rules update as well, so it may have gotten corrupted then.
Try these steps:
First, force a total new download of Snort rules by going to the UPDATES tab and clicking the FORCE button.
If that does not fix it, (it really should, though), then try removing and re-installing the Snort package. This step really should not be necessary, though.
Bill
I forced the rules updating but it didn't work. Same result reinstalling the package keeping the old settings.
I see from the log that snort exits with a signal 11:Sep 20 12:04:47 barnyard2[99050]: Waiting for new data
Sep 20 12:04:47 barnyard2[99050]: Opened spool file '/var/log/snort/snort_rl023326/snort_23326_rl0.u2.1410818949'
Sep 20 12:04:47 barnyard2[99050]: Using waldo file '/var/log/snort/snort_rl023326/barnyard2/23326_rl0.waldo': spool directory = /var/log/snort/snort_rl023326 spool filebase = snort_23326_rl0.u2 time_stamp = 1410818949 record_idx = 28
Sep 20 12:04:47 barnyard2[99050]: Barnyard2 initialization completed successfully (pid=99050)
Sep 20 12:04:47 barnyard2[99050]: –== Initialization Complete ==--
Sep 20 12:04:47 barnyard2[99050]:
Sep 20 12:04:47 barnyard2[99050]: Writing PID "99050" to file "/var/run/barnyard2_rl023326.pid"
Sep 20 12:04:47 barnyard2[99050]: PID path stat checked out ok, PID path set to /var/run
Sep 20 12:04:47 barnyard2[98746]: Daemon parent exiting
Sep 20 12:04:47 barnyard2[99050]: Daemon initialized, signaled parent pid: 98746
Sep 20 12:04:47 barnyard2[98746]: Initializing daemon mode
Sep 20 12:04:47 barnyard2[98746]: Log directory = /var/log/snort/snort_rl023326
Sep 20 12:04:47 barnyard2[98746]: Barnyard2 spooler: Event cache size set to [8192]
Sep 20 12:04:46 barnyard2[98746]: –-------------------------- +[ Signature Suppress list ]+
Sep 20 12:04:46 barnyard2[98746]: +[No entry in Signature Suppress List]+
Sep 20 12:04:46 barnyard2[98746]: +[ Signature Suppress list ]+ –--------------------------
Sep 20 12:04:46 barnyard2[98746]: Found pid path directive (/var/run)
Sep 20 12:04:46 barnyard2[98746]: Parsing config file "/usr/pbi/snort-i386/etc/snort/snort_23326_rl0/barnyard2.conf"
Sep 20 12:04:46 barnyard2[98746]: Initializing Output Plugins!
Sep 20 12:04:46 barnyard2[98746]: Initializing Input Plugins!
Sep 20 12:04:46 barnyard2[98746]: –== Initializing Barnyard2 ==--
Sep 20 12:04:46 barnyard2[98746]:
Sep 20 12:04:46 barnyard2[98746]: Running in Continuous mode
Sep 20 12:04:46 barnyard2[98746]: Found pid path directive (/var/run)
Sep 20 12:04:46 SnortStartup[98424]: Barnyard2 START for Snort su LAN(23326_rl0)…
Sep 20 12:04:44 kernel: pid 91088 (snort), uid 0: exited on signal 11
Sep 20 12:04:44 barnyard2[90144]: Waiting for new data
Sep 20 12:04:44 barnyard2[90144]: Opened spool file '/var/log/snort/snort_rl143270/snort_43270_rl1.u2.1410818919'
Sep 20 12:04:44 barnyard2[90144]: Using waldo file '/var/log/snort/snort_rl143270/barnyard2/43270_rl1.waldo': spool directory = /var/log/snort/snort_rl143270 spool filebase = snort_43270_rl1.u2 time_stamp = 1410818919 record_idx = 26
Sep 20 12:04:44 barnyard2[90144]: Barnyard2 initialization completed successfully (pid=90144)
Sep 20 12:04:44 SnortStartup[90747]: Snort START for Snort su LAN(23326_rl0)…
Sep 20 12:04:44 barnyard2[90144]: –== Initialization Complete ==--
Sep 20 12:04:44 barnyard2[90144]:
Sep 20 12:04:44 barnyard2[90144]: Writing PID "90144" to file "/var/run/barnyard2_rl143270.pid"
Sep 20 12:04:44 barnyard2[90144]: PID path stat checked out ok, PID path set to /var/run
Sep 20 12:04:44 barnyard2[90144]: Daemon initialized, signaled parent pid: 87874
Sep 20 12:04:44 barnyard2[87874]: Daemon parent exiting
Sep 20 12:04:44 barnyard2[87874]: Initializing daemon mode
Sep 20 12:04:44 barnyard2[87874]: Log directory = /var/log/snort/snort_rl143270
Sep 20 12:04:44 barnyard2[87874]: Barnyard2 spooler: Event cache size set to [8192]
Sep 20 12:04:44 barnyard2[87874]: –-------------------------- +[ Signature Suppress list ]+
Sep 20 12:04:44 barnyard2[87874]: +[No entry in Signature Suppress List]+
Sep 20 12:04:44 barnyard2[87874]: +[ Signature Suppress list ]+ –--------------------------
Sep 20 12:04:44 barnyard2[87874]: Found pid path directive (/var/run)
Sep 20 12:04:44 barnyard2[87874]: Parsing config file "/usr/pbi/snort-i386/etc/snort/snort_43270_rl1/barnyard2.conf"
Sep 20 12:04:44 barnyard2[87874]: Initializing Output Plugins!
Sep 20 12:04:44 barnyard2[87874]: Initializing Input Plugins!
Sep 20 12:04:44 barnyard2[87874]: –== Initializing Barnyard2 ==--
Sep 20 12:04:44 barnyard2[87874]:Quite strange!
-
If you use Barnyard2, maybe disable that Feature, and then do your Upgrade to see if you can atleast get it upgraded? Re-enable Barnyard after that?
-
If you use Barnyard2, maybe disable that Feature, and then do your Upgrade to see if you can atleast get it upgraded? Re-enable Barnyard after that?
Tried but same result:
Sep 22 10:03:03 kernel: pid 7377 (snort), uid 0: exited on signal 11
Sep 22 10:03:02 SnortStartup[7290]: Snort START for Snort su LAN(23326_rl0)…
Sep 22 10:03:01 kernel: pid 6966 (snort), uid 0: exited on signal 11
Sep 22 10:03:00 SnortStartup[6686]: Snort START for Snort su WAN(43270_rl1)…
Sep 22 09:45:54 check_reload_status: Reloading filter
Sep 22 09:45:39 kernel: pid 26501 (snort), uid 0: exited on signal 11
Sep 22 09:45:39 SnortStartup[26164]: Snort START for Snort su LAN(23326_rl0)…
Sep 22 09:45:39 php: /pkg_mgr_install.php: [Snort] Package post-installation tasks completed…
Sep 22 09:45:37 kernel: pid 24335 (snort), uid 0: exited on signal 11
Sep 22 09:45:37 SnortStartup[24035]: Snort START for Snort su WAN(43270_rl1)…
Sep 22 09:45:37 php: /pkg_mgr_install.php: [Snort] Starting Snort using rebuilt configuration…
Sep 22 09:45:37 php: /pkg_mgr_install.php: [Snort] Finished rebuilding installation from saved settings…I think that I need to purge the configuration... :'(
Thanks,
Stenio -
I think that I need to purge the configuration… :'(
Configuration purged and reinstalled. Now it works properly.
Thanks,
Stenio -
It happens every time now the rules get updated. After every update the unicode.map file is blank again. Found no solution so snort is nearly useless now.
I did a complete reinstall of pfsense and snort. I can't even start snort the first time… all i get is the FATAL ERROR.
Something seems really f***** up with Netgate APU & pfsense 2.1.5 & snort. I never had any problems before pfsense 2.1.5. Even on the older Netgate m1n1wall i had no problems for years.
So if a fresh install didn't fix the issue then it's really f***** up somewhere.
I will pass this along to the Netgate guys. One time in the past there was a temporary issue similar to this. Perhaps it has occurred again. BTW, if you have a support contract with Netgate, you could certainly contact them about your issue. That may be faster than my e-mail to them.
Bill
Yes i have a support contract with Netgate. I thought that this is related to just a package so they can't help anyway. But it seems that only Netgate APUs are affected so i contacted the support about the issue.
I will post the answer here is they are aware of it. Let me know what they told you.
I passed along a link to this thread and notified them some users were having issues and what I thought the issue might be. They acknowledged receipt and said they would look into it. I provided them my personal contact information if something else was needed from me or if it was later determined the problem was something I might need to fix within the Snort package.
Bill
I finally got an answer:
I see you purchased an APU with NanoBSD pfSense installed on an SD card. Snort and many other packages will not function on pfSense when installed on a SD card. A full installation of pfSense is required and to do so, pfSense must be installed on an SSD.
Your options are limited. If Snort is essential, you will need an mSATA SSD drive with pfSense installed. We could sell you an mSATA drive, preloaded with pfSense 2.1.5. Otherwise, you could reset your device to factory default settings and operate without Snort.
–--------------
That's it. Not really hepful because it worked fine prior to pfsense 2.1.5. So i guess it can't be fixed then.
-T5000
-
Thats a REALLY weird answer since it worked on 2.1.4…..
-
Thats a REALLY weird answer since it worked on 2.1.4…..
Yes, it worked very well for months.
-
Thats a REALLY weird answer since it worked on 2.1.4…..
Yes, it worked very well for months.
One thing that has slowly been changing is the growth in the number of rules as different exploits are handled by rules updates. The amount of free space required to unpack and install everything incrementally increases. Same goes for the binary parts of some packages (the PBI files). These must be downloaded, unzipped, and then installed. That takes additional disk space. Compared to a full SSD or HD install, there is not much space on CF (NanoBSD) installs due to the default partition layouts. What is happening, I think, is the package installer is running out of space and silently failing in a number of different ways depending on exactly where in the install process it runs out of disk space.
Bill
-
Today I found snort not running. If I start it I see:
Sep 29 15:50:28 php: /snort/snort_interfaces.php: The command '/usr/local/bin/snort -R 39369 -D -q -l /var/log/snort/snort_rl139369 –pid-path /var/run --nolock-pidfile -G 39369 -c /usr/pbi/snort-i386/etc/snort/snort_39369_rl1/snort.conf -i rl1' returned exit code '1', the output was ''
Sep 29 15:50:28 snort[13094]: FATAL ERROR: /usr/pbi/snort-i386/etc/snort/snort_39369_rl1/snort.conf(169) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Sep 29 15:50:27 php: /snort/snort_interfaces.php: [Snort] Snort START for WAN(rl1)…It worked for a few days after a complete reinstall. I'm using nanobsd but I have 1.4 GB of free space:
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/pfsense0 1.8G 271M 1.4G 16% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ufs/cf 49M 5.2M 40M 11% /cf
/dev/md0 38M 2.2M 33M 6% /tmp
/dev/md1 57M 23M 30M 44% /var
devfs 1.0k 1.0k 0B 100% /var/dhcpd/devWhat can I do?
Thanks,
Stenio -
Today I found snort not running. If I start it I see:
Sep 29 15:50:28 php: /snort/snort_interfaces.php: The command '/usr/local/bin/snort -R 39369 -D -q -l /var/log/snort/snort_rl139369 –pid-path /var/run --nolock-pidfile -G 39369 -c /usr/pbi/snort-i386/etc/snort/snort_39369_rl1/snort.conf -i rl1' returned exit code '1', the output was ''
Sep 29 15:50:28 snort[13094]: FATAL ERROR: /usr/pbi/snort-i386/etc/snort/snort_39369_rl1/snort.conf(169) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Sep 29 15:50:27 php: /snort/snort_interfaces.php: [Snort] Snort START for WAN(rl1)…It worked for a few days after a complete reinstall. I'm using nanobsd but I have 1.4 GB of free space:
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/pfsense0 1.8G 271M 1.4G 16% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/ufs/cf 49M 5.2M 40M 11% /cf
/dev/md0 38M 2.2M 33M 6% /tmp
/dev/md1 57M 23M 30M 44% /var
devfs 1.0k 1.0k 0B 100% /var/dhcpd/devWhat can I do?
Thanks,
StenioWell, while I'm sure it's not what you want to hear, I would ditch Nano and go the full-install route on a hard disk (either SSD or conventional). I suspect that will end your problems. There are lots of users here running Snort on conventional full installs with no issues.
If you want to stay with Nano, then I suggest ditching the Snort package (and Suricata as well). I think you are going to face continual problems otherwise. What probably happened to you is an updated Snort VRT rules package downloaded, and due to the issues with Nano, did not unzip and install itself correctly. The unicode.map file is probably corrupted again.
Bill
-
Well, while I'm sure it's not what you want to hear, I would ditch Nano and go the full-install route on a hard disk (either SSD or conventional). I suspect that will end your problems. There are lots of users here running Snort on conventional full installs with no issues.
OUCH!!! :'(
-
Well, while I'm sure it's not what you want to hear, I would ditch Nano and go the full-install route on a hard disk (either SSD or conventional). I suspect that will end your problems. There are lots of users here running Snort on conventional full installs with no issues.
OUCH!!! :'(
Sorry… :'(.
However, I am willing to try some experimentation if you are game. Maybe we can make this work better. Send me a PM if you are willing to help test a little bit.
Bill
-
I think I got it:
Oct 1 00:03:39 php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN …
Oct 1 00:03:31 kernel: pid 81598 (bsdtar), uid 0 inumber 1362 on /tmp: filesystem full
Oct 1 00:03:28 kernel: pid 9265 (php), uid 0 inumber 1361 on /tmp: filesystem full
Oct 1 00:03:28 kernel: pid 80954 (bsdtar), uid 0 inumber 1355 on /tmp: filesystem full
Oct 1 00:03:14 kernel: pid 80259 (bsdtar), uid 0 inumber 1380 on /tmp: filesystem full
Oct 1 00:02:59 kernel: pid 58582 (bsdtar), uid 0 inumber 1365 on /tmp: filesystem full
Oct 1 00:02:55 kernel: pid 58582 (bsdtar), uid 0 inumber 1365 on /tmp: filesystem full
Oct 1 00:02:55 kernel: pid 58582 (bsdtar), uid 0 inumber 1364 on /tmp: filesystem full
Oct 1 00:02:54 php: snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Oct 1 00:02:46 php: snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz…Need to increase the /tmp filesystem size I think.
-
Need to increase the /tmp filesystem size I think.
Changed from the default 40MB to 80MB and now it seems to work.
During the rules update the size went to 51MB! -
Need to increase the /tmp filesystem size I think.
Changed from the default 40MB to 80MB and now it seems to work.
During the rules update the size went to 51MB!Glad you found it. The rules update process downloads the rules tarball archives and then unpacks them in a directory under /tmp. Once it finishes, it deletes the folder. But if that directory fills up, then unpredictable stuff happens.
Bill
-
I found that if you disable the HTTP Inspect component, that ignores the IIS Unicode map and starts Snort without issue.
Here's how you disable it: Snort Interface -> Edit your Interface, (mine is named WAN)-> Select the <wan>Preprocs tab, navigate to the HTTP Inspect section and UNCHECK it. That will allow your snort IDS to start back up without issue.
I'm running pfSense 2.1.5 with Snort 2.9.7.0 pkg v.3.2.1 on a 4GB CF Card.
Additional Troubleshooting:
I tried to just limit the webservers in the HTTP Inspect section to just inspect an Apache Web server, and ignore IIS completely. That did not work and it just failed again, so I just disabled the HTTP inspect section entirely.Error Messages:
After enabling Snort via the WebUI, I received the following error message -Dec 21 23:29:57 my.pfsensefirewall.com Dec 21 23:30:00 snort[99416]: FATAL ERROR: /usr/pbi/snort-amd64/etc/snort/snort_xxxx_em0/snort.conf(166) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file. ```</wan> -
I found that if you disable the HTTP Inspect component, that ignores the IIS Unicode map and starts Snort without issue.
Here's how you disable it: Snort Interface -> Edit your Interface, (mine is named WAN)-> Select the <wan>Preprocs tab, navigate to the HTTP Inspect section and UNCHECK it. That will allow your snort IDS to start back up without issue.
I'm running pfSense 2.1.5 with Snort 2.9.7.0 pkg v.3.2.1 on a 4GB CF Card.
Additional Troubleshooting:
I tried to just limit the webservers in the HTTP Inspect section to just inspect an Apache Web server, and ignore IIS completely. That did not work and it just failed again, so I just disabled the HTTP inspect section entirely.Error Messages:
After enabling Snort via the WebUI, I received the following error message -Dec 21 23:29:57 my.pfsensefirewall.com Dec 21 23:30:00 snort[99416]: FATAL ERROR: /usr/pbi/snort-amd64/etc/snort/snort_xxxx_em0/snort.conf(166) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file. ```</wan>You are going to experience more issues with disabling the HTTP_INSPECT preprocessor. Snort and Suricata are becoming too "big" to install and update reliably on Nano installs of pfSense. I strongly encourage Snort and Suricata users to stick with full installs on either conventional hard disks or SSD. Both packages need plenty of free disk space to work (and free RAM).
Bill
