Snort services stopped after update

Sander88

This morning I found one of my PFSense boxes with stopped Snort services. It's running Snort on 3 different interfaces and stopped working when the update script was applying the updated files.

My syslog (Snort related messages):
Oct 25 00:03:00 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2983.tar.gz…
Oct 25 00:04:27 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort VRT rules file update downloaded successfully
Oct 25 00:04:27 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz…
Oct 25 00:04:29 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Oct 25 00:04:34 pfsense kernel: pid 13602 (snort), uid 0: exited on signal 11
Oct 25 00:04:34 pfsense kernel: pid 12826 (snort), uid 0: exited on signal 10
Oct 25 00:04:37 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF1 …
Oct 25 00:04:38 pfsense kernel: pid 24601 (snort), uid 0: exited on signal 4
Oct 25 00:04:38 pfsense kernel: pppoe0: promiscuous mode disabled
Oct 25 00:04:44 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF1…
Oct 25 00:04:45 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF2 …
Oct 25 00:04:52 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF2…
Oct 25 00:04:53 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF3 …
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_cron_misc.inc: [Snort] Alert pcap file cleanup job removed 1 pcap file(s) from /var/log/snort/snort_em465338/…
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_cron_misc.inc: [Snort] Alert pcap file cleanup job removed 1 pcap file(s) from /var/log/snort/snort_pppoe019272/…
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF3…
Oct 25 00:05:01 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Oct 25 00:05:03 pfsense check_reload_status: Syncing firewall
Oct 25 00:05:04 pfsense php-fpm[45441]: /rc.filter_synchronize: Beginning XMLRPC sync to https://1.2.3.4:443.
Oct 25 00:05:13 pfsense php-fpm[45441]: /rc.filter_synchronize: XMLRPC sync successfully completed with https://1.2.3.4:443.
Oct 25 00:05:20 pfsense php-fpm[45441]: /rc.filter_synchronize: Filter sync successfully completed with https://1.2.3.4:443.

Based on this logs and script (/usr/local/pkg/snort/snort_check_for_rule_updates.php) I think this is a bug:
1. Snort crashes when the rule definition files are replaced by the updater. Snort is probably using/reading them at the same time.
2. The update scripts only checks if Snort is running after these update have been applied. So it the script will see no running Snort processing and skips the (re)start.

An easy fix would probably to check the Snort service status a few seconds earlier, before appling the update files and use that result to determine if services need to be (re)started afterwards.

My box is running 2.3.3-RELEASE (amd64) and Snort 3.2.9.2_16.

Do I see this correctly? Or any other idea's or possible Snort crash causes?

bmeeks

You may have a hardware problem. According to the FreeBSD docs, Signal 10 is a memory bus error. Found this reference on Google –


bus error

 <processor>A fatal failure in the execution of a machine language
instruction resulting from the processor detecting an anomalous condition
on its bus. Such conditions include invalid address alignment (accessing a
multi-byte number at an odd address), accessing a physical address that
does not correspond to any device, or some other device-specific hardware
error. A bus error triggers a processor-level exception which Unix
translates into a "SIGBUS" signal which, if not caught, will terminate the
current process.

This can quite plausibly be caused by hardware error, or memory problems
in particular.</processor>

Signal 4 is an "illegal instruction" error.

I have Snort running on three interfaces on my personal firewall. I have had no issues with the daily updates. I use Snort VRT rules which typically update twice weekly, and Emerging Threats Open rules which typically update daily. My box checks for updates twice daily at 0130 and 1330.

Bill

Raffi_

I have the same problem. It doesn't always crash the services after an update, I think it depends on the specific types of rule being updated. Signal 10 may be related to a hardware issue, but why would that issue only come up after snort updates it's rules? Everything else has been rock solid for months.

I'm running pfsense 2.4.1
Snort package 3.2.9.5_2

Oct 27 00:05:04 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2990.tar.gz…
Oct 27 00:05:16 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort VRT rules file update downloaded successfully
Oct 27 00:06:09 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort OpenAppID detectors md5 download failed…
Oct 27 00:06:09 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Server returned error code 503…
Oct 27 00:06:15 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort OpenAppID RULES detectors posted. Downloading appid_rules.tar.gz…
Oct 27 00:06:16 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort OpenAppID RULES detectors file update downloaded successfully
Oct 27 00:06:38 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort GPLv2 Community Rules are up to date…
Oct 27 00:06:39 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz…
Oct 27 00:06:40 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Oct 27 00:06:48 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN …
Oct 27 00:06:55 kernel pid 81481 (snort), uid 0: exited on signal 10
Oct 27 00:06:55 kernel re0: promiscuous mode disabled
Oct 27 00:06:56 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: WAN…
Oct 27 00:06:57 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for WAN…
Oct 27 00:07:00 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Oct 27 00:07:00 check_reload_status Syncing firewall

bmeeks

@Sander88:

This morning I found one of my PFSense boxes with stopped Snort services. It's running Snort on 3 different interfaces and stopped working when the update script was applying the updated files.

My syslog (Snort related messages):
Oct 25 00:03:00 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2983.tar.gz…
Oct 25 00:04:27 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort VRT rules file update downloaded successfully
Oct 25 00:04:27 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz…
Oct 25 00:04:29 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Oct 25 00:04:34 pfsense kernel: pid 13602 (snort), uid 0: exited on signal 11
Oct 25 00:04:34 pfsense kernel: pid 12826 (snort), uid 0: exited on signal 10
Oct 25 00:04:37 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF1 …
Oct 25 00:04:38 pfsense kernel: pid 24601 (snort), uid 0: exited on signal 4
Oct 25 00:04:38 pfsense kernel: pppoe0: promiscuous mode disabled
Oct 25 00:04:44 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF1…
Oct 25 00:04:45 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF2 …
Oct 25 00:04:52 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF2…
Oct 25 00:04:53 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: INTERF3 …
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_cron_misc.inc: [Snort] Alert pcap file cleanup job removed 1 pcap file(s) from /var/log/snort/snort_em465338/…
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_cron_misc.inc: [Snort] Alert pcap file cleanup job removed 1 pcap file(s) from /var/log/snort/snort_pppoe019272/…
Oct 25 00:05:00 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for INTERF3…
Oct 25 00:05:01 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Oct 25 00:05:03 pfsense check_reload_status: Syncing firewall
Oct 25 00:05:04 pfsense php-fpm[45441]: /rc.filter_synchronize: Beginning XMLRPC sync to https://1.2.3.4:443.
Oct 25 00:05:13 pfsense php-fpm[45441]: /rc.filter_synchronize: XMLRPC sync successfully completed with https://1.2.3.4:443.
Oct 25 00:05:20 pfsense php-fpm[45441]: /rc.filter_synchronize: Filter sync successfully completed with https://1.2.3.4:443.

Based on this logs and script (/usr/local/pkg/snort/snort_check_for_rule_updates.php) I think this is a bug:
1. Snort crashes when the rule definition files are replaced by the updater. Snort is probably using/reading them at the same time.
2. The update scripts only checks if Snort is running after these update have been applied. So it the script will see no running Snort processing and skips the (re)start.

An easy fix would probably to check the Snort service status a few seconds earlier, before appling the update files and use that result to determine if services need to be (re)started afterwards.

My box is running 2.3.3-RELEASE (amd64) and Snort 3.2.9.2_16.

Do I see this correctly? Or any other idea's or possible Snort crash causes?

Just noticed this line in your log output –


Oct 25 00:03:00 pfsense php: /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2983.tar.gz...

That is the wrong version of Snort rules. The current version should be snortrules-snapshot-2990.tar.gz. The only way I can imagine your box trying to download that old version is if your Snort binary is the wrong version. The precompiled shared-object rules are FreeBSD version specific. Could be that something is getting tripped up there as the older Snort version might be bringing down FreeBSD 10.x precompiled rules.

Why don't you try the following steps?

1. Remove the Snort package under System…Package Manager.

2. Open a CLI session to the firewall and then remove any directories with snort in their name under /usr/local/etc and /usr/local/lib.

3. Reinstall Snort from the System…Package Manager menu.

If you have "Save Settings" selected on the GLOBAL SETTINGS tab, you won't lose any Snort settings. That option is enabled by default.

Bill

bmeeks

For both posters in this thread –

I looked further into this issue and think I may have found a potential breaking point in the Snort GUI code. It would show up for users of pfSense 2.4.x which is based on FreeBSD 11 instead of FreeBSD 10 as the older pfSense versions are. I have submitted a fix that can be viewed here: https://github.com/pfsense/FreeBSD-ports/pull/471. It should be approved and merged in the near future.

Bill

Raffi_

@bmeeks:

For both posters in this thread –

I looked further into this issue and think I may have found a potential breaking point in the Snort GUI code. It would show up for users of pfSense 2.4.x which is based on FreeBSD 11 instead of FreeBSD 10 as the older pfSense versions are. I have submitted a fix that can be viewed here: https://github.com/pfsense/FreeBSD-ports/pull/471. It should be approved and merged in the near future.

Bill

Thanks Bill!

I downloaded the latest snort update which includes this change. I'll let you know how it goes. Although, right now I'm dealing with a different issue so I have snort disabled for the moment.

Raffi

Raffi_

Bill,

Snort has been working fine for the last few weeks. I haven't received any notifications from the watchdog service for snort. I'll post back here if anything comes up again, but it seems solid now.
Thanks for the fix!

Raffi