Snort 2.9.4.6 Pkg v 2.5.9

gogol

I am digging deeper and deeper.

An extract from system.log:

Jul 23 10:11:52 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 10:11:53 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jul 23 10:11:53 pfsensetest php: rc.newwanip: Creating rrd update script
Jul 23 10:11:54 pfsensetest check_reload_status: Syncing firewall
Jul 23 10:11:55 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 ->  192.168.67.161 ... Restarting packages.
Jul 23 10:11:55 pfsensetest check_reload_status: Starting packages
Jul 23 10:11:55 pfsensetest check_reload_status: Reloading filter
Jul 23 10:11:57 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 10:11:58 pfsensetest SnortStartup[14194]: Snort START for WAN(54137_em0)...
Jul 23 10:12:01 pfsensetest login: login on ttyv0 as root
Jul 23 10:12:01 pfsensetest sshlockout[32403]: sshlockout/webConfigurator v3.0 starting up
Jul 23 10:12:05 pfsensetest check_reload_status: Syncing firewall
Jul 23 10:12:08 pfsensetest sshd[38417]: Accepted keyboard-interactive/pam for admin from 192.168.1.3 port 56484 ssh2
Jul 23 10:12:10 pfsensetest SnortStartup[46525]: Snort START for WAN(54137_em0)...
Jul 23 10:13:49 pfsensetest kernel: em0: promiscuous mode enabled
Jul 23 10:13:53 pfsensetest snort[64616]: SMTP reload: Changing the file_depth requires a restart.
Jul 23 10:13:54 pfsensetest kernel: em0: promiscuous mode disabled
Jul 23 10:14:05 pfsensetest kernel: em0: promiscuous mode enabled

I thought that it might be possible that while a new WAN IP is detected and rc.start_packages restarts the packages Snort has not yet written a pid file and so was not detected already running, a new process would be started. So I disabled the interface that Snort was running on to prevent an automatic startup and manually start Snort.
I was very surprised to see Snort running after a reboot. At the Snort tab I see everything disabled and at Running Services I see Snort running, also when I look at "ps aux | grep snort".

I am very confused.

Supermule

Can you post the config file? OR reinstall a 2.1 VM with Snort without importing the config file?

gogol

Snort has serious timing issues at boot up on my systems.

I tried a lot of combinations with rulesets. It was even possible to start snort without rules, which in my opinion should not be possible.

On my VM with only IPS balanced, which takes about 15 seconds to start on 1 interface, it goes well with only 1 snort process running, but I see the -E argument which is weird.

root   49494 69.0  6.0 387540 124584  ??  Rs    1:11PM   0:04.96 /usr/pbi/snort-i386/bin/snort -R 4082 -E -q -l /var/log/snort/snort_em04082 --pid-path /var/run --nolock-pidfile -G 4082 -c /usr/pbi/snort-i386/etc/snort/snort_4082_em0/snort.conf -i em0

With only 1 ET ruleset enabled I can see only 1 process, but with the -D argument.

Another log snippet with only 1 ruleset enabled:

Jul 23 12:59:09 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 12:59:10 pfsensetest php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jul 23 12:59:10 pfsensetest php: rc.newwanip: Creating rrd update script
Jul 23 12:59:11 pfsensetest check_reload_status: Syncing firewall
Jul 23 12:59:12 pfsensetest php: rc.newwanip: pfSense package system has detected an ip change 192.168.67.161 ->  192.168.67.161 ... Restarting packages.
Jul 23 12:59:12 pfsensetest check_reload_status: Starting packages
Jul 23 12:59:12 pfsensetest check_reload_status: Reloading filter
Jul 23 12:59:14 pfsensetest php: rc.start_packages: Restarting/Starting all packages.
Jul 23 12:59:15 pfsensetest SnortStartup[19390]: Snort START for WAN(4082_em0)...
Jul 23 12:59:15 pfsensetest kernel: em0: promiscuous mode enabled
Jul 23 12:59:16 pfsensetest login: login on ttyv0 as root
Jul 23 12:59:16 pfsensetest sshlockout[31161]: sshlockout/webConfigurator v3.0 starting up
Jul 23 12:59:17 pfsensetest SnortStartup[45501]: Snort STOP for WAN(4082_em0)...
Jul 23 12:59:18 pfsensetest snort[22786]: *** Caught Term-Signal
Jul 23 12:59:18 pfsensetest kernel: em0: promiscuous mode disabled
Jul 23 12:59:23 pfsensetest SnortStartup[47764]: Snort START for WAN(4082_em0)...
Jul 23 12:59:23 pfsensetest kernel: em0: promiscuous mode enabled

Watch the Snort STOP which is missing in another log snippet I posted before.

To resume:

The boot process is interfering with Snort Startup in my opinion or the other way around.

rc.newwanip detects an ip change while there isn't one and triggers a restart packages while Snort is starting, which takes a while
check_reload_status is also Starting Packages

Sometimes there is the -E argument instead of the -D in the process.

@Supermule I did start from scratch and there is nothing special in my config. Snort in test mode gives no errors.

Supermule

Can you pls. make a FRESH install on 2.1 and NOT import the config? And see if it happens again?

gogol

I understand it is a long story but I reported a clean install of latest 2.1 snapshot and a removal of Snort without saving settings and configuring from scratch of few posts ago ;)

Supermule

Yes but its not the snort imported config but that of the firewall.

I understand that you installed from scratch and then imported the FW config afterwards.

gogol

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

bmeeks

@gogol:

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.

Bill

dotOne

Unfortunately I have the same issue.
After a reboot snort is always started twice and sometimes even three times.
It's depending on the machine I run it on.

on an Atom D525 based machine it's alway 2 times and sometimes three
on a i5 3.3G machine it's most of the time only a single process.

It looks like a timing issue indeed.

gogol

@bmeeks:

@gogol:

Ok, for me it is definitely the check for the pid file that messes things up while booting and restarting packages twice as I understand it now. The pid file is written after snort has completely started up and that can take a while. So if the packages are restarted for the second time after a new WAN IP is detected the first snort process is still running without a pid file, so the snort.sh startup script thinks that snort is not running and doesn't stop the first process and just starts a new one. Hence two snort processes.

Is it possible for the snort.sh script to write a dummy pid file that will be replaced after snort has started up? Or another check?

Sorry to chime in late. I am working out of town until the weekend and can't look at this until then. I think you may be correct about the timing, but you can't really write a dummy file since that is in the hands of the OS. I have not seen this behavior, and apparently it is rare or I would expect lots of posts here with the same problem. My current theory is perhaps there is something weird in your particular situation.

Hi Bill,

I supposed you were away for some time, because I always get a rapid reply from you.
I have an Atom (N270) based system and a VM on a quad core i5(3,1Ghz) with 2 processors dedicated to the VM. Both have this issue.
This month /etc/rc.newwanip has changed on July 5th and as fas as my understanding of PHP goes I think that this caused it:


46	46	  if($g['booting'])
47	47	    exit;
48	48	  
 	49	 +/* NOTE: Check #2495 before being smart here */
 	50	 +interface_ipalias_cleanup($interface, "inet4");
 	51	 +
49	52	  function restart_packages() {
50	53	    global $oldip, $curwanip, $g;
51	54	  
...	...	 @@ -97,7 +100,6 @@ system_resolvconf_generate(true);
97	100	  /* write current WAN IP to file */
98	101	  file_put_contents("{$g['vardb_path']}/{$interface}_ip", $curwanip);
99	102	  
100	 	 -interface_ipalias_cleanup($interface, "inet4");
101	103	  link_interface_to_vips($interface, "update");
102	104	  
103	105	  unset($gre);

Going through my log files I have the double startup of snort process since a day later (not exactly sure).

eri--

Yeah that's something i am working on.
Its not the final solution.

gogol

It is good to read that this is acknowledged. Thanks. In the meantime I kill the snort processes after boot and restart them again manually.

bmeeks

@gogol:

It is good to read that this is acknowledged. Thanks. In the meantime I kill the snort processes after boot and restart them again manually.

I am back home and read the response from Ermal. If the problem started on July 5 and seems to coincide with a change being made in the pfSense core, I will hold off making any Snort adjustments.

Bill

pfSenseRocks

I repro multiple snort processes on the latest 2.1 builds as well. What additional information can I provide to help debug this issue and get a solution? Thanks!

SystemActivity_snort.PNG_thumb

kilthro

Has anyone else been experiencing this when Snort tries to update? I been getting it for a couple of weeks now. I have set to update once per day automatically which is around midnight.. Should the update run at a different time?
I do pay the yearly subscription to snort. I just have the snort.org rules and Emerging ones enabled. I am guessing its more of an issue with snort site than the app here with pfsense. Just wondering if I was the only one seeing this.

Bill would it be possible if everyone (mass population) would like it to, to have the most recent snort update entry at the top of the log rather than the bottom? Not a major issue just save me some scrolling which at times is a P.I.T.A. on a mobile device.. No worries if its left as is.. just thought I would ask.

system log
Aug 4 00:08:03 php: : [Snort] The Rules update has finished.
Aug 4 00:08:03 php: : [Snort] Emerging Threat rules are up to date…
Aug 4 00:08:02 php: : [Snort] Server returned error code '0'…
Aug 4 00:08:02 php: : [Snort] Snort VRT md5 download failed…
Aug 4 00:08:02 php: : File 'snortrules-snapshot-2946.tar.gz.md5' download attempts: 4 ...
Aug 4 00:07:47 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:07:47 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:06:32 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:06:32 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:05:17 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:05:17 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:04:02 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:04:02 php: : [Snort] Rules download error: Empty reply from server

Snort update log
Starting rules update… Time: 2013-08-04 00:03:01
Downloading Snort VRT md5 file...
Snort VRT md5 download failed.
Server returned error code '0'.
Server error message was 'Empty reply from server'
Snort VRT rules will not be updated.
Downloading EmergingThreats md5 file...
Checking EmergingThreats md5.
Emerging Threats rules are up to date.
The Rules update has finished. Time: 2013-08-04 00:08:03

bmeeks

@kilthro:

Has anyone else been experiencing this when Snort tries to update? I been getting it for a couple of weeks now. I have set to update once per day automatically which is around midnight.. Should the update run at a different time?
I do pay the yearly subscription to snort. I just have the snort.org rules and Emerging ones enabled. I am guessing its more of an issue with snort site than the app here with pfsense. Just wondering if I was the only one seeing this.

Bill would it be possible if everyone (mass population) would like it to, to have the most recent snort update entry at the top of the log rather than the bottom? Not a major issue just save me some scrolling which at times is a P.I.T.A. on a mobile device.. No worries if its left as is.. just thought I would ask.

system log
Aug 4 00:08:03 php: : [Snort] The Rules update has finished.
Aug 4 00:08:03 php: : [Snort] Emerging Threat rules are up to date…
Aug 4 00:08:02 php: : [Snort] Server returned error code '0'…
Aug 4 00:08:02 php: : [Snort] Snort VRT md5 download failed…
Aug 4 00:08:02 php: : File 'snortrules-snapshot-2946.tar.gz.md5' download attempts: 4 ...
Aug 4 00:07:47 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:07:47 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:06:32 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:06:32 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:05:17 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:05:17 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:04:02 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:04:02 php: : [Snort] Rules download error: Empty reply from server

Snort update log
Starting rules update… Time: 2013-08-04 00:03:01
Downloading Snort VRT md5 file...
Snort VRT md5 download failed.
Server returned error code '0'.
Server error message was 'Empty reply from server'
Snort VRT rules will not be updated.
Downloading EmergingThreats md5 file...
Checking EmergingThreats md5.
Emerging Threats rules are up to date.
The Rules update has finished. Time: 2013-08-04 00:08:03

I would frequently see this error in my own logs using the default update time of 3 minutes past midnight and 12:03 PM. Using the new option added in the 2.5.9 update that allows choosing other update times, I've eliminated this error by moving my updates to 1:00 AM and 1:00 PM. My guess is maybe the default time is frequently hitting some maintenance or backup process on the rules update server at Snort.org. I don't have anything to backup that supposition, but I can say that after moving my times to an hour later I have not had the issue happen again.

As for reordering the logs, I can take a look. It would complicate the code because now it just appends to the end of the file. The viewer is just a simple file reader (actually it just copies the contents into a HTML textarea object). I could add some sorting of the lines prior to display, but it will be tricky due to the formatting.

Bill

bmeeks

@pfSenseRocks:

I repro multiple snort processes on the latest 2.1 builds as well. What additional information can I provide to help debug this issue and get a solution? Thanks!

According to a post earlier in this thread, Ermal is making some changes to the 2.1 Snapshot code that are likely responsible for this behavior. It started after a Snapshot update and well after the Snort 2.5.9 package was released. I have been quite busy the past three weeks and have not had a chance to test this nor investigate in my VMware test environment. I will see if I can reproduce it.

FOLLOW UP: I had a 2.1 Snapshot VM running the July 4 code. It does not exhibit the multiple Snort processes problem. I rebooted it several times and only the correct number of Snort processes started (two on this particular VM since I have Snort instances on the WAN side and the LAN side). Others have reported the odd behavior started with the July 5 update.

Bill

kilthro

@bmeeks:

@kilthro:

Has anyone else been experiencing this when Snort tries to update? I been getting it for a couple of weeks now. I have set to update once per day automatically which is around midnight.. Should the update run at a different time?
I do pay the yearly subscription to snort. I just have the snort.org rules and Emerging ones enabled. I am guessing its more of an issue with snort site than the app here with pfsense. Just wondering if I was the only one seeing this.

Bill would it be possible if everyone (mass population) would like it to, to have the most recent snort update entry at the top of the log rather than the bottom? Not a major issue just save me some scrolling which at times is a P.I.T.A. on a mobile device.. No worries if its left as is.. just thought I would ask.

system log
Aug 4 00:08:03 php: : [Snort] The Rules update has finished.
Aug 4 00:08:03 php: : [Snort] Emerging Threat rules are up to date…
Aug 4 00:08:02 php: : [Snort] Server returned error code '0'…
Aug 4 00:08:02 php: : [Snort] Snort VRT md5 download failed…
Aug 4 00:08:02 php: : File 'snortrules-snapshot-2946.tar.gz.md5' download attempts: 4 ...
Aug 4 00:07:47 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:07:47 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:06:32 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:06:32 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:05:17 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:05:17 php: : [Snort] Rules download error: Empty reply from server
Aug 4 00:04:02 php: : [Snort] Will retry in 15 seconds…
Aug 4 00:04:02 php: : [Snort] Rules download error: Empty reply from server

Snort update log
Starting rules update… Time: 2013-08-04 00:03:01
Downloading Snort VRT md5 file...
Snort VRT md5 download failed.
Server returned error code '0'.
Server error message was 'Empty reply from server'
Snort VRT rules will not be updated.
Downloading EmergingThreats md5 file...
Checking EmergingThreats md5.
Emerging Threats rules are up to date.
The Rules update has finished. Time: 2013-08-04 00:08:03

I would frequently see this error in my own logs using the default update time of 3 minutes past midnight and 12:03 PM. Using the new option added in the 2.5.9 update that allows choosing other update times, I've eliminated this error by moving my updates to 1:00 AM and 1:00 PM. My guess is maybe the default time is frequently hitting some maintenance or backup process on the rules update server at Snort.org. I don't have anything to backup that supposition, but I can say that after moving my times to an hour later I have not had the issue happen again.

As for reordering the logs, I can take a look. It would complicate the code because now it just appends to the end of the file. The viewer is just a simple file reader (actually it just copies the contents into a HTML textarea object). I could add some sorting of the lines prior to display, but it will be tricky due to the formatting.

Bill

Thanks Bill! I will move the time to 1 am. I was more concerned if it was just me or if others were seeing it. I appreciate the followup on it.

On the log issue, it seems like more work than what its worth. You dont have to worry about it unless you really feel bored. lol

shinzo

Starting rules update… Time: 2013-08-04 17:44:54
Downloading Snort VRT md5 file...
Checking Snort VRT md5 file...
There is a new set of Snort VRT rules posted. Downloading...
Done downloading rules file.
Downloading EmergingThreats md5 file...
Checking EmergingThreats md5.
There is a new set of EmergingThreats rules posted. Downloading...
Done downloading EmergingThreats rules file.
Extracting and installing EmergingThreats.org rules...
Installation of EmergingThreats.org rules completed.
Extracting and installing Snort VRT rules...
Using Snort VRT precompiled SO rules for FreeBSD-8-1 ...
Installation of Snort VRT rules completed.
Copying new config and map files...
Updating rules configuration for: WAN ...
The Rules update has finished. Time: 2013-08-04 17:45:30

I did a clean install of 2.1 rc-1, i imported a backup file i had of 2.0.3 . So my issue is, the snort package runs fine but i am not getting hits on any of the rules sets. To make sure the rules are working, i do a shields up scan which usually does a few nmap scans of my machine but i dont get hits anymore. I noticed that its saying freebsd-8-1 for the precompiled instead of 8-3. so i dont know if that might have something to do with it?

kilthro

Do you have the rules enabled? Also do you have the block detected port scans enabled? If not, Snort will not be looking for them thus not blocking them.