SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped)

bmeeks

@wolfsden3 said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

Nov 14 13:42:00 php-cgi 56692 servicewatchdog_cron.php: Service Watchdog detected service snort stopped. Restarting snort (Snort IDS/IPS Daemon)

I've posted here on the forums like a million times -- DO NOT monitor Snort with Service Watchdog. That package does not understand how Snort works internally as a package, does not understand when Snort is automatically restarting itself, and does not even now how to properly monitor all the possible Snort configured interfaces.

Service Watchdog is a very simpleton package. It monitors for process name, and if that single process name disappears, it calls the registered shell script to attempt a restart. This causes big problems for Snort and Suricata. Both of those packages routinely restart themselves when auto-updating rules or even when you as the admin save certain configuration changes. Service Watchdog is ignorant of this simple fact. It sees the monitored process daemon go away, so it blindly issues its own restart command. Now it is very likely that multiple Snort processes will attempt to startup on the same interface. And that means trouble.

Not saying this is your only issue, but it certainly is making things worse!

wolfsden3

@bmeeks said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

ut it certainly is making things worse!

Cool, I thought about that too but decided to post asking. I've turned it off in watchdog and will monitor.

Oh, looks like it restarted again...I'll check the logs again.

Thanks!

wolfsden3

@bmeeks

Perhaps it's SSH Guard...?

Nov 14 14:53:00 	sshguard 	62136 	Now monitoring attacks.
Nov 14 14:53:00 	sshguard 	28621 	Exiting on signal.
Nov 14 14:52:02 	kernel 		igb2: promiscuous mode disabled
Nov 14 14:52:02 	kernel 		pid 38492 (snort), jid 0, uid 0: exited on signal 11 (core dumped)

SNORT is still restarting but now after the dump sshguard is logged as "Now monitoring attacks". I hadn't noticed that before, the first post also mentions sshguard. I had just assumed it was snort since it was "core dumped".

Thoughts?

bmeeks

@wolfsden3 said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

@bmeeks

Perhaps it's SSH Guard...?
Nov 14 14:53:00 	sshguard 	62136 	Now monitoring attacks.
Nov 14 14:53:00 	sshguard 	28621 	Exiting on signal.
Nov 14 14:52:02 	kernel 		igb2: promiscuous mode disabled
Nov 14 14:52:02 	kernel 		pid 38492 (snort), jid 0, uid 0: exited on signal 11 (core dumped) 
SNORT is still restarting but now after the dump sshguard is logged as "Now monitoring attacks". I hadn't noticed that before, the first post also mentions sshguard. I had just assumed it was snort since it was "core dumped".

Thoughts?

No, sshguard has nothing to do with this. That is restarting due to system log rotations most likely.

Snort is faulting for some other reason. In one of your posts up above you intimated this had been happening before you updated to the latest package version. Is that correct?

wolfsden3

@bmeeks Correct. In the prior version I didn't see this. In the prior version there was yet another bug where I had to add a \ in front of a $ somewhere on line 42 I think it was for the interface to even start in some snort config file. I've since lost that link but it fixed it until the upgrade which is now in a perpetual restart.

I just tried reinstalling snort > no worky. I also tried an uninstall > reinstall, still the same behavior on both boxes.

I'm surprised nobody else has this issue. Maybe it's specific to something I have configured. I'm in legacy mode.

Did more digging online...looks like you replied to a similar thread with the core dump problem here:

https://forum.netgate.com/topic/166076/snort-exited-on-signal-11-core-dumped/7

Still looking for a fix.

bmeeks

Try this for me to help focus the troubleshooting:

Go to the INTERFACE SETTINGS tab in Snort and uncheck the Kill States option. Save that change and manually start or restart Snort.

Let me know how that works.

wolfsden3

@bmeeks

Done. I also ticked an option somewhere in the config for more robust logging. I saw this and tried to untick "Resolve Flowbits" for giggles but it still bombed later so I turned resolving flowbits back on.

Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'file.pkp' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'blackhole.jar' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'MQTT_Connect' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'ET.000webhostpost' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'FraudLoad' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'smb.neoteris' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'DigiWatcher232_detection' is set but not ever checked.
Nov 14 16:54:59 	snort 	87355 	WARNING: flowbits key 'tlsv1.1_handshake' is set but not ever checked.

wolfsden3

@wolfsden3

So far...after unticking kill states, no crash. What's that mean? :P

bmeeks

@wolfsden3 said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

@wolfsden3

So far...after unticking kill states, no crash. What's that mean? :P

Sorry to be late replying. Was away at an appreciation dinner getting stuffed .

This tells me where the problem is in the code. This area was modified about two weeks ago by a Netgate kernel developer who manages pf in FreeBSD. The kill states section of kernel system calls changed back in late August in FreeBSD and thus in the latest pfSense. That developer made some fixes in the custom module for Legacy Blocking specifically in the area that calls "kill states" in FreeBSD. Since you no longer get the crash when kill states is disabled, that means that the area he changed is almost certainly where the crash is happening now in Snort. So, at least we have the area narrowed down.

This will not be as quick of a fix as a GUI PHP file. I will need to get that developer's assistance again to help track down the failure point and fix it.

bmeeks

I've opened Redmine Issue #14986 to track this. I have also exchanged emails with the Netgate developer team and engaged their kernel experts to help out tracking this bug down and fixing it.

DD

We have same problem with Snort - exited on signal 11 (core dumped) after updated to 4.1.6_13. After untick kill states it is not crashing.

bmeeks

@DD said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

We have same problem with Snort - exited on signal 11 (core dumped) after updated to 4.1.6_13. After untick kill states it is not crashing.

Thanks for the report. Can you please confirm to me what pfSense version and family you are using?

Are you running pfSense CE or pfSense Plus, and what is the specific version?

bmeeks

@wolfsden3 said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

I saw this and tried to untick "Resolve Flowbits" for giggles but it still bombed later

Flowbits warnings are not a problem and could not cause a Signal 11 segfault.

There is an issue somewhere in the custom blocking plugin and/or in the new FreeBSD libpfctl library. The right guy is looking into it. He is one of the main guys maintaining the pf firewall engine code in FreeBSD.

DD

@bmeeks We are using pfSense CE 2.7.0-RELEASE.

bmeeks

@DD said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

@bmeeks We are using pfSense CE 2.7.0-RELEASE.

Hmm... okay. Was not expecting that because I have thus far been unable to reproduce the issue in my CE 2.7.0-RELEASE virtual machine. Obviously I need to try harder .

Thank you for the additional information.

DD

@bmeeks If I enable "kill states", then Snort is working for sometime (sometime 5 minutes, sometime 2 minutes, sometime 30 minutes.....) a then it will stop. After start, it is working again for sometime. When I disabled kill states, snort is working without stoping.

fireodo

@DD said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

@bmeeks If I enable "kill states", then Snort is working for sometime (sometime 5 minutes, sometime 2 minutes, sometime 30 minutes.....) a then it will stop. After start, it is working again for sometime. When I disabled kill states, snort is working without stoping.

Same here - i guess this fault is present only on very "busy" interfaces where frecvent "kill state" are necessary.

bmeeks

@DD said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

@bmeeks If I enable "kill states", then Snort is working for sometime (sometime 5 minutes, sometime 2 minutes, sometime 30 minutes.....) a then it will stop. After start, it is working again for sometime. When I disabled kill states, snort is working without stoping.

Yes, that behavior correlates with the current theory of the bug. My suspicion is that when Snort blocks an IP and then attempts the follow-up action of clearing any open firewall states for that IP, it encounters the bug and crashes. But apparently this is not "always", because if it was "always and everytime", I could reproduce it easily and all users would be seeing the crash. So, something else is also likely at play. Still searching for the actual root cause. For example, it may be that two differnet events have to occur together to trigger the bug ???

bmeeks

@fireodo said in SNORT: promiscuous mode disabled + promiscuous mode enabled error + exited on signal 11 (core dumped):

Same here - i guess this fault is present only on very "busy" interfaces where frecvent "kill state" are necessary.

Yes, it does seem to require some extra "something" to trigger it because I have thus far not been successful. But my puny test environment with VMware Workstation virtual machines can't generate a ton of traffic.

DD

@bmeeks Yes, because if I looking to alerts and blocked tab then alerts and blocked items are increasing and Snort is working ok. But after sometime snort will stop and log only one row: exited on signal 11 (core dumped).