Snort intermittently seems to crash, trying to find why

pftdm007

Hello,

I just logged in to my pfsense router, and noticed on the front page that the snort service was down. I went to the service page (Services > Snort) , and all interfaces were down. Manually starting the service works without issues. This is like if something or someone was just manually shutting the service down on all interfaces. Its probably just crashing for some reasons.

This doesnt happen often, but I noticed it happening 1 or 2 times in the last 6 months. I am worried because if I hadn't logged in to Pfsense, I wouldn't have know leaving me without protection.

Now the issue I am facing is, I cannot find any relevant information about why this happens. I tried to find information in Pfsense's logs but unfortunately, the logs are so filled up that changing the number of lines doesnt help.

In which log should I look for info on this? /var/log/system.log has nothing in it, and its the same content as displayed in the webinterface.

bmeeks

pfSense uses a circular logging package (clog) that automatically overwrites old entries. You can configure a length for the clog, but once the circle starts over again you can't recover overwritten entries.

If Snort is crashing, then something will get written to the system log. I've run Snort continuously for years on my firewall with no crash of any kind. In fact, the only time it stops is during a firewall software upgrade.

One issue that can happen is a rule update might contain a syntax error. This has happened in the past with some rules. When restarting after a rules archive update, if Snort encounters a rule syntax error, it will abort startup. So if you use a ton of rules, then you could get hit with this issue. But something like that would be logged in the firewall's system log. How often do you check on your firewall and the logs? I would recommend at least once per day if your network security is important.

I don't enable and use every single Snort rule, so I've never been bitten by the "bad syntax" thing I described. My guess for you, is that you were bitten by one of the bad syntax rules and you just didn't notice until after a significant amount of time had passed that Snort had failed to start back up after the rules update. Unless your firewall is logging a ton of stuff to the system log, I would expect events to stay visible for at least two days. I have my log set for 250 GUI entries and I am currently seeing events from 3 days ago.

pftdm007

@bmeeks

Hello Bill, Sorry for the very late reply... I do check the firewall's status often, perhaps not every day but 2-3x per week.

I have enabled the watchdog service in pfsense, lets see what happens next time it crashes.

That would be very nice to have a dedicated notification feature in case the snort IF crashes. On the other hand that has not happened since I wrote here back last month.

Crossing my fingers, but whenever it happens, I will try to look at the logs a bit more..

Thanks!

bmeeks

@pftdm007 said in Snort intermittently seems to crash, trying to find why:

@bmeeks

Hello Bill, Sorry for the very late reply... I do check the firewall's status often, perhaps not every day but 2-3x per week.

I have enabled the watchdog service in pfsense, lets see what happens next time it crashes.

That would be very nice to have a dedicated notification feature in case the snort IF crashes. On the other hand that has not happened since I wrote here back last month.

Crossing my fingers, but whenever it happens, I will try to look at the logs a bit more..

Thanks!

NEVER use Service Watchdog with Snort or Suricata!!! It does not understand how Snort and Suricata work internally. Using Service Watchdog with either Snort or Suricata can actually cause a crash instead of recovering one. I've mentioned this many times in this sub-forum.

pftdm007

@bmeeks

Thanks for letting me know that! I did not happen to browse the forums about this specific issue/topic... I think these services should not be available in Watchdog's available services to monitor if this is that critical? Anyways, +1 for implementing a real monitoring feature for such critical processes I think..

bmeeks

@pftdm007 said in Snort intermittently seems to crash, trying to find why:

@bmeeks

Thanks for letting me know that! I did not happen to browse the forums about this specific issue/topic... I think these services should not be available in Watchdog's available services to monitor if this is that critical? Anyways, +1 for implementing a real monitoring feature for such critical processes I think..

Snort and Suricata are unique in how they operate. They are quite different from services such as the ntpd daemon or something like sshd, for example. The two main differences are Snort and Suricata spawn separate executables for each configured interface, and both services restart themselves following a rules update. If you have LAN and WAN instances of Snort or Suricata configured, that means two separate snort or suricata processes are running: one per interface. Service Watchdog simply does the equivalent of a

ps -ax | grep snort

to see if a monitored process is running. So if it finds any snort process with that query, it is happy. But Snort might be running on the LAN yet have crashed on the WAN. Service Watchdog would never know that.

The other thing that trips up Service Watchdog is that it does not understand that Snort and Suricata restart themselves following a rules update. So if Service Watchdog happens to test for the existence of a Snort or Suricata process during that rules update restart, it will find no running instance and immediately try to start one up. That will likely collide with the restarting Snort or Suricata is doing itself following the rules update and can result in a crash.