Watchdog needed for services to keep them running.
-
Services for Clamav, DNS, Snort, and Squid proxy are all randomly stopping, I reboot the Netgate firewall and they run again for about 4 hours. I have reinstalled the packages and it happens again. I recently installed was watchdog and it fixed my issues. What could cause the services to randomly stopping. I have attached some screenshots to show everything is running fine.
Services page showing everything working currently.
SNORT running
Blocked IPs per SNORT IPS IDS running
Squid Proxy running for content accelerator
Squid realtime showing ClamAV running for proxy's catches
Status showing Proxy running it is still early so hit rate is low right now.
This is my fix is the Service watchdog package. What would cause the services to stop?
-
I am the Snort and Suricata package developer/maintainer, and I can most emphatically tell you that using Service Watchdog with either the Snort or Suricata packages will cause you problems. That's because Service Watchdog does not understand how Snort and Suricata work internally, and thus it makes "bad decisions" about when to restart those packages. So remove Snort from your Service Watchdog monitoring.
As to why Snort is stopping for you, have you checked the system log to see what Snort-related messages might be logged? That would help you troubleshoot the problem.
Snort is a fairly reliable package. I've personally run it for years, and I honestly don't think I've ever had an unplanned stoppage of the service on my firewalls. This is through multiple generations of hardware. My current hardware is a Netgate SG-5100.
What kind of hardware are you running Snort on? And what is the version of pfSense (or pfSense+) and the Snort package?
-
@bmeeks I am using Netgate 2100 max and pFsense that came with the device upgraded to the latest developer released version. Services just randomly turn off.
-
@jonathanlee said in Watchdog needed for services to keep them running.:
@bmeeks I am using Netgate 2100 max and pFsense that came with the device upgraded to the latest developer released version. Services just randomly turn off.
But you have not given me anything from the pfSense system log. Saying the services "just randomly turn off" is not helpful at all in terms of troubleshooting "why". There is almost always a reason logged in the system log when something goes wrong (like a service unexpectedly stopping). You can access the log under STATUS > SYSTEM LOGS.
One possibility that comes to mind is exhaustion of disk storage space. The base SG-2100 has only 8 GB of disk space. If logging from your installed packages is not carefully monitored, and logs pruned regularly, you can easily run out of disk space. That will cause all kinds of problems.
-
Logs
Software version
I have the MAX edition with the large hard drive.
My memory is being used up to 70-90 sometimes -
@jonathanlee could the logs and blocks not be saved to ssd and are going to memory instead?? Why does it not use that SSD to free up some memory?
Make sure to upvote
-
@jonathanlee said in Watchdog needed for services to keep them running.:
@jonathanlee could the logs and blocks not be saved to ssd and are going to memory instead?? Why does it not use that SSD to free up some memory?
@JonathanLee, we are never going to solve this problem without seeing the actual content of your pfSense system log. Post that. I don't need to see pictures of the Dashboard. Nothing on there tells me anything about what was logged from the issues you report.
I don't understand what you mean about using the SSD to free up memory. If you mean swapping out to disk, that is horribly inefficient and would slow everything to a complete crawl. You pretty much never want to see swap space in use.
You have memory-hungry packages installed (Snort and Squid), but only 4 GB of RAM. And that RAM has to be shared with the operating system. You should probably rethink what you are doing on this level of hardware. But again, still need to see the system log messages to make any troubleshooting headway.
-
Adminning pfSense with a phone is hard for you - and for us.
I presume this shows 91 % of the RAM memory used.
That's a lot, consider going to the safe side, and make your system use 'under the 50 %'.In the main system log, look for 'oom' (out of memory) messages.
As said, if the shut down of a process was controlled or instructed by some other process, there will be a log message when process, like snort, starts - and stops (to be restarted).
IDS/IPS processes analyse in real time 'known content' with what comes in on interfaces like WAN and LAN. This content has to be in memory = RAM, not disk as the disk speed will never meet up with the incoming packet speed, the lookup process would be to slow.
Btw : I would never ever scan or analyse traffic that hits the WAN interface : I don't care / don't want to tune into what is the Internet's background noise. The WAN is the jungle.I was told that if I had to do "IPS/IPS" I would need a device like this or even better this.
A small arm processor, far less as my 2 years old Smartphone - would have issues will running
Snort or Surricata (ClamAV) or Squid.You run a 3 of time on one device ??
I'm impressed ....I know, "they said" it could be done. Here, take this typical SoHO device and do what the big guys do.
@bmeeks pulls it of on a 5100 - Intel based, not Arm, are able to do so, as they know where - on what interface, for example - to scan for what. Using only the most important rules and settings. Knowing how things work comes in handy if you want (make) things to work for you.The IDS/IPS packages are the ultimate resource eaters. Big processors, several dozens of Gb of RAM, fast, big drivers are needed.
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
I removed unneeded snort rules and got it down to 40 percent use now.
-
Trimming the IDS rules in use will certainly help. But you are asking a lot for that box to run both Squid and Snort. By the way, when you use the Snort Subscriber Rules (meaning you have an Oinkcode), you do not need to enable the Snort GPLv2 Community Rules. They are already included as part of the Snort Subscriber Rules.
And unless you are doing MITM (man-in-the-middle) proxying, your ClamAV package is going to be pretty much worthless as it cannot scan SSL-encrypted content. And 95% or more of web traffic today is SSL (https).
For instance, on my SG-5100, Snort is the only installed package. I run nothing else on it.
-
@gertjan I just wish they used more of the SSD for items, it is alot faster over a hard drive of the past. Use that SSD for items that are not part of non stop scanning. Remember upper and lower memory blocks back with MS-DOS 3.11 in the 90s with a 14mhz 8088 processor? It seems to me the resources that are available like that huge SSD are not being used as much as they could. I do understand the need for speed with the packet processing. However I can't stop wondering what could be put on that drive and not be constantly ready in NVRAM.
