New Package: Service Watchdog
-
Last night I added a very basic package for restarting services if they are detected down. It's still pretty "young" as packages go, but the basics all work.
It only works on pfSense 2.1, and only works with services registered in pfSense, meaning the ones that show up under Status > Services.
Given its newness, it will probably have bugs here and there.
It's simple to use. After install, go to Services > Service Watchdog, click +, pick a service to monitor, click Add. Every minute it will check to see if the services in the list are running, and if they aren't, they will be restarted.
You can reorder the services in the list in case there are dependencies to worry about (service X should be started before service Y)
Known Issues:
Does not work properly with dhcprelay6 service until version 1.1
Does not work properly with OpenVPN services until version 1.1 and a pfSense 2.1 snapshot from after Aug 28 9:00AM EST.
May show blank services or empty descriptions in some cases before version 1.2 -
I don't care what all those people say about you. You are the bomb…
Is this a 2.1 thing or works for 2.03 also? -
I don't care what all those people say about you. You are the bomb…
Is this a 2.1 thing or works for 2.03 also?2.1 only.
-
Yeah - You already posted that… I should learn to read.
Thanks again. I'll install and see how it goes. -
Yeah - You already posted that… I should learn to read.
Thanks again. I'll install and see how it goes.Maybe I did, or maybe I edited the OP to state that afterward. ;D
-
Great package, this is much appreciated.
A few cosmetic issues. I'm on the latest snapshot (AMD64) and both 0.1 and 0.2 don't show the Cron service in the dropdown, only a ":"
Also, iperf and pfflowd show the service name but not the description.
-
Odd, cron shows fine here for me on the latest snapshot (also amd64). I may have to blacklist cron since it doesn't really make sense for it to be handled via this script. (if cron isn't running, the script would never run)
I haven't tried many packages. I'll have to install them and see what is different there.
-
Odd, cron shows fine here for me on the latest snapshot (also amd64).
Ditto, and yeah, cron makes no sense there. :D
As for packages, seems like almost all of them are missing descriptions. (gwled, blinkled, nut, darkstat). The only thing I have installed and can see the description is unbound.
-
The packages apparently aren't setting their own/proper service description. The status page pulled their package descriptions instead. I just pushed a fix for that (and to skip cron and empty services)
-
Looks good - I can stop services from status services, and they spring back to life 1 minute later. Looking forward to trying listing multiple OpenVPN servers (a site-to-site and a road-warrior) once the necessary new snapshot comes.
One thought - during the boot process cron is configured/started. The service watchdog job might run while the boot is still doing more stuff? If so, it should not really do anything, as the boot might still be getting things up and running. Should the code check for if $g['booting'] and bail out in that case? -
Yeah you're right. I just pushed a new rev to make it do nothing if it's booting.
-
You can as well blacklist unbound I'd say, it has its own "watchdog" script - /usr/local/bin/unbound_monitor.sh. Though, not completely sure whether we'd not be better off dropping the looping shell script from unbound and using this package instead. :)
-
Squid also has it own sqp_monitor process. That function could now be done by this package and the special sqp_monitor code removed, if anyone cares or thinks it is a good idea.
-
Just a quick thought…Snort running a heavy set of rules can take minutes to start and running this every minute could cause Snort to start multiple times... Would it be a thing to make a 5 minute penalty period after a boot before the script begins to monitor packages??
-
2.1-RC1 (i386)
built on Wed Aug 28 16:55:08 EDT 2013
FreeBSD 8.3-RELEASE-p10I created 2 OpenVPN servers on a test system. They both appear in the dropdown list of services to add for Service Watchdog. After adding the 1st server, the 2nd server no longer appears in the dropdown list, so I can't add it as well.
I stopped NTPD and both OpenVPN server services from status services. Waited a few minutes. NTPDÂ and Test Server 1 restarted.
Does the dropdown list need a bit more tweaking to allow multiple individual OpenVPN servers to be added to the watch list?
-
Yeah you're right. I just pushed a new rev to make it do nothing if it's booting.
We also have situations where the boot script itself gets "killed: out of swap space". In that case, /var/run/bootup file nevers gets removed, so it always looks like the system is booting. I submitted pull requests so that Service Watchdog will start doing its thing anyway 15 minutes after boot time. See what you think.
I engineered it to put a new function get_uptime_sec into the base system for the benefit of anything that cares to call it. But that means that the base system change has to go into 2.1 branch, and people have to have it in their snapshot to use my changes to Service Watchdog. So feel free to engineer it however… -
As I mentioned on the pull request, that isn't a good workaround. There are other things that will break if that flag file is left around too long, and the package shouldn't have to care about that.
It would be better to find a way to clean that up automatically in the base system, but that isn't really a discussion for this thread since it's unrelated.
-
Just a quick thought…Snort running a heavy set of rules can take minutes to start and running this every minute could cause Snort to start multiple times... Would it be a thing to make a 5 minute penalty period after a boot before the script begins to monitor packages??
The "snort" binary would be in the list and it should show that it's running as far as this check is concerned.
That said, it probably would not work right with snort anyhow, since an instance for one interface would die and this would never know, because of how snort handles its instances. It would only show it down if all instances of snort were dead.
-
2.1-RC1 (i386)
built on Wed Aug 28 16:55:08 EDT 2013
FreeBSD 8.3-RELEASE-p10I created 2 OpenVPN servers on a test system. They both appear in the dropdown list of services to add for Service Watchdog. After adding the 1st server, the 2nd server no longer appears in the dropdown list, so I can't add it as well.
I stopped NTPD and both OpenVPN server services from status services. Waited a few minutes. NTPDÂ and Test Server 1 restarted.
Does the dropdown list need a bit more tweaking to allow multiple individual OpenVPN servers to be added to the watch list?
Yes that still needs some work.
-
OpenVPN and captive portal instance matching should be fixed in 1.4, up now.
