New Package: Service Watchdog

doktornotor

Thanks again for the work on this, this feature hopefully should get to the base install once the whole stuff gets polished.

jimp

@doktornotor:

Thanks again for the work on this, this feature hopefully should get to the base install once the whole stuff gets polished.

Seems better as a package for me. Not everyone needs/wants it, and it can react to changes faster as a package. After it's fairly well set it may not change much, but it still seems like a better fit as an add-on.

kejianshi

There are certain conditions that, in the past, have meant that raccoon was essentially offline (not about to connect anyone properly), even though the process still shows as running.

If all those conditions haven't been cleared up, any chance to use this package to restart it when raccoon goes buggy?

jimp

Not likely, if the process is running this would believe it to be up.

That kind of check would add a whole mess of code that would be irrelevant to anything else it does. Seems maybe maybe a better fit as some sort of dedicated racoon watchdog that is capable of more than a running/not running check.

phil.davis

Another little tweak:

function is_service_enabled($service_name)

servicewatchdog_check_services() could check is_service_enabled and only bother to try and start it if it is both enabled and not running.
It is probably best to allow people to add whatever services they like to the Service Watchdog watch list, as it is now. Then they can be in the list ready to be watched, even if they happen to be disabled at any particular time.

phil.davis

@jimp:

OpenVPN and captive portal instance matching should be fixed in 1.4, up now.

I can "watchdog" multiple OpenVPN instances now, and Watchdog restarts them if I stop them. A great thing.

jimp

@phil.davis:

Another little tweak:
function is_service_enabled($service_name)
servicewatchdog_check_services() could check is_service_enabled and only bother to try and start it if it is both enabled and not running.
It is probably best to allow people to add whatever services they like to the Service Watchdog watch list, as it is now. Then they can be in the list ready to be watched, even if they happen to be disabled at any particular time.

I can add that but that function does only work for packages, not for base system services. And even then, only packages that actually support an enable option using the exact option name for which that function checks.

Seems safer to put the burden on the user to only watch services they know they need to stay active.

Nachtfalke

Hi,

I did not install this package until now but when reading the thread I was asking if it could be possible to make an GUI option to say "if service isn't running for 3 x 1min then try to restart". Something similar like apinger for the WAN interfaces to check if a gateway is down.

So if there is a service not running and the package detects that then it will count 1.
If the same service isn't running one minute later then counts +1.
And if it is not running another minute later then restarts the service because the service was detected as not running three minutes after another.

If this value would be configurable for every service I think this could be helpful.

jimp

It might be nice but it would probably triple the size of the code if not more. I'm trying to keep it simple.

It's not a persistent daemon, so it would have to somehow store the values of those checks somewhere, read them in, increment them, etc, etc. Not trivial to accomplish. Also that would require making an edit screen rather than an add screen, which makes it a bit different to work with, even more code, etc.

webdawg

Can you explain how this plugin works?

How does it monitor and start processes?

Is there any reason it would still be working after I completely removed the package from pfsense?

jimp

@webdawg:

Can you explain how this plugin works?

It uses the functions built into pfSense to check the status and control services.

@webdawg:

How does it monitor and start processes?

It sets up a cron job that runs once per minute to inspect the service status, and if it detects a monitored service as down, it restarts it.

@webdawg:

Is there any reason it would still be working after I completely removed the package from pfsense?

Only if somehow its files and cron job were not removed.

Keep in mind some other packages like squid have their own monitoring which is not related to this package.

webdawg

Can I ask what account that the cronjob is created under?

jimp

There is no concept of that. If you want to view the cron job, install the Cron package or inspect the config.xml file. It's probably best to start your own thread for help diagnosing your actual issue rather than approaching it this way.

webdawg

No, this is good!

I forgot how pfSense did it. That is all I needed, thanks.

niccarp89

Hi friends.

I just want to ask if its possible some way to add the redis server to the watchdog list. Because i have installed the ntopng package and redis server keep stop it alone.
In the watchdog i config ntopng but when redis its down the ntopng doesn start.

Thanks

–----------
EDIT: At the momenti did it with a script and crontab checking the redis pid, also the directory /var/run/redis/redis.pid doesnt exists so maybe that was making errors.