PfSense 2.3 Check_mk working with xinetd



  • As my company uses check_mk for monitoring  pfsense, it is frustrating that the package for ckeck_mk has been removed. To be honest, the old package didn't work without manual modification, so it is understandable that is was removed.

    I've managed to get check_mk working with the pfsense 2.3 implementation of xinetd, which survives reboots, and allows config changed without rebooting the firewall..

    As this is a manual installation, you are advised that following the information below at your own risk.

    Before you install check_mk, you first need to understand how xinetd has been implemented in pfsense 2.3.

    The config file stored in /var/etc/xinetd.conf is regenerated every time you perform a filter reload. This means everytime you change a NAT rule, a filewall rule, change the TFTP config, etc… the xinetd.conf file is deleted, regenerated and then xinetd is restarted. The consequences of this is any manual change to the xinetd.conf get wiped out.

    My intention is to try and write a package for check_mk that makes the changes below  automatically, but it does depend on whether pfsense will change the way it has implemented xinetd in 2.3

    So, to allow us to make make changes to the xinetd.conf file, we need to modify the filter reload process.

    The way I have chosen to do this is to modify the function, in /etc/inc/filter.inc that generates the xinetd.conf file, to also include xinetd config files from another directory. This will allow config changes to individual xinetd services, without having to modify the /etc/inc/filter.inc everytime.

    1. First we need to make sure the check_mk agent can run without modifying the agent executable each time; for this we need to install bash.
    pkg install bash
    
    1. Next we need to create a location to store the check_mk_agent executable, and a location to store the xinetd config. I've chosen to use /opt as it is less likely to be deleted during an upgrade.
    
    mkdir -p /opt/bin
    mkdir -p /opt/etc/xinetd.d
    
    
    1. Download the latest version of the check_mk_agent.freebsd from the check_mk public git.

    Place it is /opt/bin/check_mk_agent

    http://git.mathias-kettner.de/git/?p=check_mk.git;a=blob_plain;f=agents/check_mk_agent.freebsd;hb=HEAD

    1. Make it executable
    chmod +x /opt/bin/check_mk_agent
    
    1. At this point you should be able to run the agent and see the check_mk output
    /opt/bin/check_mk_agent
    
    1. To able to use xinetd we need to create a new config file with the service details.

    Please edit this file to suit your needs.

    vi /opt/etc/xinetd.d/check_mk
    
    
    # +------------------------------------------------------------------+
    # |             ____ _               _        __  __ _  __           |
    # |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
    # |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
    # |           | |___| | | |  __/ (__|   <    | |  | | . \            |
    # |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
    # |                                                                  |
    # | Copyright Mathias Kettner 2014             mk@mathias-kettner.de |
    # +------------------------------------------------------------------+
    #
    # This file is part of Check_MK.
    # The official homepage is at http://mathias-kettner.de/check_mk.
    #
    # check_mk is free software;  you can redistribute it and/or modify it
    # under the  terms of the  GNU General Public License  as published by
    # the Free Software Foundation in version 2.  check_mk is  distributed
    # in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
    # out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
    # PARTICULAR PURPOSE. See the  GNU General Public License for more de-
    # ails.  You should have  received  a copy of the  GNU  General Public
    # License along with GNU Make; see the file  COPYING.  If  not,  write
    # to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
    # Boston, MA 02110-1301 USA.
    
    service check_mk
    {
    	type           = UNLISTED
    	port           = 6556
    	socket_type    = stream
    	protocol       = tcp
    	wait           = no
    	user           = root
    	server         = /opt/bin/check_mk_agent
    
    	# If you use fully redundant monitoring and poll the client
    	# from more then one monitoring servers in parallel you might
    	# want to use the agent cache wrapper:
    
    	#server         = /usr/bin/check_mk_caching_agent
    
    	# configure the IP address(es) of your Nagios server here:
    	#only_from      = 127.0.0.1 10.0.20.1 10.0.20.2
    
    	# Don't be too verbose. Don't log every check. This might be
    	# commented out for debugging. If this option is commented out
    	# the default options will be used for this service.
    	log_on_success =
    
    	disable        = no
    }
    
    
    1. Now we need to modify the filter reload function to include our config files.
    vi /etc/inc/filter.inc
    

    You need to find the line which begins fclose($xinetd_fd);
    This is the last part of the xinetd.conf file generation function. We need to insert our changes just above this line, it will need to look like this:

    
            fwrite($xinetd_fd, "includedir /opt/etc/xinetd.d");
    
            fclose($xinetd_fd);             // Close file handle
    
    
    1. That should be it to get xinetd to pick up the changes, you now need to manually initiate a filter reload.

    Status / Filter Reload / Reload Filters

    You can check the System Logs to see if xinetd has picked up the new service; you should see something like below.

    
    May 9 11:41:57	check_reload_status		Reloading filter
    May 9 11:41:58	xinetd	8327	Starting reconfiguration
    May 9 11:41:58	xinetd	8327	Swapping defaults
    May 9 11:41:58	xinetd	8327	readjusting service 6969-udp
    May 9 11:41:58	xinetd	8327	readjusting service check_mk
    May 9 11:41:58	xinetd	8327	Reconfigured: new=0 old=2 dropped=0 (services)
    
    
    1. By default your LAN interface will allow access to port 6556, if you need access on your WAN, you can create a firewall or NAT rule and pass 6556 to localhost.

    Do let me know if you have any feedback about this, or if it doesn't work.

    Joe



  • It works very well!



  • Thanks, good to know.



  • Hi  joeclifford,
    I had problem with step 5, I could not install libstatgrab.
    The pfsense not find the package.
    Would you help me?



  • The check_mk_agent will just output an empty statgrab section is libstatgrab isn't installed.

    By default pfsense 2.3 doesn't pull packages from the FreeBSD repo.

    You can install it manually if you want to; you can download the latest build from the FreeBSD Ports Monitoring System.

    http://portsmon.freebsd.org/portoverview.py?category=devel&portname=libstatgrab

    Just grab the package for your processor type. E.g, for pfsense 2.3 x64;

    
    curl --remote-name "http://pkg.freebsd.org/freebsd:10:x86:64/latest/All/libstatgrab-0.91.txz"
    pkg install libstatgrab-0.91.txz
    
    

    You might have to restart the firewall once you've installed statgrab to be able to use it.



  • Worked perfectly.
    Thanks for the help there, man.
    I will continue the following steps …

    :D :D :D



  • Like previous post, it works so good
    Thx



  • Tested on 2.3.1
    works !! :)

    Thanks



  • All worked then I came back to work the next week and found the agent is unreachable. I went through the steps againt and found that the filter file changed back to default settings and is missing the line:
    fwrite($xinetd_fd, "includedir /opt/etc/xinetd.d");

    If I re-add the line again, the agent works again. How do I make that filter file save the changes persistently through a reboot?



  • Thank you, worked like a charm.



  • After a update to 2.3.1_1 the filter.inc have to edit again.
    and a filter reload.



  • Make it update-persistent

    Problem: After an update the filter.inc has to be modified again.
    Solution: Modify the /etc/inc/filter.inc-file automatically via cron-Script.

    A) Create the cron-script

    vi /opt/filter_check_mk_cron

    #!/bin/sh

    grep includedir /etc/inc/filter.inc
    if [ $? -eq 0 ]
    then
            exit 0
    else
            awk '/Close file handle/{print "fwrite($xinetd_fd, "includedir /opt/etc/xinetd.d");"}1' /etc/inc/filter.inc > /etc/inc/filter.inc.temp
            mv /etc/inc/filter.inc.temp /etc/inc/filter.inc
    fi
    exit 0

    B) Make it executable

    chmod +x /opt/filter_check_mk_cron

    C) Create the cron-job

    In the web-ui go to
    System -> Package Manager
    If cron is not installed -> install

    Then open Services -> Cron and click on Add

    */15 * * * * root /opt/filter_check_mk_cron

    This will check the config-file every 15 Minutes

    Somebody might want to create an official package by now? :)



  • FJerusalem,

    Good idea to cron job a script to add in the needed line to the filter.inc. It might be worth modifying your script so it finds the line based on the fclose statement, rather than the comment. It's unlikely, but if and update adds an identical comment to the filter.inc file it could mess it up. I've used the below script.

    #!/bin/sh
    
    grep includedir /etc/inc/filter.inc
    if [ $? -eq 0 ]
    then
            exit 0
    else
            awk '/fclose\(\$xinetd_fd\)\;/{print "fwrite($xinetd_fd, \"includedir /opt/etc/xinetd.d\");"}1' /etc/inc/filter.inc > /etc/inc/filter.inc.temp
            mv /etc/inc/filter.inc.temp /etc/inc/filter.inc
    fi
    exit 0
    


  • @MontanaIce:

    All worked then I came back to work the next week and found the agent is unreachable. I went through the steps againt and found that the filter file changed back to default settings and is missing the line:
    fwrite($xinetd_fd, "includedir /opt/etc/xinetd.d");

    If I re-add the line again, the agent works again. How do I make that filter file save the changes persistently through a reboot?

    Hmm, I haven't yet had the filter.inc file revert on just a reboot, it normally only happens when there is an update that overrides the filter.inc file. Like this bug fix introduced in 2.3.1 . https://redmine.pfsense.org/issues/6307

    Are you running the nano version of pfsense? Pre 2.3.1 the filesystem is set to read only by default.

    Thankfully, FJerusalem's script will make the changes for you.



  • Thank you very much for the manual solution. Any intentions to continue to develop the package?



  • Trying to get this running on pfSense 2.3.3 for external monitoring and I am running into a problem.

    When I modify filter.inc and then reload filter I get this message in the System Log
    "Unable to read included directory: /opt/etc/xinetd.d [file=/var/etc/xinetd.conf] [line=1]"

    The /var/etc/xinetd.conf files contains
    includedir /opt/etc/xinetd.d

    I am not sure if I have made an error, or something otherwise is happening.



  • Thanks for a working Check_MK Agent on pfSense!

    I've written a simple local check for Check_MK to check whether a new version of pfSense is available. Maybe someone is interested in it.

    Create a new file (I've named it "check_version.php") in your $LOCALDIR of Check_MK (which is "$MK_LIBDIR/local") containing the following content:

    #! /usr/bin/env php
    include "pkg-utils.inc";
    
    $info = get_system_pkg_version();
    
    if ($info["installed_version"] == $info["version"]) {
        echo "0 pfSense_Version - No new update available\n";
    } else {
        echo "1 pfSense_Version - Version " . $info["version"] . " available\n";
    }
    

    Make it executable: chmod +x check_version.php

    You probably don't want to check for a new version on every run of check_mk_agent, so move the script into a subfolder containing the run interval as the name. I've saved the script to $LOCALDIR/3600/check_update.php so it will be executed every hour.



  • Thank you for the guide!

    Everything it still works for me under 2.4rc - just remember to grab a new libstatgrab-0.91.txz for Freebsd11.

    Best, Bernd



  • I've also updated pfSense (in my case the final release of version 2.4) and can confirm it still works. I didn't had to grab a new libstatgrab, it just works for me out of the box (I only had to add the additional include in /etc/inc/filter.inc just like on every update).



  • Hey guys

    I just updated to 2.4 and started getting these errors:

    **EDIT / SOLVED

    Needed to update libstatgrab
    fetch http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/libstatgrab-0.91.txz && pkg install libstatgrab-0.91.txz



  • After an upgrade to 2.4(.1) I seem to be getting errors now with the Memory Used plugin: UNKNOWN - check failed - please submit a crash report!
    (The crash report says: KeyError: 'MemTotal')

    Agent output:
    Relevant Check_MK output:

    <<<mem>>>
    SwapTotal: 782336 kB
    SwapFree: 782336 kB</mem>
    ```(No MemTotal line)
    
    My copy of the agent includes a <<<mem>>> section:
    

    Memory Usage

    currently we'll need sysutils/muse for this.

    if [ -x /usr/local/bin/muse ]
    then
    echo '<<<mem>>>'

    yes, i don't know sed well.

    muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'
    swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'
    fi</mem>

    
    I've installed muse with 'fetch http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/muse-0.2.txz && pkg install muse-0.2.txz'
    
    But 'muse -k' outputs:
    

    kvm_open: kvm_nlist: No such file or directory
    kvm_open: kvm_nlist: No such file or directory
    kvm_open: kvm_nlist: No such file or directory
    kvm_open: kvm_nlist: No such file or directory
    kvm_nlist failed

    
    Possibly the <<<mem>>> code no longer works?  It looks like it used to be in check_mk_agent.freebsd but is not there now.
    
    EDIT: I realised we are using an old, customised version of the agent.  I'm not willing to update wholesale at this point, but this seems to work now, from a less old version of the agent.  It also means we don't have to install muse now.
    

    Memory Usage. First we try statgrab, since

    muse does not seem to support >4GB

    if [ -x /usr/local/bin/statgrab ] ; then
        echo '<<<mem>>>'
        statgrab -K mem.total mem.free swap.total swap.free |
        awk '{gsub(/swap./,"Swap");}{gsub(/mem./,"Mem");}{gsub(/tot/,"Tot");}{gsub(/free/,"Free");} { print $1":  "$3" kB"}'
    elif [ -x /usr/local/bin/muse ] ; then
        echo '<<<mem>>>'
        muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'
        swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'
    fi</mem></mem>



  • Just update /opt/bin/check_mk_agent with the newest version available from http://git.mathias-kettner.de/git/?p=check_mk.git;a=blob_plain;f=agents/check_mk_agent.freebsd;hb=HEAD and install libstatgrab for the memory display to work (fetch http://pkg.freebsd.org/FreeBSD:11:amd64/latest/All/libstatgrab-0.91.txz && pkg install libstatgrab-0.91.txz)



  • Didn't work for me (pfSense 2.4.2-RELEASE-p1 (amd64)). I did all steps without any problems, but after reloading the filters, nothing happens. Nothing with xinetd or check_mk in the system logs and I can't to "nc localhost 6556" or "telnet localhost 6556".



  • @Overlord:

    Didn't work for me (pfSense 2.4.2-RELEASE-p1 (amd64)). I did all steps without any problems, but after reloading the filters, nothing happens. Nothing with xinetd or check_mk in the system logs and I can't to "nc localhost 6556" or "telnet localhost 6556".

    Does the Check_MK Agent work if called from the terminal?

    Try to execute "/opt/bin/check_mk_agent" on your pfSense host.

    And what exact error are you getting if you try to reach port 6556? Connection refused? Connection timed out?

    A filter reload should show nothing special (i.e. does not contain anything with check_mk). But you should see something like "readjusting service check_mk" in "Status -> System Logs -> System -> General" after reloading the filters.

    After a filter reload, the file "/var/etc/xinetd.conf" should contain a line "includedir /opt/etc/xinetd.d". /opt/etc/xinetd.d should be a directory containing your xinetd config file for Check_MK.

    Also make sure, xinetd is running: "ps ax | grep xinetd" should return something like "/usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid"



  • Worked perfectly for me on 2.4.2!

    Really appreciate the instructions!

    Thank you!!!!!!!



  • @Programie:

    @Overlord:

    Didn't work for me (pfSense 2.4.2-RELEASE-p1 (amd64)). I did all steps without any problems, but after reloading the filters, nothing happens. Nothing with xinetd or check_mk in the system logs and I can't to "nc localhost 6556" or "telnet localhost 6556".

    Does the Check_MK Agent work if called from the terminal?

    Try to execute "/opt/bin/check_mk_agent" on your pfSense host.

    And what exact error are you getting if you try to reach port 6556? Connection refused? Connection timed out?

    A filter reload should show nothing special (i.e. does not contain anything with check_mk). But you should see something like "readjusting service check_mk" in "Status -> System Logs -> System -> General" after reloading the filters.

    After a filter reload, the file "/var/etc/xinetd.conf" should contain a line "includedir /opt/etc/xinetd.d". /opt/etc/xinetd.d should be a directory containing your xinetd config file for Check_MK.

    Also make sure, xinetd is running: "ps ax | grep xinetd" should return something like "/usr/local/sbin/xinetd -syslog daemon -f /var/etc/xinetd.conf -pidfile /var/run/xinetd.pid"

    HI,
    I have also a problem with check_mk and xinetd.

    check_mk works from the terminal.  I cannot find a xinetd-process on my box. Also, after reloading the filters (with the manually added line to include /opt/etc/xinetd.d) I only find

    Feb 24 16:58:45 check_reload_status Reloading filter
    in the logs.    /var/etc/xinetd.conf is empty

    ps doesn't show a xinetd.

    How do I start xinetd on pfsense? Shouldn*t it be running by default?

    Any help appreciated,
    Karl



  • For those interested, I've consolidated the instructions into a new thread, and added a line in the update-persistence script to automatically reload the filters after making the change so the check_mk service is started.

    https://forum.netgate.com/topic/131967/check_mk-on-pfsense-2-4-w-update-persistence