Problem with check_mk packet



  • hello

    I need monitoring pfSense via check_mk.

    I installed the package but monitoring can not get any information.

    port 6556 is listening usually

    ip Monitoring Server reaches the pfSense …

    Any additional detail to work?



  • I think we found the problem:

    There's already a bug ticket:
    https://redmine.pfsense.org/issues/4084

    "The official Check_mk agent package doesn't work because the shebang line at the top of /usr/local/bin/check_mk_agent is wrong: it needs to be #!/usr/local/bin/bash, not #!/bin/bash."

    Maybe we can contact the maintainer to fix this.
    Who can help here ?



  • @tracer:

    I think we found the problem:

    There's already a bug ticket:
    https://redmine.pfsense.org/issues/4084

    "The official Check_mk agent package doesn't work because the shebang line at the top of /usr/local/bin/check_mk_agent is wrong: it needs to be #!/usr/local/bin/bash, not #!/bin/bash."

    Maybe we can contact the maintainer to fix this.
    Who can help here ?

    It is the smallest problem with this package.
    for his work needed statgrab.
    Removed from the system inetd, misconfiguration of which lay in /etc/inetd.conf
    And he run the script in /etc/rc.d/inetd: not found



  • ok, you might be right …
    But mine worked after changing the bash line.
    Are you saying that there a problems which prevents the package from working or are we missing infos...



  • In 2.2-RELEASE the package doesn't work.
    In the "Sockets" tab, there's no open port for the check_mk agent (default 6556).

    Also I've seen this error in the logs, I'm not 100% sure that's from check_mk_agent, but I think so…

    Jan 25 23:46:49 php-fpm[96236]: /pkg_mgr_install.php: The command '/etc/rc.d/inetd restart' returned exit code '127', the output was '/etc/rc.d/inetd: not found'
    Jan 25 23:46:49 check_reload_status: Syncing firewall
    Jan 25 23:46:49 php-fpm[96236]: /pkg_mgr_install.php: Successfully installed package: Check_mk agent.

    Best, mete



  • Yes, inetd is installed and running…

    /usr/sbin/inet
    

    But start scripts are missig???

    /etc/rc.d/inetd start
    
    service inetd start
    

    I found this…
    https://mathias-kettner.com/checkmk_freebsd.html



  • To get data for monitoring via port 6556 you have to do the following.
    Its just a workaroud because every time inetd restarts the configfile /var/etc/inetd.conf will be overwritten.
    I dont know yet how i can change this.

    First of all be sure that the check_mk_agent is running probably and than copy this line:

    check_mk  stream  tcp nowait  root  /usr/local/bin/check_mk_agent check_mk_agent
    

    into /var/etc/inetd.conf

    then you have to kill your running inetd process and start it again with this command:

    /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    

    Now there should be an open socket and the data are available for your monitoring server.



  • Her i post the missing script, but this is not the full solution.

    1. create new file /etc/rc.d/inetd
    2. insert code
    3. chmod +x /etc/rc.d/inetd

    Now service inetd start,stop,status is working

    
    #!/bin/sh
    #
    # $FreeBSD$
    #
    
    # PROVIDE: inetd
    # REQUIRE: DAEMON LOGIN cleanvar
    # KEYWORD: shutdown
    
    . /etc/rc.subr
    
    name="inetd"
    rcvar="inetd_enable"
    command="/usr/sbin/${name}"
    pidfile="/var/run/${name}.pid"
    required_files="/etc/${name}.conf"
    extra_commands="reload"
    
    load_rc_config $name
    run_rc_command "$1"
    
    


  • @WhiteSaxo: I follow your post, but it is not working.

    I have to change the first line from file /usr/local/bin/check_mk_agent from #!/bin/bash intro #!/bin/sh

    Now, check_mk_agent report no error message, but is not working yet.

    The check_mk agen (freebsd Version: 1.2.7i1) comes from:

    http://git.mathias-kettner.de/git/?p=check_mk.git;a=blob_plain;f=agents/check_mk_agent.freebsd;hb=HEAD

    Recently, some modification were carried out.

    http://git.mathias-kettner.de/git/?p=check_mk.git&a=search&h=HEAD&st=commit&s=freebsd

    I create a new installion with pfsense 2.1.5 and check_mk agent, but it even not working.

    At boot time, i get errors from /usr/local/pkg/checkmk.inc line 107.

    And i fount errors into /var/log/system.log.



  • Did you copy

    check_mk  stream  tcp nowait  root  /usr/local/bin/check_mk_agent check_mk_agent
    

    in the correct inetd.conf?

    And did you add

    check_mk        6556/tcp   #check_mk agent
    

    in /etc/services?



  • The file /usr/local/bin/check_mk_agent is wrong. pfsense install it direkt from git. Look into my older post.

    I replace check_mk_agent with a older version and it works well.

    @WhiteSaxo: Can you post your older file? This version is working well with pfsense 2.2 and pfsense 2.1.5



  • Ah okay good to know.

    Here is my working check_mk_agent

    #!/bin/sh
    # +------------------------------------------------------------------+
    # |             ____ _               _        __  __ _  __           |
    # |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
    # |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
    # |           | |___| | | |  __/ (__|   <    | |  | | . \            |
    # |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
    # |                                                                  |
    # | Copyright Mathias Kettner 2014             mk@mathias-kettner.de |
    # +------------------------------------------------------------------+
    #
    # This file is part of Check_MK.
    # The official homepage is at http://mathias-kettner.de/check_mk.
    #
    # check_mk is free software;  you can redistribute it and/or modify it
    # under the  terms of the  GNU General Public License  as published by
    # the Free Software Foundation in version 2.  check_mk is  distributed
    # in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
    # out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
    # PARTICULAR PURPOSE. See the  GNU General Public License for more de-
    # ails.  You should have  received  a copy of the  GNU  General Public
    # License along with GNU Make; see the file  COPYING.  If  not,  write
    # to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
    # Boston, MA 02110-1301 USA.
    
    # Author: Lars Michelsen <lm@mathias-kettner.de>#         Florian Heigl <florian.heigl@gmail.com>#           (Added sections: df mount mem netctr ipmitool)
    
    # NOTE: This agent has beed adapted from the Check_MK linux agent.
    #       The most sections are commented out at the moment because
    #       they have not been ported yet. We will try to adapt most
    #       sections to print out the same output as the linux agent so
    #       that the current checks can be used.
    
    # This might be a good source as description of sysctl output:
    # http://people.freebsd.org/~hmp/utilities/satbl/_sysctl.html
    
    # Remove locale settings to eliminate localized outputs where possible
    export LC_ALL=C
    unset LANG
    
    export MK_LIBDIR="/usr/lib/check_mk_agent"
    export MK_CONFDIR="/etc/check_mk"
    export MK_TMPDIR="/var/run/check_mk"
    
    # Make sure, locally installed binaries are found
    PATH=$PATH:/usr/local/bin
    
    # All executables in PLUGINSDIR will simply be executed and their
    # ouput appended to the output of the agent. Plugins define their own
    # sections and must output headers with '<<<' and '>>>'
    PLUGINSDIR=$MK_LIBDIR/plugins
    
    # All executables in LOCALDIR will by executabled and their
    # output inserted into the section <<<local>>>. Please refer
    # to online documentation for details.
    LOCALDIR=$MK_LIBDIR/local
    
    # close standard input (for security reasons) and stderr
    #if [ "$1" = -d ]
    #then
    #    set -xv
    #else
    #    exec /dev/null
    #fi
    
    # Runs a command asynchronous by use of a cache file
    
    echo '<<<check_mk>>>'
    echo Version: 1.2.7i1
    echo AgentOS: freebsd
    
    osver="$(uname -r)"
    is_jailed="$(sysctl -n security.jail.jailed)"
    
    # Partitionen (-P verhindert Zeilenumbruch bei langen Mountpunkten)
    # Achtung: NFS-Mounts werden grundsaetzlich ausgeblendet, um
    # Haenger zu vermeiden. Diese sollten ohnehin besser auf dem
    # Server, als auf dem Client ueberwacht werden.
    
    echo '<<<df>>>'
    # no special zfs handling so far, the ZFS.pools plugin has been tested to
    # work on FreeBSD
    if df -T > /dev/null ; then
        df -kTP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)'
    else
        df -kP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)' | awk '{ print $1,"ufs",$2,$3,$4,$5,$6 }'
    fi
    
    # Check NFS mounts by accessing them with stat -f (System
    # call statfs()). If this lasts more then 2 seconds we
    # consider it as hanging. We need waitmax.
    #if type waitmax >/dev/null
    #then
    #    STAT_VERSION=$(stat --version | head -1 | cut -d" " -f4)
    #    STAT_BROKE="5.3.0"
    #
    #    echo '<<<nfsmounts>>>'
    #    sed -n '/ nfs /s/[^ ]* \([^ ]*\) .*/\1/p' < /proc/mounts |
    #        while read MP
    #  do
    #   if [ $STAT_VERSION != $STAT_BROKE ]; then
    #      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" || \
    #    echo "$MP hanging 0 0 0 0"
    #   else
    #      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" && \
    #      printf '\n'|| echo "$MP hanging 0 0 0 0"
    #   fi
    #  done
    #fi
    
    # Check mount options.
    # FreeBSD doesn't do remount-ro on errors, but the users might consider
    # security related mount options more important.
    echo '<<<mounts>>>'
    mount -p -t ufs
    
    # processes including username, without kernel processes
    echo '<<<ps>>>'
    COLUMNS=10000
    if [ is_jailed = 0 ]; then
        ps ax -o state,user,vsz,rss,pcpu,command | sed -e 1d  -e '/\([^ ]*J\) */d' -e 's/*\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\2,\3,\4,\5) /'
    else
        ps ax -o user,vsz,rss,pcpu,command | sed -e 1d -e 's/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /'
    fi
    
    # Produce compatible load/cpu output to linux agent. Not so easy here.
    echo '<<<cpu>>>'
    echo `sysctl -n vm.loadavg | tr -d '{}'` `top -b -n 1 | grep -E '^[0-9]+ processes' | awk '{print $3"/"$1}'` `sysctl -n kern.lastpid` `sysctl -n hw.ncpu`
    
    # Calculate the uptime in seconds since epoch compatible to /proc/uptime in linux
    echo '<<<uptime>>>'
      up_seconds=$(( `date +%s` - `sysctl -n kern.boottime  | cut -f1 -d\, | awk '{print $4}'`))
    idle_seconds=$(ps axw | grep idle | grep -v grep | awk '{print $4}' | cut -f1 -d\: )
    
    # second value can be grabbed from "idle" process cpu time / num_cores
    echo "$idle_seconds $up_seconds"
    
    # Platten- und RAID-Status von LSI-Controlleren, falls vorhanden
    #if which cfggen > /dev/null ; then
    #   echo '<<<lsi>>>'
    #   cfggen 0 DISPLAY | egrep '(Target ID|State|Volume ID|Status of volume)[[:space:]]*:' | sed -e 's/ *//g' -e 's/:/ /'
    #fi
    
    # Multipathing is supported in FreeBSD by now
    # http://www.mywushublog.com/2010/06/freebsd-and-multipath/
    if kldstat -v | grep g_multipath > /dev/null ; then
        echo '<<<freebsd_multipath>>>'
        gmultipath status | grep -v ^Name
    fi
    
    # Soft-RAID
    echo '<<<freebsd_geom_mirrors>>>'
    gmirror status | grep -v ^Name
    
    # Performancecounter Kernel
    echo "<<<kernel>>>"
    date +%s
    forks=`sysctl -n vm.stats.vm.v_forks`
    vforks=`sysctl -n vm.stats.vm.v_vforks`
    rforks=`sysctl -n vm.stats.vm.v_rforks`
    kthreads=`sysctl -n vm.stats.vm.v_kthreads`
    echo "cpu" `sysctl -n kern.cp_time | awk ' { print $1" "$2" "$3" "$5" "$4 } '`
    echo "ctxt" `sysctl -n vm.stats.sys.v_swtch`
    echo "processes" `expr $forks + $vforks + $rforks + $kthreads`
    
    # Network device statistics (Packets, Collisions, etc)
    # only the "Link/Num" interface has all counters.
    echo '<<<lnx_if:sep(58)>>>'
    date +%s
    if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
        netstat -inb | egrep -v '(^Name|plip|enc|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    else
        # pad output for freebsd 7 and before
        netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
    fi
    
    # State of LSI MegaRAID controller via MegaCli.
    # To install: pkg install megacli
    if which MegaCli >/dev/null ; then
        echo '<<<megaraid_pdisks>>>'
        MegaCli -PDList -aALL -NoLog < /dev/null | egrep 'Enclosure|Raw Size|Slot Number|Device Id|Firmware state|Inquiry'
        echo '<<<megaraid_ldisks>>>'
        MegaCli -LDInfo -Lall -aALL -NoLog < /dev/null | egrep 'Size|State|Number|Adapter|Virtual'
        echo '<<<megaraid_bbu>>>'
        MegaCli -AdpBbuCmd -GetBbuStatus -aALL -NoLog < /dev/null | grep -v Exit
    fi
    
    # OpenVPN Clients. 
    # Correct log location unknown, sed call might also be broken
    if [ -e /var/log/openvpn/openvpn-status.log ] ; then
        echo '<<<openvpn_clients:sep(44)>>>'
        sed -n -e '/CLIENT LIST/,/ROUTING TABLE/p' < /var/log/openvpn/openvpn-status.log  | sed -e 1,3d -e '$d' 
    fi
    
    if which ntpq > /dev/null 2>&1 ; then
       echo '<<<ntp>>>'
       # remote heading, make first column space separated
       ntpq -np | sed -e 1,2d -e 's/^\(.\)/\1 /' -e 's/^ /%/'
    fi
    
    # Checks for cups monitoring
    #if which lpstat > /dev/null 2>&1; then
    #  echo '<<<cups_queues>>>'
    #  lpstat -p
    #  echo '---'
    #  for i in $(lpstat -p | grep -E "^(printer|Drucker)" | awk '{print $2}' | grep -v "@"); do
    #    lpstat -o "$i"
    #  done
    #fi
    
    # Heartbeat monitoring
    #if which cl_status > /dev/null 2>&1; then
    #  # Different handling for heartbeat clusters with and without CRM
    #  # for the resource state
    #  if [ -S /var/run/heartbeat/crm/cib_ro ]; then
    #    echo '<<<heartbeat_crm>>>'
    #    crm_mon -1 -r | grep -v ^$ | sed 's/^\s/_/g'
    #  else
    #    echo '<<<heartbeat_rscstatus>>>'
    #    cl_status rscstatus
    #  fi
    #
    #  echo '<<<heartbeat_nodes>>>'
    #  for NODE in $(cl_status listnodes); do
    #    if [ $NODE != $HOSTNAME ]; then
    #      STATUS=$(cl_status nodestatus $NODE)
    #      echo -n "$NODE $STATUS"
    #      for LINK in $(cl_status listhblinks $NODE 2>/dev/null); do
    #        echo -n " $LINK $(cl_status hblinkstatus $NODE $LINK)"
    #      done
    #      echo
    #    fi
    #  done
    #fi
    
    # Number of TCP connections in the various states
    echo '<<<tcp_conn_stats>>>'
    netstat -na | awk ' /^tcp/ { c[$6]++; } END { for (x in c) { print x, c[x]; } }'
    
    # Postfix mailqueue monitoring
    #
    # Only handle mailq when postfix user is present. The mailq command is also
    # available when postfix is not installed. But it produces different outputs
    # which are not handled by the check at the moment. So try to filter out the
    # systems not using postfix by searching for the postfix user.
    #
    # Cannot take the whole outout. This could produce several MB of agent output
    # on blocking queues.
    # Only handle the last 6 lines (includes the summary line at the bottom and
    # the last message in the queue. The last message is not used at the moment
    # but it could be used to get the timestamp of the last message.
    #if which mailq >/dev/null 2>&1 && getent passwd postfix >/dev/null 2>&1; then
    #  echo '<<<postfix_mailq>>>'
    #  mailq | tail -n 6
    #fi
    
    #Check status of qmail mailqueue
    #if type qmail-qstat >/dev/null
    #then
    #   echo "<<<qmail_stats>>>"
    #   qmail-qstat
    #fi
    
    # check zpool status
    #if [ -x /sbin/zpool ]; then
    #   echo "<<<zpool_status>>>"
    #   /sbin/zpool status -x | grep -v "errors: No known data errors"
    #fi
    
    # Memory Usage
    # currently we'll need sysutils/muse for this.
    if [ -x /usr/local/bin/muse ]
    then
    echo '<<<mem>>>'
    # yes, i don't know sed well.
    muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'
    swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'
    fi
    
    # Fileinfo-Check: put patterns for files into /etc/check_mk/fileinfo.cfg
    if [ -r "$MK_CONFDIR/fileinfo.cfg" ] ; then
        echo '<<<fileinfo:sep(124)>>>'
        date +%s
        stat -f "%N|%z|%m" $(cat "$MK_CONFDIR/fileinfo.cfg")
    fi</fileinfo:sep(124)></mem></zpool_status></qmail_stats></postfix_mailq></tcp_conn_stats></heartbeat_nodes></heartbeat_rscstatus></heartbeat_crm></cups_queues></ntp></openvpn_clients:sep(44)></megaraid_bbu></megaraid_ldisks></megaraid_pdisks></lnx_if:sep(58)></kernel></freebsd_geom_mirrors></freebsd_multipath></lsi></uptime></cpu></ps></mounts></nfsmounts></df></check_mk></local></florian.heigl@gmail.com></lm@mathias-kettner.de> 
    


  • Here is the solution: (works with pfsende 2.2 and 2.1.5)

    A) create inetd (only for pfsense 2.2)

    1. create new file /etc/rc.d/inetd
    2. insert code from my post
    3. run #chmod +x /etc/rc.d/inetd

    B) Replace check_mk_agent (older version is working well)

    replace code into file /usr/local/bin/check_mk_agent with post from WhiteSaxo

    C) Restart daemon

    service inetd restart



  • Just one little change to the check_mk_client script posted by WhiteSaxo above.

    I took the "date %s" out of the lnx_if section as my check_mk server was not picking up the interfaces. I also added lo0 to the exceptions, as it doesn't quite match up on the awk.

    So, it now reads

    echo '<<<lnx_if:sep(58)>>>'
    #date +%s
    if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
        netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t
    0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    else
        # pad output for freebsd 7 and before
        netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11
    " 0 0"}'
    fi</lnx_if:sep(58)>
    


  • Hi Roger and all.

    I add your modification but still dont get interfaces links, i notice that $osver variable its empty and i dont have defined over the network section. Anyone know if this its ok???



  • Hi, I possibly made a mistake with my cut-and-paste.

    
    #date +%s
    if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
        netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    else
        # pad output for freebsd 7 and before
        netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
    fi
    
    

    It's a line feed issue from my ssh session. Maybe that's the issue with you not getting interfaces.

    If not, what do you get from a "telnet host-ip 6556"?

    Roger



  • @Roger:

    Hi, I possibly made a mistake with my cut-and-paste.

    
    #date +%s
    if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
        netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
    else
        # pad output for freebsd 7 and before
        netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
    fi
    
    

    It's a line feed issue from my ssh session. Maybe that's the issue with you not getting interfaces.

    If not, what do you get from a "telnet host-ip 6556"?

    Roger

    Hi Roger.

    http://pastebin.com/U2yq2nxn

    Seems that <<<lnx_if:sep(58)>>>  section in the telnet its ok, but check_mk dont discover the interfaces.

    You have for example re0 re1 or whatever detected as a service in the check_mk interface??</lnx_if:sep(58)>



  • Hi

    Yes, I get re0 and re1 listed after a 'Full Scan', did you do a 'Full Scan' in check_mk ?

    My server version is the latest 1.2.4 version.

    Roger



  • @Roger:

    Hi

    Yes, I get re0 and re1 listed after a 'Full Scan', did you do a 'Full Scan' in check_mk ?

    My server version is the latest 1.2.4 version.

    Roger

    Perfect Roger, i only have to do a full scan.

    Thanks!


Log in to reply