Problem with check_mk packet

Augusto · Jan 7, 2015, 1:56 PM

hello

I need monitoring pfSense via check_mk.

I installed the package but monitoring can not get any information.

port 6556 is listening usually

ip Monitoring Server reaches the pfSense …

Any additional detail to work?

tracer · Jan 16, 2015, 5:28 PM

I think we found the problem:

There's already a bug ticket:
https://redmine.pfsense.org/issues/4084

"The official Check_mk agent package doesn't work because the shebang line at the top of /usr/local/bin/check_mk_agent is wrong: it needs to be #!/usr/local/bin/bash, not #!/bin/bash."

Maybe we can contact the maintainer to fix this.
Who can help here ?

DarkBeard · Jan 26, 2015, 1:55 PM

@tracer:

I think we found the problem:

There's already a bug ticket:
https://redmine.pfsense.org/issues/4084

"The official Check_mk agent package doesn't work because the shebang line at the top of /usr/local/bin/check_mk_agent is wrong: it needs to be #!/usr/local/bin/bash, not #!/bin/bash."

Maybe we can contact the maintainer to fix this.
Who can help here ?

It is the smallest problem with this package.
for his work needed statgrab.
Removed from the system inetd, misconfiguration of which lay in /etc/inetd.conf
And he run the script in /etc/rc.d/inetd: not found

tracer · Jan 26, 2015, 3:44 PM

ok, you might be right …
But mine worked after changing the bash line.
Are you saying that there a problems which prevents the package from working or are we missing infos...

mete · Jan 26, 2015, 9:40 PM

In 2.2-RELEASE the package doesn't work.
In the "Sockets" tab, there's no open port for the check_mk agent (default 6556).

Also I've seen this error in the logs, I'm not 100% sure that's from check_mk_agent, but I think so…

Jan 25 23:46:49 php-fpm[96236]: /pkg_mgr_install.php: The command '/etc/rc.d/inetd restart' returned exit code '127', the output was '/etc/rc.d/inetd: not found'
Jan 25 23:46:49 check_reload_status: Syncing firewall
Jan 25 23:46:49 php-fpm[96236]: /pkg_mgr_install.php: Successfully installed package: Check_mk agent.

Best, mete

MathiasM · Jan 29, 2015, 3:17 PM

Yes, inetd is installed and running…

/usr/sbin/inet

But start scripts are missig???

/etc/rc.d/inetd start

service inetd start

I found this…
https://mathias-kettner.com/checkmk_freebsd.html

WhiteSaxo · Jan 29, 2015, 10:49 PM

To get data for monitoring via port 6556 you have to do the following.
Its just a workaroud because every time inetd restarts the configfile /var/etc/inetd.conf will be overwritten.
I dont know yet how i can change this.

First of all be sure that the check_mk_agent is running probably and than copy this line:

check_mk  stream  tcp nowait  root  /usr/local/bin/check_mk_agent check_mk_agent

into /var/etc/inetd.conf

then you have to kill your running inetd process and start it again with this command:

/usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf

Now there should be an open socket and the data are available for your monitoring server.

MathiasM · Jan 30, 2015, 10:56 AM

Her i post the missing script, but this is not the full solution.

1. create new file /etc/rc.d/inetd
2. insert code
3. chmod +x /etc/rc.d/inetd

Now service inetd start,stop,status is working


#!/bin/sh
#
# $FreeBSD$
#

# PROVIDE: inetd
# REQUIRE: DAEMON LOGIN cleanvar
# KEYWORD: shutdown

. /etc/rc.subr

name="inetd"
rcvar="inetd_enable"
command="/usr/sbin/${name}"
pidfile="/var/run/${name}.pid"
required_files="/etc/${name}.conf"
extra_commands="reload"

load_rc_config $name
run_rc_command "$1"

MathiasM · Jan 30, 2015, 11:46 AM

@WhiteSaxo: I follow your post, but it is not working.

I have to change the first line from file /usr/local/bin/check_mk_agent from #!/bin/bash intro #!/bin/sh

Now, check_mk_agent report no error message, but is not working yet.

The check_mk agen (freebsd Version: 1.2.7i1) comes from:

http://git.mathias-kettner.de/git/?p=check_mk.git;a=blob_plain;f=agents/check_mk_agent.freebsd;hb=HEAD

Recently, some modification were carried out.

http://git.mathias-kettner.de/git/?p=check_mk.git&a=search&h=HEAD&st=commit&s=freebsd

I create a new installion with pfsense 2.1.5 and check_mk agent, but it even not working.

At boot time, i get errors from /usr/local/pkg/checkmk.inc line 107.

And i fount errors into /var/log/system.log.

WhiteSaxo · Jan 30, 2015, 12:05 PM

Did you copy

check_mk  stream  tcp nowait  root  /usr/local/bin/check_mk_agent check_mk_agent

in the correct inetd.conf?

And did you add

check_mk        6556/tcp   #check_mk agent

in /etc/services?

MathiasM · Jan 30, 2015, 1:17 PM

The file /usr/local/bin/check_mk_agent is wrong. pfsense install it direkt from git. Look into my older post.

I replace check_mk_agent with a older version and it works well.

@WhiteSaxo: Can you post your older file? This version is working well with pfsense 2.2 and pfsense 2.1.5

WhiteSaxo · Jan 30, 2015, 1:41 PM

Ah okay good to know.

Here is my working check_mk_agent

#!/bin/sh
# +------------------------------------------------------------------+
# |             ____ _               _        __  __ _  __           |
# |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
# |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
# |           | |___| | | |  __/ (__|   <    | |  | | . \            |
# |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
# |                                                                  |
# | Copyright Mathias Kettner 2014             mk@mathias-kettner.de |
# +------------------------------------------------------------------+
#
# This file is part of Check_MK.
# The official homepage is at http://mathias-kettner.de/check_mk.
#
# check_mk is free software;  you can redistribute it and/or modify it
# under the  terms of the  GNU General Public License  as published by
# the Free Software Foundation in version 2.  check_mk is  distributed
# in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
# out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
# PARTICULAR PURPOSE. See the  GNU General Public License for more de-
# ails.  You should have  received  a copy of the  GNU  General Public
# License along with GNU Make; see the file  COPYING.  If  not,  write
# to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
# Boston, MA 02110-1301 USA.

# Author: Lars Michelsen <lm@mathias-kettner.de>#         Florian Heigl <florian.heigl@gmail.com>#           (Added sections: df mount mem netctr ipmitool)

# NOTE: This agent has beed adapted from the Check_MK linux agent.
#       The most sections are commented out at the moment because
#       they have not been ported yet. We will try to adapt most
#       sections to print out the same output as the linux agent so
#       that the current checks can be used.

# This might be a good source as description of sysctl output:
# http://people.freebsd.org/~hmp/utilities/satbl/_sysctl.html

# Remove locale settings to eliminate localized outputs where possible
export LC_ALL=C
unset LANG

export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_TMPDIR="/var/run/check_mk"

# Make sure, locally installed binaries are found
PATH=$PATH:/usr/local/bin

# All executables in PLUGINSDIR will simply be executed and their
# ouput appended to the output of the agent. Plugins define their own
# sections and must output headers with '<<<' and '>>>'
PLUGINSDIR=$MK_LIBDIR/plugins

# All executables in LOCALDIR will by executabled and their
# output inserted into the section <<<local>>>. Please refer
# to online documentation for details.
LOCALDIR=$MK_LIBDIR/local

# close standard input (for security reasons) and stderr
#if [ "$1" = -d ]
#then
#    set -xv
#else
#    exec /dev/null
#fi

# Runs a command asynchronous by use of a cache file

echo '<<<check_mk>>>'
echo Version: 1.2.7i1
echo AgentOS: freebsd

osver="$(uname -r)"
is_jailed="$(sysctl -n security.jail.jailed)"

# Partitionen (-P verhindert Zeilenumbruch bei langen Mountpunkten)
# Achtung: NFS-Mounts werden grundsaetzlich ausgeblendet, um
# Haenger zu vermeiden. Diese sollten ohnehin besser auf dem
# Server, als auf dem Client ueberwacht werden.

echo '<<<df>>>'
# no special zfs handling so far, the ZFS.pools plugin has been tested to
# work on FreeBSD
if df -T > /dev/null ; then
    df -kTP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)'
else
    df -kP -t ufs | egrep -v '(Filesystem|devfs|procfs|fdescfs|basejail)' | awk '{ print $1,"ufs",$2,$3,$4,$5,$6 }'
fi

# Check NFS mounts by accessing them with stat -f (System
# call statfs()). If this lasts more then 2 seconds we
# consider it as hanging. We need waitmax.
#if type waitmax >/dev/null
#then
#    STAT_VERSION=$(stat --version | head -1 | cut -d" " -f4)
#    STAT_BROKE="5.3.0"
#
#    echo '<<<nfsmounts>>>'
#    sed -n '/ nfs /s/[^ ]* \([^ ]*\) .*/\1/p' < /proc/mounts |
#        while read MP
#  do
#   if [ $STAT_VERSION != $STAT_BROKE ]; then
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" || \
#    echo "$MP hanging 0 0 0 0"
#   else
#      waitmax -s 9 2 stat -f -c "$MP ok %b %f %a %s" "$MP" && \
#      printf '\n'|| echo "$MP hanging 0 0 0 0"
#   fi
#  done
#fi

# Check mount options.
# FreeBSD doesn't do remount-ro on errors, but the users might consider
# security related mount options more important.
echo '<<<mounts>>>'
mount -p -t ufs

# processes including username, without kernel processes
echo '<<<ps>>>'
COLUMNS=10000
if [ is_jailed = 0 ]; then
    ps ax -o state,user,vsz,rss,pcpu,command | sed -e 1d  -e '/\([^ ]*J\) */d' -e 's/*\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\2,\3,\4,\5) /'
else
    ps ax -o user,vsz,rss,pcpu,command | sed -e 1d -e 's/ *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *\([^ ]*\) */(\1,\2,\3,\4) /'
fi

# Produce compatible load/cpu output to linux agent. Not so easy here.
echo '<<<cpu>>>'
echo `sysctl -n vm.loadavg | tr -d '{}'` `top -b -n 1 | grep -E '^[0-9]+ processes' | awk '{print $3"/"$1}'` `sysctl -n kern.lastpid` `sysctl -n hw.ncpu`

# Calculate the uptime in seconds since epoch compatible to /proc/uptime in linux
echo '<<<uptime>>>'
  up_seconds=$(( `date +%s` - `sysctl -n kern.boottime  | cut -f1 -d\, | awk '{print $4}'`))
idle_seconds=$(ps axw | grep idle | grep -v grep | awk '{print $4}' | cut -f1 -d\: )

# second value can be grabbed from "idle" process cpu time / num_cores
echo "$idle_seconds $up_seconds"

# Platten- und RAID-Status von LSI-Controlleren, falls vorhanden
#if which cfggen > /dev/null ; then
#   echo '<<<lsi>>>'
#   cfggen 0 DISPLAY | egrep '(Target ID|State|Volume ID|Status of volume)[[:space:]]*:' | sed -e 's/ *//g' -e 's/:/ /'
#fi

# Multipathing is supported in FreeBSD by now
# http://www.mywushublog.com/2010/06/freebsd-and-multipath/
if kldstat -v | grep g_multipath > /dev/null ; then
    echo '<<<freebsd_multipath>>>'
    gmultipath status | grep -v ^Name
fi

# Soft-RAID
echo '<<<freebsd_geom_mirrors>>>'
gmirror status | grep -v ^Name

# Performancecounter Kernel
echo "<<<kernel>>>"
date +%s
forks=`sysctl -n vm.stats.vm.v_forks`
vforks=`sysctl -n vm.stats.vm.v_vforks`
rforks=`sysctl -n vm.stats.vm.v_rforks`
kthreads=`sysctl -n vm.stats.vm.v_kthreads`
echo "cpu" `sysctl -n kern.cp_time | awk ' { print $1" "$2" "$3" "$5" "$4 } '`
echo "ctxt" `sysctl -n vm.stats.sys.v_swtch`
echo "processes" `expr $forks + $vforks + $rforks + $kthreads`

# Network device statistics (Packets, Collisions, etc)
# only the "Link/Num" interface has all counters.
echo '<<<lnx_if:sep(58)>>>'
date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
else
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
fi

# State of LSI MegaRAID controller via MegaCli.
# To install: pkg install megacli
if which MegaCli >/dev/null ; then
    echo '<<<megaraid_pdisks>>>'
    MegaCli -PDList -aALL -NoLog < /dev/null | egrep 'Enclosure|Raw Size|Slot Number|Device Id|Firmware state|Inquiry'
    echo '<<<megaraid_ldisks>>>'
    MegaCli -LDInfo -Lall -aALL -NoLog < /dev/null | egrep 'Size|State|Number|Adapter|Virtual'
    echo '<<<megaraid_bbu>>>'
    MegaCli -AdpBbuCmd -GetBbuStatus -aALL -NoLog < /dev/null | grep -v Exit
fi

# OpenVPN Clients. 
# Correct log location unknown, sed call might also be broken
if [ -e /var/log/openvpn/openvpn-status.log ] ; then
    echo '<<<openvpn_clients:sep(44)>>>'
    sed -n -e '/CLIENT LIST/,/ROUTING TABLE/p' < /var/log/openvpn/openvpn-status.log  | sed -e 1,3d -e '$d' 
fi

if which ntpq > /dev/null 2>&1 ; then
   echo '<<<ntp>>>'
   # remote heading, make first column space separated
   ntpq -np | sed -e 1,2d -e 's/^\(.\)/\1 /' -e 's/^ /%/'
fi

# Checks for cups monitoring
#if which lpstat > /dev/null 2>&1; then
#  echo '<<<cups_queues>>>'
#  lpstat -p
#  echo '---'
#  for i in $(lpstat -p | grep -E "^(printer|Drucker)" | awk '{print $2}' | grep -v "@"); do
#    lpstat -o "$i"
#  done
#fi

# Heartbeat monitoring
#if which cl_status > /dev/null 2>&1; then
#  # Different handling for heartbeat clusters with and without CRM
#  # for the resource state
#  if [ -S /var/run/heartbeat/crm/cib_ro ]; then
#    echo '<<<heartbeat_crm>>>'
#    crm_mon -1 -r | grep -v ^$ | sed 's/^\s/_/g'
#  else
#    echo '<<<heartbeat_rscstatus>>>'
#    cl_status rscstatus
#  fi
#
#  echo '<<<heartbeat_nodes>>>'
#  for NODE in $(cl_status listnodes); do
#    if [ $NODE != $HOSTNAME ]; then
#      STATUS=$(cl_status nodestatus $NODE)
#      echo -n "$NODE $STATUS"
#      for LINK in $(cl_status listhblinks $NODE 2>/dev/null); do
#        echo -n " $LINK $(cl_status hblinkstatus $NODE $LINK)"
#      done
#      echo
#    fi
#  done
#fi

# Number of TCP connections in the various states
echo '<<<tcp_conn_stats>>>'
netstat -na | awk ' /^tcp/ { c[$6]++; } END { for (x in c) { print x, c[x]; } }'

# Postfix mailqueue monitoring
#
# Only handle mailq when postfix user is present. The mailq command is also
# available when postfix is not installed. But it produces different outputs
# which are not handled by the check at the moment. So try to filter out the
# systems not using postfix by searching for the postfix user.
#
# Cannot take the whole outout. This could produce several MB of agent output
# on blocking queues.
# Only handle the last 6 lines (includes the summary line at the bottom and
# the last message in the queue. The last message is not used at the moment
# but it could be used to get the timestamp of the last message.
#if which mailq >/dev/null 2>&1 && getent passwd postfix >/dev/null 2>&1; then
#  echo '<<<postfix_mailq>>>'
#  mailq | tail -n 6
#fi

#Check status of qmail mailqueue
#if type qmail-qstat >/dev/null
#then
#   echo "<<<qmail_stats>>>"
#   qmail-qstat
#fi

# check zpool status
#if [ -x /sbin/zpool ]; then
#   echo "<<<zpool_status>>>"
#   /sbin/zpool status -x | grep -v "errors: No known data errors"
#fi

# Memory Usage
# currently we'll need sysutils/muse for this.
if [ -x /usr/local/bin/muse ]
then
echo '<<<mem>>>'
# yes, i don't know sed well.
muse -k 2>/dev/null | sed 's/Total/MemTotal/' | sed 's/Free/MemFree/'
swapinfo -k 1K | tail -n 1 | awk '{ print "SwapTotal: "$2" kB\nSwapFree: "$4" kB" }'
fi

# Fileinfo-Check: put patterns for files into /etc/check_mk/fileinfo.cfg
if [ -r "$MK_CONFDIR/fileinfo.cfg" ] ; then
    echo '<<<fileinfo:sep(124)>>>'
    date +%s
    stat -f "%N|%z|%m" $(cat "$MK_CONFDIR/fileinfo.cfg")
fi</fileinfo:sep(124)></mem></zpool_status></qmail_stats></postfix_mailq></tcp_conn_stats></heartbeat_nodes></heartbeat_rscstatus></heartbeat_crm></cups_queues></ntp></openvpn_clients:sep(44)></megaraid_bbu></megaraid_ldisks></megaraid_pdisks></lnx_if:sep(58)></kernel></freebsd_geom_mirrors></freebsd_multipath></lsi></uptime></cpu></ps></mounts></nfsmounts></df></check_mk></local></florian.heigl@gmail.com></lm@mathias-kettner.de>

MathiasM · Jan 30, 2015, 2:15 PM

Here is the solution: (works with pfsende 2.2 and 2.1.5)

A) create inetd (only for pfsense 2.2)

1. create new file /etc/rc.d/inetd
2. insert code from my post
3. run #chmod +x /etc/rc.d/inetd

B) Replace check_mk_agent (older version is working well)

replace code into file /usr/local/bin/check_mk_agent with post from WhiteSaxo

C) Restart daemon

service inetd restart

Roger · Feb 24, 2015, 11:52 AM

Just one little change to the check_mk_client script posted by WhiteSaxo above.

I took the "date %s" out of the lnx_if section as my check_mk server was not picking up the interfaces. I also added lo0 to the exceptions, as it doesn't quite match up on the awk.

So, it now reads

echo '<<<lnx_if:sep(58)>>>'
#date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t
0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
else
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11
" 0 0"}'
fi</lnx_if:sep(58)>

niccarp89 · Feb 24, 2015, 1:13 PM

Hi Roger and all.

I add your modification but still dont get interfaces links, i notice that $osver variable its empty and i dont have defined over the network section. Anyone know if this its ok???

Roger · Feb 24, 2015, 1:45 PM

Hi, I possibly made a mistake with my cut-and-paste.


#date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
else
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
fi

It's a line feed issue from my ssh session. Maybe that's the issue with you not getting interfaces.

If not, what do you get from a "telnet host-ip 6556"?

Roger

niccarp89 · Feb 24, 2015, 1:54 PM

@Roger:

Hi, I possibly made a mistake with my cut-and-paste.
#date +%s
if [ "$(echo $osver | cut -f1 -d\. )" -gt "8" ]; then
    netstat -inb | egrep -v '(^Name|plip|enc|lo0|pfsync|pflog|ovpns)' | grep Link | awk '{print"\t"$1":\t"$8"\t"$5"\t"$6"\t"$7"\t0\t0\t0\t0\t"$11"\t"$9"\t"$10"\t0\t0\t0\t0\t0"}'
else
    # pad output for freebsd 7 and before
    netstat -inb | egrep -v '(^Name|lo|plip)' | grep Link | awk '{print $1" "$7" "$5" "$6" 0 0 0 0 0 "$10" "$8" "$9" 0 0 "$11" 0 0"}'
fi
It's a line feed issue from my ssh session. Maybe that's the issue with you not getting interfaces.

If not, what do you get from a "telnet host-ip 6556"?

Roger

Hi Roger.

http://pastebin.com/U2yq2nxn

Seems that <<<lnx_if:sep(58)>>> section in the telnet its ok, but check_mk dont discover the interfaces.

You have for example re0 re1 or whatever detected as a service in the check_mk interface??</lnx_if:sep(58)>

Roger · Feb 24, 2015, 3:18 PM

Hi

Yes, I get re0 and re1 listed after a 'Full Scan', did you do a 'Full Scan' in check_mk ?

My server version is the latest 1.2.4 version.

Roger

niccarp89 · Feb 24, 2015, 3:22 PM

@Roger:

Hi

Yes, I get re0 and re1 listed after a 'Full Scan', did you do a 'Full Scan' in check_mk ?

My server version is the latest 1.2.4 version.

Roger

Perfect Roger, i only have to do a full scan.

Thanks!