[SOLVED] How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?

Gertjan · Nov 29, 2020, 8:39 PM

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

But how to properly installing Bashtop in pfSense to avoid any issue?

That's a huge question.

How does a surgeon implements a pacemaker in your body ?

I really hope you already know the answer .... (spoiler : took him some time : he learned !!)

This forum is about using pfSense. It happens that a script gets modified some where to implement or circumvent an issue.
You are free to do what you want where and how you want to do it.
Nothing will break, as you're a couple of clicks away from restoring - or just pressing Ctrl-Z and place the modified file again.

Bashtop : did not find that word on this forum ... (using Google - you might try other).
So : there a two possibilities :
No one asks that question because everybody knows the answer already.
No one ever did - or even no one is planning to do so.
As for myself : up until know, I didn't even know what bashtop was ...

Your lucky : bastop exists in https://github.com/pfsense/FreeBSD-ports so it is part of the packages that exist in the pfSense / Freebsd repository.
You could install it manually.
After that, your good for some old school PHP editing and you'll be fine.

Note : real, public code talk for pfSense doesn't exist, as it is still a product made by a company, Netgate. The guys who write the line receive not only our thanks, but are also paid to do so (I hope). This means that code talk is done somewhere in Texas, Austin. Not on a public forum.

Btw : I'm just somewhat exhaustive because this question comes up often :
Can pfSense make coffee ?
Play chess ?
Lauching nukes ?

I know, I'm some what borderline here, but my goal is just to give useful info.

Sergei_Shablovsky · Nov 24, 2020, 1:06 PM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

Your found /etc/rc.initial.

Can't tell if it's the best place. What stops you from trying ?

Just nothing.
But I prefer to ask someone who known the subject better, if it possible, rater spending a hours with unknown results :)

I just re-grouping menu on my taste:

separation done by functions and by frequency of using;
most frequently using menus place close to command line prompt (to not seeking&jumping by eyes up&down);
previous numbering saved to avoid mistypes for a lot of other who work with pfsense a years...;
size of menu fit the standard hardware rack mount Terminal Console (most of them have 15 or 17 inch size);

# Check to see if SSH is running.
if pgrep -q -a -F /var/run/sshd.pid sshd >/dev/null 2>&1; then
	sshd_option="14 Disable Secure Shell (sshd)";
else
	sshd_option="14 Enable Secure Shell (sshd)";
fi

# display a cheap menu
echo ""
echo " ### RESTARTS, REBOOT, SHUTDOWN #########################################"
echo " 11 Restart webConfigurator         5 Reboot system"
echo " 16 Restart PHP-FPM                 6 Halt system"
echo " ### SHELLS, TOOLS ######################################################"
echo " ${sshd_option}      0 Logout (SSH only)"
echo " 12 PHP shell + ${product} tools"
echo "  8 Shell"
echo " ### MONITORING, LOGS ###################################################"
echo " 18 Bashtop                         7 Ping host"
echo " 17 top                            10 Filter Logs"
echo "  9 pfTop"
echo " ### SETUP, RESTORE #####################################################"
echo "  1 Assign Interfaces               3 Reset webConfigurator password"
echo "  2 Set interface(s) IP address     4 Reset to factory defaults"
echo " 13 Update from console"
echo " 15 Restore recent configuration"

echo
read -p "Enter an option: " opmode
echo

and

17)
	top
	;;
18)
	bashtop
	;;

Sergei_Shablovsky · Nov 24, 2020, 12:47 PM

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

Your found /etc/rc.initial.

Can't tell if it's the best place. What stops you from trying ?

I just re-grouping menu on my taste

How to keep the changes intact during periodically updates by System / Update menu item ?

Gertjan · Nov 24, 2020, 2:43 PM

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

rater spending a hours with unknown results

IMHO : this won't take you hours ;)
The code you shows looks just fine.

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

changes intact during periodically updates

There is no 100 % guaranted solution for this.
As long as you change just a couple of lines, I advise you to install the smallest pfSense package : Notes.

Use this to document your settings, with copies of fragments of changed files.
Put in also the URLS to these posts on the forum here.
The name of the Freebsd (pfsense ported) package used.
etc.

You could 'diff' between the original files, and your files, and have this diff merged automatically during startup or even update (I use the 'Patches' package from pfSense).
But what if Netgate decided to fundamentally change the /etc/rc.initial file in the next version ? The merge won't work out. Your back to manual again.

So, my advice : take notes. Do not try to 'fix' in some way ever moving code.
If an upgrade comes in, it will take you 5 minutes to re patch.
Because you know now how to do it ;)

kiokoman · Nov 24, 2020, 3:51 PM

it would be better to ask it on redmine for a new feature and see if they can add it to pfsense, it need some dependencies not available on pfsense repo, idk if they would have any side effect.
I don't suggest doing this on a production device

it's not good as the one on linux,
imho better use htop that is already available

you do it on your own responsibility

pkg install py37-psutil
pkg install bash
pkg add https://pkg.FreeBSD.org/FreeBSD:11:amd64/release_3/All/gsed-4.7.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:11:amd64/release_3/All/coreutils-8.31.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:11:amd64/latest/All/bashtop-0.9.25_1.txz
ln -s /usr/local/bin/python3.7 /bin/python3
/usr/local/bin/bash 
/usr/local/bin/bashtop

Sergei_Shablovsky · Nov 24, 2020, 8:08 PM

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

I just re-grouping menu on my taste:

So, I make a little bit upgrade for previous menu:

each group have each numbering (easy to remember that 1x - setup/restore, 2x - monitoring/logs, but 4x - restart/reboot/shutdown, steps from easy operation to radically changing device behaviors and state)
text color help to easy navigate and preventing mistypes;
visually divided on groups for easy recognition (because all sysadmins are TOO busy and sometimes mistakes come in);
the menu size fit into common rack hardware Terminal Console screen size 15-17inch/landscape orientation, even have extra free space to adding Your custom menu items;

P.S.
I really know what is a HEADPAIN when we (as busy ingeneers who have family, friends, cats :) and a lot of other works) need spend additional time to re-learning something that we know very well and work with during a years....
But sometime very old solution need to be changed because time go fast forward. May be this is one of this cases...

By the way Bashtop are just blasting a little time ago, even we know bash colors and formatting for YEARS!
Is it useful? OF COURSE! Like iPod Classic after Walkman or iRiver :)

Here the /etc/rc.initial file (If You find any mistake (sorry for that), please write corrections here)

#!/bin/sh
#
# rc.initial
#
# part of pfSense (https://www.pfsense.org)
# Copyright (c) 2004-2013 BSD Perimeter
# Copyright (c) 2013-2016 Electric Sheep Fencing
# Copyright (c) 2014-2020 Rubicon Communications, LLC (Netgate)
# All rights reserved.
#
# originally based on m0n0wall (http://neon1.net/m0n0wall)
# Copyright (c) 2003-2004 Manuel Kasper <mk@neon1.net>.
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# make sure the user can't kill us by pressing Ctrl-C,
# ctrl-z, etc.
trap : 2
trap : 3
trap : 4

unset do_sleep
if [ -f /etc/rc.local ]; then
	if ! ps auxwww | egrep -q '[r]c.local$'; then
		echo ">>> Launching rc.local in background..."
		sh /etc/rc.local &
		do_sleep=1
	fi
	if [ -f /etc/rc.local.running ] && \
	    ! ps auxwww | egrep -q '[r]c.local.running$'; then
		[ -n "${do_sleep}" ] && sleep 1
		echo ">>> Launching rc.local.running in background..."
		sh /etc/rc.local.running &
	fi
fi

# Parse command line parameters
while [ $# -gt 0 ]; do
	case $1 in
		-c )	shift
			/bin/sh -c "$@"
			exit
			;;
		* )
	esac
	shift
done

CONFIG="/cf/conf/config.xml"
WORD="https"

# endless loop
while : ; do

if [ -f /tmp/ttybug ]; then
	rm /tmp/ttybug
	exit && exit && logout
fi

/etc/rc.banner

# Read product_name from $g, defaults to pfSense
product=$(/usr/local/sbin/read_global_var product_name pfSense)

# Check to see if SSH is running.
if pgrep -q -a -F /var/run/sshd.pid sshd >/dev/null 2>&1; then
	sshd_option="\e[1m31 \e[31;93mDisable Secure Shell \e[0m(sshd)\e[0m";
else
	sshd_option="\e[1m31 \e[31;92mEnable Secure Shell \e[0m(sshd)\e[0m";
fi

# display a cheap menu
echo ""
echo -e " \e[100m 4x \e[1mRESTARTS, REBOOT, SHUTDOWN                                          \e[0m"
echo -e "  \e[1m41 \e[21;93mRestart webConfigurator\e[0m            \e[1m43 \e[21;31mReboot system\e[0m"
echo -e "  \e[1m42 \e[21;93mRestart PHP-FPM\e[0m                    \e[1m44 \e[21;41mShutdown system\e[0m"
echo -e " \e[100m 3x \e[1mSHELLS, TOOLS                                                       \e[0m"
echo -e "  ${sshd_option}         \e[1m34 \e[21;93mLogout \e[0m(SSH only)\e[0m"
echo -e "  \e[1m32 \e[21;92mPHP shell + ${product} tools\e[0m"
echo -e "  \e[1m33 \e[0mShell\e[0m"
echo -e " \e[100m 2x \e[1mMONITORING, LOGS                                                    \e[0m"
echo -e "  \e[1m21 \e[21;92mBashtop\e[0m                            \e[1m24 \e[0mFilter Logs\e[0m"
echo -e "  \e[1m22 \e[21;92mPing host\e[0m                          \e[1m25 \e[0mtop\e[0m"
echo -e "  \e[1m23 \e[21;92mpfTop\e[0m"
echo -e " \e[100m 1x \e[1mSETUP, RESTORE                                                      \e[0m"
echo -e "  \e[1m11 \e[21;92mAssign Interfaces\e[0m                  \e[1m15 \e[21;31mReset webConfigurator password\e[0m"
echo -e "  \e[1m12 \e[21;92mSet interface(s) IP address\e[0m"
echo -e "  \e[1m13 \e[0mUpdate from console\e[0m"
echo -e "  \e[1m14 \e[21;92mRestore recent configuration\e[0m       \e[1m16 \e[21;41mReset to factory defaults\e[0m"

echo
read -p "Enter an option: " opmode
echo

# see what the user has chosen
case ${opmode} in
34)
	exit && exit && logout
	;;
11)
	/etc/rc.initial.setports
	;;
12)
	/etc/rc.initial.setlanip
	;;
15)
	/etc/rc.initial.password
	;;
16)
	/etc/rc.initial.defaults
	;;
43)
	/etc/rc.initial.reboot
	;;
44)
	/etc/rc.initial.halt
	;;
22)
	/etc/rc.initial.ping
	;;
33)
	/bin/tcsh
	;;
23)
	/usr/local/sbin/pftop
	;;
24)
	/usr/bin/tail -F /var/log/filter.log
	;;
41 | 111)
	/etc/rc.restart_webgui
	;;
32)
	/usr/local/sbin/pfSsh.php
	;;
13)
	/bin/sh /usr/local/sbin/${product}-upgrade
	;;
31)
	php -f /etc/rc.initial.toggle_sshd
	;;
14)
	/etc/rc.restore_config_backup
	;;
42)
	/etc/rc.php-fpm_restart
	;;
25)
	top
	;;
21)
	bashtop
	;;
100)
	if grep "$WORD" "$CONFIG"; then
		links "https://localhost"
	else
		links "http://localhost"
	fi
	;;
"")
	ps -o command='' -c -p $PPID | grep -E '^sshd$' > /dev/null
	if [ $? -eq 0 ]; then
		exit
	else
		kill $PPID ; exit
	fi
	;;
esac

done

Thank You for attention and time!

Sergei_Shablovsky · Nov 24, 2020, 8:33 PM

@kiokoman said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

it would be better to ask it on redmine for a new feature and see if they can add it to pfsense, it need some dependencies not available on pfsense repo, idk if they would have any side effect.
I don't suggest doing this on a production device

it's not good as the one on linux,
imho better use htop that is already available

you do it on your own responsibility

This words must be on a wall opposite to each sysadmin workplace! :)

Thank You for reasonably reply!

Just look on the pictures in top of topic: more accurate UI, more information and very easy for taking a quick decision about device state. Even several clicks - and You are to kill processes, see detailed resources, etc, etc...
Is this the same Bashtop from pfSense or some tuning/skin/configuration ? Looks VERY different...

Sergei_Shablovsky · Nov 24, 2020, 8:46 PM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

rater spending a hours with unknown results

IMHO : this won't take you hours ;)
The code you shows looks just fine.

Thank You! I spend a little bit more time to remind my childhood with bash, and make a little bit improved (color coding, grouping by frequency of using and grouping by functions) version, please see below.

@Sergei_Shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in in Console)?:

changes intact during periodically updates

There is no 100 % guaranted solution for this.

I know very well this phrase and hate it! :) Of course, You are absolutely right.

As long as you change just a couple of lines, I advise you to install the smallest pfSense package : Notes.

Use this to document your settings, with copies of fragments of changed files.
Put in also the URLS to these posts on the forum here.
The name of the Freebsd (pfsense ported) package used.
etc.

Thank You for suggestion! I doing this already before reading :) Anyway, THANK YOU!
Why this suggestion is not in official docs?????

You could 'diff' between the original files, and your files, and have this diff merged automatically during startup or even update (I use the 'Patches' package from pfSense).

Thank You for suggestion! I using the Kaleidoscope app on macOS. VERY PRO and USEFUL app. I try on a real coding work most of apps, but Kaleidoscope in conjunction with Tower - my choice, best of the best!

But what if Netgate decided to fundamentally change the /etc/rc.initial file in the next version ? The merge won't work out. Your back to manual again.

This is a life. Life is live. :) May be some stuff from Netgate look and just implement some of this ideas. Whit no charge from our side :)

So, my advice : take notes. Do not try to 'fix' in some way ever moving code.
If an upgrade comes in, it will take you 5 minutes to re patch.
Because you know now how to do it ;)

Thank You another one time. I'l doing this way.

Sergei_Shablovsky · Nov 24, 2020, 11:02 PM

This post is deleted!

kiokoman · Nov 25, 2020, 1:47 AM

the instructions was for 2.4.5-p1 i will check later for 2.5.0

kiokoman · Nov 25, 2020, 8:32 AM

2.5.0

pkg install bash
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/gsed-4.8.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/coreutils-8.32.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.7.3.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bashtop-0.9.25_1.txz
ln -s /usr/local/bin/python3.7 /bin/python3
/usr/local/bin/bashtop

Sergei_Shablovsky · Nov 26, 2020, 5:56 PM

@kiokoman said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

2.5.0

pkg install bash
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/gsed-4.8.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/coreutils-8.32.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.7.3.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bashtop-0.9.25_1.txz
ln -s /usr/local/bin/python3.7 /bin/python3
/usr/local/bin/bashtop

Thank You for help!

I was on road, and not able to answering at time, so we both doing the same simultaneously;)

All 4 packages downloaded and installed correctly, ln command - success, but the problem is after Bashtop started, the initial screen halt on last command:

Creating psutil coprocess.....  ok
Checking available tools......ok
Checking cpu......ok
Calculating sizes....ok
Running cpu collection init......


39f
39f
39f
39f
39f
39f

Please, do You have any idea how to know what happened with Bashtop process at all?

P.S. Carl+z and “exit” command after return me to initial screen with “cheap menu”.

kiokoman · Nov 26, 2020, 6:12 PM

did you try to launch bash before bashtop ?

/usr/local/bin/bash
/usr/local/bin/bashtop

it works and looks good on my 2.5.0 without bash

[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/gsed-4.8.txz
Fetching gsed-4.8.txz: 100%  315 KiB 322.5kB/s    00:01
Installing gsed-4.8...
Extracting gsed-4.8: 100%
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/coreutils-8.32.txz
Fetching coreutils-8.32.txz: 100%    3 MiB   2.7MB/s    00:01
Installing coreutils-8.32...
Extracting coreutils-8.32: 100%
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.7.3.txz
Fetching py37-psutil-5.7.3.txz: 100%  337 KiB 345.1kB/s    00:01
Installing py37-psutil-5.7.3...
Extracting py37-psutil-5.7.3: 100%
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bashtop-0.9.25_1.txz
Fetching bashtop-0.9.25_1.txz: 100%   48 KiB  49.6kB/s    00:01
Installing bashtop-0.9.25_1...
Extracting bashtop-0.9.25_1: 100%
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: ln -s /usr/local/bin/python3.7 /bin/python3
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: /usr/local/bin/bashtop

you can try to launch bashtop with truss to see if you can understand why it show that "39f"

truss /usr/local/bin/bashtop

Sergei_Shablovsky · Nov 28, 2020, 7:18 PM

@kiokoman said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

did you try to launch bash before bashtop ?

/usr/local/bin/bash
/usr/local/bin/bashtop

it works and looks good on my 2.5.0 without bash

Thank You for suggestions!

I try

by choosing from menu (this is actually I want);
by starting Shell (menuitem “33” from menu above, pls seek in code), then trying /usr/local/bin/bashtop w/ and w/o /usr/local/bin/bash;

The result are the same,- stuck on the same “Running cpu collection init.....”

P.S. I notice that in pfSense Dashboard “CPU Usage” stuck on “Retrieving CPU data *”. Is this linked to something like “CPU Legacy” in UEFI ?

Sergei_Shablovsky · Nov 29, 2020, 8:39 PM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

Bashtop : did not find that word on this forum ... (using Google - you might try other).
So : there a two possibilities :
No one asks that question because everybody knows the answer already.
No one ever did - or even no one is planning to do so.
As for myself : up until know, I didn't even know what bashtop was ...

I have another one option an addition to that two You mention:

ordinary SysAdmins are too lazy to improving their skills and step out the boundaries. Because of this they never improving the good tool that working good already, and very rarely reading more outside the original software documentation.
Just absence of motivation in nowadays.

stephenw10 · Nov 29, 2020, 9:31 PM

Just to be clear nobody should ever do this on a production firewall without fully understanding the consequences of doing so.

This is a fun experiment and I'm all about that but should only be done on a test/development box.

If it's something that proved useful then a package could potentially be created that would be properly vetted and built against our repo.

Steve

Gertjan · Nov 30, 2020, 6:28 AM

@Sergei_Shablovsky :

A screen that shows the status of a lot of parameters.
Good.
But what happens when the system goes down ? Where is the screen ? Are you really want to look at the screen 24/24h to detect if something is about to go go bad, or something needs attention ?
Real admins automatize.

Means years ago I installed this on pfSense : Munin.

When my system goes down, I can actually see what happened after box died - or when the symptoms started. Munin even mails me when something goes off-scale.
And I can see the info no matter where I am ^^

Simple perl/bash/sh/php/whatever scripts can be added t add more charts.

Munin is part of the base FreeBSD and present in the pfSense repo - at least, is was so in the past.
It needs to be set up manually, as no GUI part exists.

Btw : other - better ? tools (might) exist. But hey, as you already said : we stick with what we (think !) we know.

edit : the main security issue is that the centralized Munin client - the site where I host all the charts of all my devices, connects every 5 minutes to a TCP port 4949 of my pfSense.
A WAN firewall rule accepts incoming TCP 4949 connections from one IP only : the IP from my munin client.
Also : most of the scripts used don't run with admin or root rights, they can only 'read'.

stephenw10 · Nov 30, 2020, 4:38 PM

Both tools look useful. Bashtop is mostly realtime, or close to it, Munin looks to be about logging histirical changes. I can imagine using both.

Sergei_Shablovsky · Nov 30, 2020, 4:38 PM

@stephenw10 said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

Both tools look useful. Bashtop is mostly realtime, or close to it, Munin looks to be about logging histirical changes. I can imagine using both.

As I note before, I just need to extend this cheap menu because it not useful nowadays:

quick initializing action (setup WANs and LANs interfaces quickly) when installing pfSense Bare Metal, or reinstalling after hardware failure;
be able to doing some base network measurement (ping, trace, iperf) just on local console, when the technician are physically near the hardware in DC;
have on local VGA/rs232/usb console REALTIME system health state (because when hardware failure, the System Management Module with watchdogs save all sensors data records and a screenshot into separate memory for future analyses (with other health records) the source of hardware problem;

Only servers has this System Managemet Module (I am sure about IBM, Dell, HP, and even some of Supermicro servers both Enterprise and Entry Level models), ordinary “home PC” - no.

The constantly health monitoring better to doing by separate tools, we prefer Splunk (agent-based system, for overall and detailed) and LibreNMS for detailed net flow (but also able overall, SMNP-based system). Here some examples of LibreNMS

So, the conclusion are: You are right, for serious Monitoring, professional SysAdmin need both local Monitoring tool constant remote metrics collection system.

All this “story with the rewriting the cheap Main Menu” just for ability to make first part - local overall health monitoring tool and several tool for base network testing

Reasonably?

Sergei_Shablovsky · Nov 30, 2020, 4:57 PM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

@Sergei_Shablovsky :

A screen that shows the status of a lot of parameters.
Good.
But what happens when the system goes down ? Where is the screen ? Are you really want to look at the screen 24/24h to detect if something is about to go go bad, or something needs attention ?

Most of servers from reputable brands (I am sure of IBM, Dell, HP) have internal built in Server Health Monitoring PCB/chip, and common function are constantly saving logs and screenshot before hardware crash/unattended restart in separate nonvioation memory, sending alerts by sms/email.

No one sitting behind the server after hi placed in DC or separate server room.

Real admins automatize.

Means years ago I installed this on pfSense : Munin.

When my system goes down, I can actually see what happened after box died - or when the symptoms started. Munin even mails me when something goes off-scale.

Munin give You symptoms, but not picture what happened in real-time, and also have some delay between issue and you receive the alert by sms/email.

The same but **much more detailed and tunable statistic and analyse with filters and ton of plugins” give You the Splunk system. Worth to take a closer look.

And I can see the info no matter where I am ^^

And a whole word with a hackers also. What about security ? No one bank, finance structure, law structure (even a small company) agree with this data open to all worldwide.

Simple perl/bash/sh/php/whatever scripts can be added t add more charts.

Munin is part of the base FreeBSD and present in the pfSense repo - at least, is was so in the past.
It needs to be set up manually, as no GUI part exists.

Btw : other - better ? tools (might) exist. But hey, as you already said : we stick with what we (think !) we know.

This our human habits mostly play a negative role in also fast running and changing IT-world. Especially when we speak about security and a Firewalling.

edit : the main security issue is that the centralized Munin client - the site where I host all the charts of all my devices, connects every 5 minutes to a TCP port 4949 of my pfSense.
A WAN firewall rule accepts incoming TCP 4949 connections from one IP only : the IP from my munin client.
Also : most of the scripts used don't run with admin or root rights, they can only 'read'.

Thank You for a suggestion. ;) May be usable for someone who read this thread and decide to setup Munin.