[SOLVED] How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?

stephenw10 · Nov 30, 2020, 4:38 PM

Both tools look useful. Bashtop is mostly realtime, or close to it, Munin looks to be about logging histirical changes. I can imagine using both.

Sergei_Shablovsky · Nov 30, 2020, 4:38 PM

@stephenw10 said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

Both tools look useful. Bashtop is mostly realtime, or close to it, Munin looks to be about logging histirical changes. I can imagine using both.

As I note before, I just need to extend this cheap menu because it not useful nowadays:

quick initializing action (setup WANs and LANs interfaces quickly) when installing pfSense Bare Metal, or reinstalling after hardware failure;
be able to doing some base network measurement (ping, trace, iperf) just on local console, when the technician are physically near the hardware in DC;
have on local VGA/rs232/usb console REALTIME system health state (because when hardware failure, the System Management Module with watchdogs save all sensors data records and a screenshot into separate memory for future analyses (with other health records) the source of hardware problem;

Only servers has this System Managemet Module (I am sure about IBM, Dell, HP, and even some of Supermicro servers both Enterprise and Entry Level models), ordinary “home PC” - no.

The constantly health monitoring better to doing by separate tools, we prefer Splunk (agent-based system, for overall and detailed) and LibreNMS for detailed net flow (but also able overall, SMNP-based system). Here some examples of LibreNMS

So, the conclusion are: You are right, for serious Monitoring, professional SysAdmin need both local Monitoring tool constant remote metrics collection system.

All this “story with the rewriting the cheap Main Menu” just for ability to make first part - local overall health monitoring tool and several tool for base network testing

Reasonably?

Sergei_Shablovsky · Nov 30, 2020, 4:57 PM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

@Sergei_Shablovsky :

A screen that shows the status of a lot of parameters.
Good.
But what happens when the system goes down ? Where is the screen ? Are you really want to look at the screen 24/24h to detect if something is about to go go bad, or something needs attention ?

Most of servers from reputable brands (I am sure of IBM, Dell, HP) have internal built in Server Health Monitoring PCB/chip, and common function are constantly saving logs and screenshot before hardware crash/unattended restart in separate nonvioation memory, sending alerts by sms/email.

No one sitting behind the server after hi placed in DC or separate server room.

Real admins automatize.

Means years ago I installed this on pfSense : Munin.

When my system goes down, I can actually see what happened after box died - or when the symptoms started. Munin even mails me when something goes off-scale.

Munin give You symptoms, but not picture what happened in real-time, and also have some delay between issue and you receive the alert by sms/email.

The same but **much more detailed and tunable statistic and analyse with filters and ton of plugins” give You the Splunk system. Worth to take a closer look.

And I can see the info no matter where I am ^^

And a whole word with a hackers also. What about security ? No one bank, finance structure, law structure (even a small company) agree with this data open to all worldwide.

Simple perl/bash/sh/php/whatever scripts can be added t add more charts.

Munin is part of the base FreeBSD and present in the pfSense repo - at least, is was so in the past.
It needs to be set up manually, as no GUI part exists.

Btw : other - better ? tools (might) exist. But hey, as you already said : we stick with what we (think !) we know.

This our human habits mostly play a negative role in also fast running and changing IT-world. Especially when we speak about security and a Firewalling.

edit : the main security issue is that the centralized Munin client - the site where I host all the charts of all my devices, connects every 5 minutes to a TCP port 4949 of my pfSense.
A WAN firewall rule accepts incoming TCP 4949 connections from one IP only : the IP from my munin client.
Also : most of the scripts used don't run with admin or root rights, they can only 'read'.

Thank You for a suggestion. ;) May be usable for someone who read this thread and decide to setup Munin.

Sergei_Shablovsky · Dec 3, 2020, 1:15 AM

@Gertjan said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

As for myself : up until know, I didn't even know what bashtop was ...

I told You more: zsh and fish shells able to impress You if You work in command line.

And Bashtop, Pingtop, Hegemon, Gotop, htop, Gping, Vping, also able to give a You a little help if a You have a servers without (or just not need and prefer going to each server by ssh manually for maintenance) agent-server monitoring system like Netdata, LibreNMS, or Splunk.

But another one time:
all this tools is not for ordinary home using, only if You need 24/7 running service and clients and have an obligation.

Sergei_Shablovsky · Feb 11, 2021, 3:40 AM

@kiokoman said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

2.5.0

pkg install bash
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/gsed-4.8.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/coreutils-8.32.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.7.3.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bashtop-0.9.25_1.txz
ln -s /usr/local/bin/python3.7 /bin/python3
/usr/local/bin/bashtop

On a follow system:

2.5.0-RC (amd64) 
built on Wed Feb 10 09:40:34 EST 2021 
FreeBSD 12.2-STABLE

The are next error:

Updating pfSense-core repository catalogue...
pkg: https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.txz: No address record
repository pfSense-core has no meta file, using default settings
pkg: https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz: No address record
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.txz: No address record
repository pfSense has no meta file, using default settings
pkg: https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz: No address record
Unable to update repository pfSense
Error updating repositories!

Soooo... How to install bashop now ? :)

Thank You for Your time and patience!

Gertjan · Feb 11, 2021, 6:00 AM

@sergei_shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

Error updating repositories!

Before calling it a day, yesterday, I saw some posts signalling that the "repository " was out of order.
Probably some product name changes or big updates or ahead.
At least, in the "2.5.0" tree.
Btw : me just thinking here. The 2.4.5-p1 tree seems fine.

kiokoman · Feb 11, 2021, 8:18 AM

@Sergei_Shablovsky
my 2 cent, they are probably moving 2.5.0 out of beta ...

Gertjan · Feb 11, 2021, 8:29 AM

Yeah, I'm also afraid that that's what's coming up.

The 'when is it ready ?' public will get transformed in the 'help a problem' group.
The calm and all goes well period comes to an end.

Sergei_Shablovsky · Feb 11, 2021, 3:13 PM

@gertjan said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

@sergei_shablovsky said in How to installing Bashtop and adding it to Main Menu (menu in Console)?:

Error updating repositories!

Before calling it a day, yesterday, I saw some posts signalling that the "repository " was out of order.
Probably some product name changes or big updates or ahead.
At least, in the "2.5.0" tree.
Btw : me just thinking here. The 2.4.5-p1 tree seems fine.

Is any other way to install bpytop manually ?

Gertjan · Feb 11, 2021, 3:13 PM

@sergei_shablovsky Is that a package in the 'github' pfSense repository ?
Here https://github.com/pfsense/FreeBSD-ports ?

Installing manually packages on a must-always-be-connected-to-the-net-router is ...... awkward.

From higher levels (Netgate), very recently - on this forum - the advise was : don't.

You are using 2.5.0. : isn't it time to go vanilla ? Or prepare yourself for a re install. (not the moment to try try things out .....)

Btw : its doesn't exist for 2.4.5-p1 :

[2.4.5-RELEASE][root@pfsense.core2-wall.localnet]/root: pkg install bpytop
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
pkg: No packages available to install matching 'bpytop' have been found in the repositories

kiokoman · Feb 11, 2021, 3:31 PM

ahh @Sergei_Shablovsky, you need to be more creative !

pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.8.0.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bpytop-1.0.61.txz

open a ticket and ask jimp to add it if you like bpy that much, i don't think he will bite you.. maybe ...

Gertjan · Feb 11, 2021, 3:43 PM

@kiokoman said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

i don't think he will bite you.. maybe ...

You said it yourself : we entered an exceptional moment.

( !!!! Just one left !!!! (February 11, 2021 - 4:38 pm UTC )

edit : oops : this is the son of a b****** bug, with 5 years of history.

stephenw10 · Feb 11, 2021, 3:48 PM

You are set to pull from master still which is now 2.6.
Set your update repo to 2.5.0 RC and you should see something like:

[2.5.0-RC][admin@25dev.stevew.lan]/root: pkg -d update
DBG(1)[11583]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[11583]> PkgRepo: verifying update for pfSense-core
DBG(1)[11583]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[11583]> Request to fetch pkg+https://packages.netgate.com/pfSense_v2_5_0_amd64-core/meta.conf
DBG(1)[11583]> opening libfetch fetcher
DBG(1)[11583]> Fetch > libfetch: connecting
DBG(1)[11583]> Fetch: fetching from: https://files01.netgate.com/pfSense_v2_5_0_amd64-core/meta.conf with opts "i"
DBG(1)[11583]> Fetch: fetcher chosen: https
DBG(1)[11583]> Request to fetch pkg+https://packages.netgate.com/pfSense_v2_5_0_amd64-core/packagesite.txz
DBG(1)[11583]> opening libfetch fetcher
DBG(1)[11583]> Fetch > libfetch: connecting
DBG(1)[11583]> Fetch: fetching from: https://files01.netgate.com/pfSense_v2_5_0_amd64-core/packagesite.txz with opts "i"
DBG(1)[11583]> Fetch: fetcher chosen: https
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[11583]> PkgRepo: verifying update for pfSense
DBG(1)[11583]> Pkgrepo, begin update of '/root/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[11583]> Request to fetch pkg+https://packages.netgate.com/pfSense_v2_5_0_amd64-pfSense_v2_5_0/meta.conf
DBG(1)[11583]> opening libfetch fetcher
DBG(1)[11583]> Fetch > libfetch: connecting
DBG(1)[11583]> Fetch: fetching from: https://files00.netgate.com/pfSense_v2_5_0_amd64-pfSense_v2_5_0/meta.conf with opts "i"
DBG(1)[11583]> Fetch: fetcher chosen: https
DBG(1)[11583]> Request to fetch pkg+https://packages.netgate.com/pfSense_v2_5_0_amd64-pfSense_v2_5_0/packagesite.txz
DBG(1)[11583]> opening libfetch fetcher
DBG(1)[11583]> Fetch > libfetch: connecting
DBG(1)[11583]> Fetch: fetching from: https://files00.netgate.com/pfSense_v2_5_0_amd64-pfSense_v2_5_0/packagesite.txz with opts "i"
DBG(1)[11583]> Fetch: fetcher chosen: https
pfSense repository is up to date.
All repositories are up to date.

Steve

Sergei_Shablovsky · Feb 11, 2021, 7:11 PM

@kiokoman said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

ahh @Sergei_Shablovsky, you need to be more creative !
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py37-psutil-5.8.0.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bpytop-1.0.61.txz
open a ticket and ask jimp to add it if you like bpy that much, i don't think he will bite you.. maybe ...

I still thinking that bpytop are so informative and flexible comparing to native pfSense top/pftop, so may be jimp would be agree to include it as standard tool for on-screen Monitoring....

P.S. The same for Gping/vping, but on 3 month ago they are not available for FreeBSD. Sad but true...

stephenw10 · Feb 11, 2021, 8:56 PM

Did you open a feature request yet? https://redmine.pfsense.org/

Sergei_Shablovsky · Feb 12, 2021, 11:49 AM

@stephenw10 said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

Did you open a feature request yet? https://redmine.pfsense.org/

Right now no. But I would be happy when You doing this (for example)... :)

Due You have a much more valuable reputation in community :)

Sergei_Shablovsky · Feb 12, 2021, 9:54 PM

@stephenw10 said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

Did you open a feature request yet? https://redmine.pfsense.org/

Already done

Please upvote / assist to make this happened.

Sergei_Shablovsky · Jul 26, 2021, 12:04 AM

UPDATE

on
Version 2.6.0-DEVELOPMENT (amd64)
built on Sun Jul 25 01:08:17 ED

pkg install bash

pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/gsed-4.8.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/coreutils-8.32.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/py38-psutil-5.8.0.txz
pkg add https://pkg.FreeBSD.org/FreeBSD:12:amd64/latest/All/bpytop-1.0.67.txz

ln -s /usr/local/bin/python3.8 /bin/python3
/usr/local/bin/bpytop

Sergei_Shablovsky · Nov 4, 2021, 10:15 PM

@gertjan said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

You could 'diff' between the original files, and your files, and have this diff merged automatically during startup or even update (I use the 'Patches' package from pfSense).

Could You be so please explain on an example (for /etc/rc.initial) how to make this according this topic.

To save my time a little bit. ;) Anyway, a little bit later I try to study how to using the Patches. I found that official pfSense doc from NetGate have no such info and examples...

I would be thankful for Yoyr time and efforts about helping people’s here on forum! Thank You again one time!

But what if Netgate decided to fundamentally change the /etc/rc.initial file in the next version ? The merge won't work out. Your back to manual again.

As I able to see, nothing much changed (just nothing) in last 3 year.

Anyway, I make all system updates in manual mode, to look precisely on each changes.

Sergei_Shablovsky · Nov 4, 2021, 10:23 PM

@gertjan said in How to installing bpytop (former Bashtop) and adding it to Main Menu (menu in Console)?:

edit : the main security issue is that the centralized Munin client - the site where I host all the charts of all my devices, connects every 5 minutes to a TCP port 4949 of my pfSense.
A WAN firewall rule accepts incoming TCP 4949 connections from one IP only : the IP from my munin client.

Let’s to note, better not to exposing ports to outside, and SSH keys are so easy to brute force nowadays on powerful i7/i9 or NVidia cards.
As minimal, using OpenVPN / WireGuard + port knock + limit to x connection in y time.