Monitoring Gateway
-
And still nothing in the system log when it happened?
-
@stephenw10 I cleared all logs, unninstalled all packages.
drwxr-xr-x 2 nobody wheel 20 Jan 8 11:35 . drwxr-xr-x 16 root wheel 27 Jan 8 08:57 .. -rw-r--r-- 1 root wheel 147848 Jan 8 11:42 WANGW-quality.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 ipsec-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 ipsec-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 lan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 lan-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 opt1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 opt1-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 ovpns1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 ovpns1-traffic.rrd -rw-r--r-- 1 nobody wheel 49720 Jan 8 11:42 ovpns1-vpnusers.rrd -rw-r--r-- 1 nobody wheel 588592 Jan 8 11:42 system-mbuf.rrd -rw-r--r-- 1 nobody wheel 735320 Jan 8 11:42 system-memory.rrd -rw-r--r-- 1 nobody wheel 245976 Jan 8 11:42 system-processor.rrd -rw-r--r-- 1 nobody wheel 275648 Jan 8 11:42 system-sensors.rrd -rw-r--r-- 1 nobody wheel 245976 Jan 8 11:42 system-states.rrd -rw-r--r-- 1 root wheel 8493 Jan 8 11:35 updaterrd.sh -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 wan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 8 11:42 wan-traffic.rrd
Everytime i run
sh ./updaterrd.sh
A recording appears in the graphics. Then nothing appears. I run it again, it appears again. Then nothing. Can I run that command in crontab to update the graphs every 1 minute?
-
You probably could but you definitely shouldn't need to. That script should be running continually anyway:
[23.01-BETA][root@1100.stevew.lan]/root: ps -auxwwd | grep rrd root 28762 0.0 0.3 13336 2844 u0- IN 13:53 0:00.11 |-- /bin/sh /var/db/rrd/updaterrd.sh
Steve
-
[2.6.0-RELEASE][admin@ccia1.home.arpa]/root: ps -auxwwd | grep rrd root 96131 0.0 0.1 11212 2484 0 S+ 16:08 0:00.00 | `-- grep rrd root 30031 0.0 0.1 11584 2664 v0- IN 11:35 0:00.03 |-- /bin/sh /var/db/rrd/updaterrd.sh root 23555 0.0 0.1 11584 2660 - IN 11:54 0:00.00 | `-- /bin/sh /var/db/rrd/updaterrd.sh
I don't know why it appears twice
-
Mmm, me either. I don't see that anywhere here. Did you already start it manually or add it to some script?
-
@stephenw10 I haven't added it to cron yet and haven't run updaterrd.sh yet. This morning I deleted everything with .rrd from /var/db/rrd. Then reboot. It still doesn't update.
-
Clearly something in your install is starting that script incorrectly or the script is getting hung up on something trying to update it. I would have expected clearing the RRD data to remove whatever that might be though.
I'd probably look at reinstalling at this point. At least that would narrow it to a config problem or something specific to your install.Steve
-
@stephenw10 I concur with the reinstall.. If there was something amiss with it in general you would think the forums would be a fire with people having issues with their monitoring. So it has to be something odd with the install or issue with something being monitored that is unique to the setup that is causing the issue.
-
I will reinstall pfsense at some point. After installation, monitoring worked until a certain point. Unfortunately, I don't have all the logs anymore, but I think it coincided with the moment when I added another network card for a separate network. Thanks
-
I have now installed ntopng. Monitoring has started to work. Let's see for how long.
-
Huh. That's.... interesting
-
He stopped again. I don't know what or how I should search in the logs.
-
It appears the update script is hitting something it's failing to process but it doesn't do it immediately. So what changes when it fails? Is there anything logged at all at that time?
-
[2.6.0-RELEASE][admin@ccia1.home.arpa]/root: ls -lt /var/db/rrd total 402 -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 ovpns1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 ovpns1-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 ipsec-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 ipsec-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 opt1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 opt1-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 lan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 lan-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 wan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 Jan 10 17:05 wan-traffic.rrd -rw-r--r-- 1 root wheel 147848 Jan 10 17:04 WANGW-quality.rrd -rw-r--r-- 1 nobody wheel 882048 Jan 10 17:04 ntpd.rrd -rw-r--r-- 1 nobody wheel 275648 Jan 10 17:04 system-sensors.rrd -rw-r--r-- 1 nobody wheel 588592 Jan 10 17:04 system-mbuf.rrd -rw-r--r-- 1 nobody wheel 735320 Jan 10 17:04 system-memory.rrd -rw-r--r-- 1 nobody wheel 245976 Jan 10 17:04 system-processor.rrd -rw-r--r-- 1 nobody wheel 245976 Jan 10 17:04 system-states.rrd -rw-r--r-- 1 nobody wheel 49720 Jan 10 17:04 ovpns1-vpnusers.rrd -rw-r--r-- 1 root wheel 9205 Jan 10 15:48 updaterrd.sh
[2.6.0-RELEASE][admin@ccia1.home.arpa]/root: grep "rrd" /var/log/system.log Jan 8 11:35:05 ccia1 php[415]: rc.bootup: Creating rrd update script Jan 8 16:46:33 ccia1 php[415]: rc.bootup: Creating rrd update script Jan 8 17:29:28 ccia1 php-fpm[367]: /services_ntpd.php: Creating rrd update script Jan 8 17:36:42 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script Jan 8 17:37:29 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script Jan 8 17:40:40 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script Jan 8 17:42:29 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script Jan 8 17:45:32 ccia1 php-fpm[4876]: /status_monitoring.php: Creating rrd update script Jan 8 17:46:03 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script Jan 8 18:00:17 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script Jan 8 18:39:20 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script Jan 8 19:02:27 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script Jan 8 19:04:55 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
[2.6.0-RELEASE][admin@ccia1.home.arpa]/root: ps -auxwwd | grep rrd root 21285 0.0 0.1 11212 2484 0 S+ 19:57 0:00.00 | `-- grep rrd root 90514 0.0 0.1 11592 2860 - IN 15:48 0:00.62 |-- /bin/sh /var/db/rrd/updaterrd.sh root 56237 0.0 0.1 11592 2856 - IN 17:05 0:00.00 | `-- /bin/sh /var/db/rrd/updaterrd.sh
Is the problem because updaterrd.sh is duplicated? It seems to coincide with the time when the logs stop. But why does it run twice?
-
Is there anything in the logs at 17.04/5 though? Not just rrd entries?
-
Jan 10 03:23:00 ccia1 sshguard[90295]: Exiting on signal. Jan 10 03:23:00 ccia1 sshguard[3397]: Now monitoring attacks. Jan 10 06:49:00 ccia1 sshguard[3397]: Exiting on signal. Jan 10 06:49:00 ccia1 sshguard[13163]: Now monitoring attacks. Jan 10 09:25:00 ccia1 sshguard[13163]: Exiting on signal. Jan 10 09:25:00 ccia1 sshguard[94432]: Now monitoring attacks. Jan 10 11:09:00 ccia1 sshguard[94432]: Exiting on signal. Jan 10 11:09:00 ccia1 sshguard[51714]: Now monitoring attacks. Jan 10 12:00:00 ccia1 sshguard[51714]: Exiting on signal. Jan 10 12:00:00 ccia1 sshguard[78531]: Now monitoring attacks. Jan 10 12:30:00 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Starting up. Jan 10 12:30:00 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Sleeping for 58 seconds. Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating. Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating. Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_PRI1_v4 does not need updating. Jan 10 13:24:00 ccia1 sshguard[78531]: Exiting on signal. Jan 10 13:24:00 ccia1 sshguard[22732]: Now monitoring attacks. Jan 10 15:47:52 ccia1 php-fpm[367]: /status_logs_settings.php: Configuration Change: admin@::1 (Local Database): Changed system logging options. Jan 10 15:47:52 ccia1 check_reload_status[397]: Syncing firewall Jan 10 15:47:52 ccia1 syslogd: exiting on signal 15 Jan 10 15:47:52 ccia1 syslogd: kernel boot file is /boot/kernel/kernel Jan 10 15:47:52 ccia1 nginx: 2023/01/10 15:47:52 [error] 776#100532: send() failed (54: Connection reset by peer) Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Configuration Change: admin@::1 (Local Database): Status: Monitoring: RRD graphing has been enabled. Jan 10 15:48:02 ccia1 check_reload_status[397]: Syncing firewall Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script Jan 10 15:55:35 ccia1 sshguard[22836]: Now monitoring attacks. Jan 10 15:55:44 ccia1 php-fpm[368]: /index.php: Successful login for user 'admin' from: ::1 (Local Database) Jan 10 15:56:16 ccia1 php-fpm[368]: /pkg_mgr_install.php: Configuration Change: admin@::1 (Local Database): Creating restore point before package installation. Jan 10 15:56:16 ccia1 check_reload_status[397]: Syncing firewall Jan 10 15:56:22 ccia1 pkg-static[67905]: libpaper-1.1.28 installed Jan 10 15:56:22 ccia1 pkg-static[67905]: jpeg-turbo-2.1.1_1 installed Jan 10 15:56:22 ccia1 pkg-static[67905]: jbigkit-2.1_1 installed Jan 10 15:56:22 ccia1 pkg-static[67905]: png-1.6.37_1 installed Jan 10 15:56:22 ccia1 pkg-static[67905]: uchardet-0.0.7 installed Jan 10 15:56:22 ccia1 pkg-static[67905]: psutils-1.17_5 installed Jan 10 15:56:23 ccia1 pkg-static[67905]: tiff-4.3.0 installed Jan 10 15:56:24 ccia1 pkg-static[67905]: giflib-5.2.1 installed Jan 10 15:56:24 ccia1 pkg-static[67905]: libfontenc-1.1.4 installed Jan 10 15:56:24 ccia1 pkg-static[67905]: freetype2-2.11.0 installed Jan 10 15:56:24 ccia1 pkg-static[67905]: openpgm-5.2.122_6 installed Jan 10 15:56:24 ccia1 pkg-static[67905]: norm-1.5r6_1 installed Jan 10 15:56:25 ccia1 pkg-static[67905]: groff-1.22.4_4 installed Jan 10 15:56:26 ccia1 pkg-static[67905]: protobuf-3.17.3,1 installed Jan 10 15:56:26 ccia1 pkg-static[67905]: webp-1.2.1 installed Jan 10 15:56:26 ccia1 pkg-static[67905]: mkfontscale-1.2.1 installed Jan 10 15:56:27 ccia1 pkg-static[67905]: fontconfig-2.13.94_1,1 installed Jan 10 15:56:27 ccia1 pkg-static[67905]: libsodium-1.0.18 installed Jan 10 15:56:28 ccia1 pkg-static[67905]: ndpi-4.0.d20210818,1 installed Jan 10 15:56:28 ccia1 pkg-static[67905]: libzmq4-4.3.4 installed Jan 10 15:56:28 ccia1 pkg-static[67905]: libmaxminddb-1.6.0 installed Jan 10 15:56:29 ccia1 pkg-static[67905]: lua53-5.3.6 installed Jan 10 15:56:30 ccia1 pkg-static[67905]: mysql57-client-5.7.36 installed Jan 10 15:56:30 ccia1 pkg-static[67905]: zstd-1.5.0 installed Jan 10 15:56:30 ccia1 pkg-static[67905]: libgd-2.3.1,1 installed Jan 10 15:56:33 ccia1 pkg-static[67905]: webfonts-0.30_14 installed Jan 10 15:56:35 ccia1 pkg-static[67905]: ntopng-5.0.d20210923,1 installed Jan 10 15:56:36 ccia1 pkg-static[67905]: graphviz-2.44.1_17 installed Jan 10 15:56:37 ccia1 pkg-static[67905]: redis-6.2.6 installed Jan 10 15:56:37 ccia1 pkg-static[67905]: gdbm-1.22 installed Jan 10 15:56:37 ccia1 php[68850]: /etc/rc.packages: Beginning package installation for ntopng . Jan 10 15:56:38 ccia1 php[68850]: /etc/rc.packages: Configuration Change: (system): Intermediate config write during package install for ntopng. Jan 10 15:56:38 ccia1 check_reload_status[397]: Syncing firewall Jan 10 15:57:21 ccia1 php[68850]: /etc/rc.packages: Configuration Change: (system): Installed ntopng package. Jan 10 15:57:21 ccia1 check_reload_status[397]: Syncing firewall Jan 10 15:57:21 ccia1 php[68850]: /etc/rc.packages: Successfully installed package: ntopng. Jan 10 15:57:22 ccia1 pkg-static[67905]: pfSense-pkg-ntopng-0.8.13_10 installed Jan 10 15:57:23 ccia1 check_reload_status[397]: Reloading filter Jan 10 15:57:23 ccia1 check_reload_status[397]: Starting packages Jan 10 15:57:24 ccia1 php-fpm[368]: /rc.start_packages: Restarting/Starting all packages. Jan 10 15:57:24 ccia1 php-fpm[368]: /rc.start_packages: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found' Jan 10 15:57:27 ccia1 kernel: re0: promiscuous mode enabled Jan 10 15:57:27 ccia1 kernel: re1: promiscuous mode enabled Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1391] ERROR: [HTTP] set_ports_option: cannot bind to 3000s: Address already in use Jan 10 15:57:45 ccia1 ntopng[62182]: [mongoose.c:4645] ERROR: set_ports_option: cannot bind to 3000s: No error: 0 Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1681] ERROR: Unable to start HTTP server (IPv4) on ports 3000s Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1687] ERROR: Either port in use or another ntopng instance is running (using the same port) Jan 10 16:19:09 ccia1 php-fpm[367]: /pkg_edit.php: Configuration Change: Jan 10 16:19:09 ccia1 check_reload_status[397]: Syncing firewall Jan 10 16:19:12 ccia1 kernel: re0: promiscuous mode disabled Jan 10 16:19:12 ccia1 kernel: re1: promiscuous mode disabled Jan 10 16:19:12 ccia1 php-fpm[367]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found' Jan 10 16:19:13 ccia1 kernel: re0: promiscuous mode enabled Jan 10 16:19:13 ccia1 kernel: re1: promiscuous mode enabled Jan 10 16:25:00 ccia1 sshguard[22836]: Exiting on signal. Jan 10 16:25:00 ccia1 sshguard[38778]: Now monitoring attacks. Jan 10 16:32:00 ccia1 sshguard[38778]: Exiting on signal. Jan 10 16:32:00 ccia1 sshguard[58980]: Now monitoring attacks. Jan 10 17:08:00 ccia1 sshguard[99400]: Now monitoring attacks. Jan 10 17:19:28 ccia1 php-fpm[368]: /pkg_edit.php: Configuration Change: Jan 10 17:19:28 ccia1 check_reload_status[397]: Syncing firewall Jan 10 17:19:32 ccia1 kernel: re0: promiscuous mode disabled Jan 10 17:19:32 ccia1 kernel: re1: promiscuous mode disabled Jan 10 17:19:35 ccia1 php-fpm[368]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found' Jan 10 17:19:35 ccia1 kernel: re0: promiscuous mode enabled Jan 10 17:19:35 ccia1 kernel: re1: promiscuous mode enabled Jan 10 17:19:35 ccia1 kernel: re2: promiscuous mode enabled Jan 10 17:36:46 ccia1 php-fpm[367]: /pkg_edit.php: Configuration Change: Jan 10 17:36:46 ccia1 check_reload_status[397]: Syncing firewall Jan 10 17:36:49 ccia1 kernel: re0: promiscuous mode disabled Jan 10 17:36:49 ccia1 kernel: re1: promiscuous mode disabled Jan 10 17:36:49 ccia1 kernel: re2: promiscuous mode disabled Jan 10 17:36:52 ccia1 php-fpm[367]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found' Jan 10 17:36:53 ccia1 kernel: re0: promiscuous mode enabled Jan 10 17:36:53 ccia1 kernel: re1: promiscuous mode enabled Jan 10 17:36:53 ccia1 kernel: re2: promiscuous mode enabled Jan 10 17:37:06 ccia1 php-fpm[368]: /pkg_edit.php: Configuration Change: Jan 10 17:37:06 ccia1 check_reload_status[397]: Syncing firewall Jan 10 17:37:09 ccia1 kernel: re0: promiscuous mode disabled Jan 10 17:37:09 ccia1 kernel: re1: promiscuous mode disabled Jan 10 17:37:09 ccia1 kernel: re2: promiscuous mode disabled Jan 10 17:37:12 ccia1 php-fpm[368]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found' Jan 10 17:37:13 ccia1 kernel: re0: promiscuous mode enabled Jan 10 17:37:13 ccia1 kernel: re2: promiscuous mode enabled Jan 10 17:43:00 ccia1 sshguard[99400]: Exiting on signal. Jan 10 17:43:00 ccia1 sshguard[88504]: Now monitoring attacks. , client: 10.1.0.2, server: 0.0.0.0:48562 Jan 10 18:26:00 ccia1 sshguard[88504]: Exiting on signal. Jan 10 18:26:00 ccia1 sshguard[14898]: Now monitoring attacks. Jan 10 19:15:00 ccia1 sshguard[14898]: Exiting on signal. Jan 10 19:15:00 ccia1 sshguard[45834]: Now monitoring attacks. Jan 10 20:04:00 ccia1 sshguard[45834]: Exiting on signal. Jan 10 20:04:00 ccia1 sshguard[87400]: Now monitoring attacks. Jan 10 20:04:04 ccia1 nginx: 2023/01/10 20:04:04 [crit] 601#100529: *2472 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 10.1.0.2, server: 0.0.0.0:48562 Jan 10 20:56:00 ccia1 sshguard[87400]: Exiting on signal. Jan 10 20:56:00 ccia1 sshguard[16073]: Now monitoring attacks. Jan 10 21:48:00 ccia1 sshguard[16073]: Exiting on signal. Jan 10 21:48:00 ccia1 sshguard[47268]: Now monitoring attacks. Jan 10 22:43:00 ccia1 sshguard[47268]: Exiting on signal. Jan 10 22:43:00 ccia1 sshguard[82446]: Now monitoring attacks. Jan 10 23:39:00 ccia1 sshguard[82446]: Exiting on signal. Jan 10 23:39:00 ccia1 sshguard[18269]: Now monitoring attacks.
-
Hmm, so not really.
What packages do you have running there?
-
ntopng net 0.8.13_10
openvpn-client-export security 1.6_9
pfBlockerNG-devel net 3.1.0_11 -
Hmm, you might try disabling ntop-ng as a test. It can use resources and cause problems. Though I don't see any direct evidence of that here.
-
@stephenw10 said in Monitoring Gateway:
Hmm, you might try disabling ntop-ng as a test. It can use resources and cause problems. Though I don't see any direct evidence of that here.
I tried this from the beginning but without result.