Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Monitoring Gateway

    Scheduled Pinned Locked Moved
    General pfSense Questions
    5
    55
    6.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • johnpozJ
      johnpoz LAYER 8 Global Moderator @stephenw10
      last edited by

      @stephenw10 I concur with the reinstall.. If there was something amiss with it in general you would think the forums would be a fire with people having issues with their monitoring. So it has to be something odd with the install or issue with something being monitored that is unique to the setup that is causing the issue.

      An intelligent man is sometimes forced to be drunk to spend time with his fools
      If you get confused: Listen to the Music Play
      Please don't Chat/PM me for help, unless mod related
      SG-4860 24.03 | Lab VMs 2.7.2, 24.03

      1 Reply Last reply Reply Quote 1
      • hendiH
        hendi
        last edited by

        I will reinstall pfsense at some point. After installation, monitoring worked until a certain point. Unfortunately, I don't have all the logs anymore, but I think it coincided with the moment when I added another network card for a separate network. Thanks

        1 Reply Last reply Reply Quote 0
        • hendiH
          hendi
          last edited by

          I have now installed ntopng. Monitoring has started to work.😍 Let's see for how long.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Huh. That's.... interesting 🤔

            1 Reply Last reply Reply Quote 0
            • hendiH
              hendi
              last edited by

              He stopped again. I don't know what or how I should search in the logs.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                It appears the update script is hitting something it's failing to process but it doesn't do it immediately. So what changes when it fails? Is there anything logged at all at that time?

                1 Reply Last reply Reply Quote 0
                • hendiH
                  hendi
                  last edited by 

                  [2.6.0-RELEASE][admin@ccia1.home.arpa]/root: ls -lt /var/db/rrd
total 402
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 ovpns1-packets.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 ovpns1-traffic.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 ipsec-packets.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 ipsec-traffic.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 opt1-packets.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 opt1-traffic.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 lan-packets.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 lan-traffic.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 wan-packets.rrd
-rw-r--r--  1 nobody  wheel  393168 Jan 10 17:05 wan-traffic.rrd
-rw-r--r--  1 root    wheel  147848 Jan 10 17:04 WANGW-quality.rrd
-rw-r--r--  1 nobody  wheel  882048 Jan 10 17:04 ntpd.rrd
-rw-r--r--  1 nobody  wheel  275648 Jan 10 17:04 system-sensors.rrd
-rw-r--r--  1 nobody  wheel  588592 Jan 10 17:04 system-mbuf.rrd
-rw-r--r--  1 nobody  wheel  735320 Jan 10 17:04 system-memory.rrd
-rw-r--r--  1 nobody  wheel  245976 Jan 10 17:04 system-processor.rrd
-rw-r--r--  1 nobody  wheel  245976 Jan 10 17:04 system-states.rrd
-rw-r--r--  1 nobody  wheel   49720 Jan 10 17:04 ovpns1-vpnusers.rrd
-rw-r--r--  1 root    wheel    9205 Jan 10 15:48 updaterrd.sh

                  [2.6.0-RELEASE][admin@ccia1.home.arpa]/root: grep "rrd" /var/log/system.log
Jan  8 11:35:05 ccia1 php[415]: rc.bootup: Creating rrd update script
Jan  8 16:46:33 ccia1 php[415]: rc.bootup: Creating rrd update script
Jan  8 17:29:28 ccia1 php-fpm[367]: /services_ntpd.php: Creating rrd update script
Jan  8 17:36:42 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
Jan  8 17:37:29 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script
Jan  8 17:40:40 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script
Jan  8 17:42:29 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script
Jan  8 17:45:32 ccia1 php-fpm[4876]: /status_monitoring.php: Creating rrd update script
Jan  8 17:46:03 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
Jan  8 18:00:17 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
Jan  8 18:39:20 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script
Jan  8 19:02:27 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
Jan  8 19:04:55 ccia1 php-fpm[367]: /status_monitoring.php: Creating rrd update script
Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script

                  [2.6.0-RELEASE][admin@ccia1.home.arpa]/root: ps -auxwwd | grep rrd
root    21285   0.0  0.1  11212   2484  0  S+   19:57      0:00.00 |       `-- grep rrd
root    90514   0.0  0.1  11592   2860  -  IN   15:48      0:00.62 |-- /bin/sh /var/db/rrd/updaterrd.sh
root    56237   0.0  0.1  11592   2856  -  IN   17:05      0:00.00 | `-- /bin/sh /var/db/rrd/updaterrd.sh

                  Is the problem because updaterrd.sh is duplicated? It seems to coincide with the time when the logs stop. But why does it run twice?

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Is there anything in the logs at 17.04/5 though? Not just rrd entries?

                    hendiH 1 Reply Last reply Reply Quote 0
                    • hendiH
                      hendi @stephenw10
                      last edited by stephenw10

                      @stephenw10

                      Jan 10 03:23:00 ccia1 sshguard[90295]: Exiting on signal.
Jan 10 03:23:00 ccia1 sshguard[3397]: Now monitoring attacks.
Jan 10 06:49:00 ccia1 sshguard[3397]: Exiting on signal.
Jan 10 06:49:00 ccia1 sshguard[13163]: Now monitoring attacks.
Jan 10 09:25:00 ccia1 sshguard[13163]: Exiting on signal.
Jan 10 09:25:00 ccia1 sshguard[94432]: Now monitoring attacks.
Jan 10 11:09:00 ccia1 sshguard[94432]: Exiting on signal.
Jan 10 11:09:00 ccia1 sshguard[51714]: Now monitoring attacks.
Jan 10 12:00:00 ccia1 sshguard[51714]: Exiting on signal.
Jan 10 12:00:00 ccia1 sshguard[78531]: Now monitoring attacks.
Jan 10 12:30:00 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Starting up.
Jan 10 12:30:00 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Sleeping for 58 seconds.
Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: Starting URL table alias updates
Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v4 does not need updating.
Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_Top_v6 does not need updating.
Jan 10 12:30:58 ccia1 php[94000]: rc.update_urltables: /etc/rc.update_urltables: pfB_PRI1_v4 does not need updating.
Jan 10 13:24:00 ccia1 sshguard[78531]: Exiting on signal.
Jan 10 13:24:00 ccia1 sshguard[22732]: Now monitoring attacks.
Jan 10 15:47:52 ccia1 php-fpm[367]: /status_logs_settings.php: Configuration Change: admin@::1 (Local Database): Changed system logging options.
Jan 10 15:47:52 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 15:47:52 ccia1 syslogd: exiting on signal 15
Jan 10 15:47:52 ccia1 syslogd: kernel boot file is /boot/kernel/kernel
Jan 10 15:47:52 ccia1 nginx: 2023/01/10 15:47:52 [error] 776#100532: send() failed (54: Connection reset by peer)
Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Configuration Change: admin@::1 (Local Database): Status: Monitoring: RRD graphing has been enabled.
Jan 10 15:48:02 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 15:48:02 ccia1 php-fpm[368]: /status_monitoring.php: Creating rrd update script
Jan 10 15:55:35 ccia1 sshguard[22836]: Now monitoring attacks.
Jan 10 15:55:44 ccia1 php-fpm[368]: /index.php: Successful login for user 'admin' from: ::1 (Local Database)
Jan 10 15:56:16 ccia1 php-fpm[368]: /pkg_mgr_install.php: Configuration Change: admin@::1 (Local Database): Creating restore point before package installation.
Jan 10 15:56:16 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 15:56:22 ccia1 pkg-static[67905]: libpaper-1.1.28 installed
Jan 10 15:56:22 ccia1 pkg-static[67905]: jpeg-turbo-2.1.1_1 installed
Jan 10 15:56:22 ccia1 pkg-static[67905]: jbigkit-2.1_1 installed
Jan 10 15:56:22 ccia1 pkg-static[67905]: png-1.6.37_1 installed
Jan 10 15:56:22 ccia1 pkg-static[67905]: uchardet-0.0.7 installed
Jan 10 15:56:22 ccia1 pkg-static[67905]: psutils-1.17_5 installed
Jan 10 15:56:23 ccia1 pkg-static[67905]: tiff-4.3.0 installed
Jan 10 15:56:24 ccia1 pkg-static[67905]: giflib-5.2.1 installed
Jan 10 15:56:24 ccia1 pkg-static[67905]: libfontenc-1.1.4 installed
Jan 10 15:56:24 ccia1 pkg-static[67905]: freetype2-2.11.0 installed
Jan 10 15:56:24 ccia1 pkg-static[67905]: openpgm-5.2.122_6 installed
Jan 10 15:56:24 ccia1 pkg-static[67905]: norm-1.5r6_1 installed
Jan 10 15:56:25 ccia1 pkg-static[67905]: groff-1.22.4_4 installed
Jan 10 15:56:26 ccia1 pkg-static[67905]: protobuf-3.17.3,1 installed
Jan 10 15:56:26 ccia1 pkg-static[67905]: webp-1.2.1 installed
Jan 10 15:56:26 ccia1 pkg-static[67905]: mkfontscale-1.2.1 installed
Jan 10 15:56:27 ccia1 pkg-static[67905]: fontconfig-2.13.94_1,1 installed
Jan 10 15:56:27 ccia1 pkg-static[67905]: libsodium-1.0.18 installed
Jan 10 15:56:28 ccia1 pkg-static[67905]: ndpi-4.0.d20210818,1 installed
Jan 10 15:56:28 ccia1 pkg-static[67905]: libzmq4-4.3.4 installed
Jan 10 15:56:28 ccia1 pkg-static[67905]: libmaxminddb-1.6.0 installed
Jan 10 15:56:29 ccia1 pkg-static[67905]: lua53-5.3.6 installed
Jan 10 15:56:30 ccia1 pkg-static[67905]: mysql57-client-5.7.36 installed
Jan 10 15:56:30 ccia1 pkg-static[67905]: zstd-1.5.0 installed
Jan 10 15:56:30 ccia1 pkg-static[67905]: libgd-2.3.1,1 installed
Jan 10 15:56:33 ccia1 pkg-static[67905]: webfonts-0.30_14 installed
Jan 10 15:56:35 ccia1 pkg-static[67905]: ntopng-5.0.d20210923,1 installed
Jan 10 15:56:36 ccia1 pkg-static[67905]: graphviz-2.44.1_17 installed
Jan 10 15:56:37 ccia1 pkg-static[67905]: redis-6.2.6 installed
Jan 10 15:56:37 ccia1 pkg-static[67905]: gdbm-1.22 installed
Jan 10 15:56:37 ccia1 php[68850]: /etc/rc.packages: Beginning package installation for ntopng .
Jan 10 15:56:38 ccia1 php[68850]: /etc/rc.packages: Configuration Change: (system): Intermediate config write during package install for ntopng.
Jan 10 15:56:38 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 15:57:21 ccia1 php[68850]: /etc/rc.packages: Configuration Change: (system): Installed ntopng package.
Jan 10 15:57:21 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 15:57:21 ccia1 php[68850]: /etc/rc.packages: Successfully installed package: ntopng.
Jan 10 15:57:22 ccia1 pkg-static[67905]: pfSense-pkg-ntopng-0.8.13_10 installed
Jan 10 15:57:23 ccia1 check_reload_status[397]: Reloading filter
Jan 10 15:57:23 ccia1 check_reload_status[397]: Starting packages
Jan 10 15:57:24 ccia1 php-fpm[368]: /rc.start_packages: Restarting/Starting all packages.
Jan 10 15:57:24 ccia1 php-fpm[368]: /rc.start_packages: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found'
Jan 10 15:57:27 ccia1 kernel: re0: promiscuous mode enabled
Jan 10 15:57:27 ccia1 kernel: re1: promiscuous mode enabled
Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1391] ERROR: [HTTP] set_ports_option: cannot bind to 3000s: Address already in use
Jan 10 15:57:45 ccia1 ntopng[62182]: [mongoose.c:4645] ERROR: set_ports_option: cannot bind to 3000s: No error: 0
Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1681] ERROR: Unable to start HTTP server (IPv4) on ports 3000s
Jan 10 15:57:45 ccia1 ntopng[62182]: [HTTPserver.cpp:1687] ERROR: Either port in use or another ntopng instance is running (using the same port)
Jan 10 16:19:09 ccia1 php-fpm[367]: /pkg_edit.php: Configuration Change:
Jan 10 16:19:09 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 16:19:12 ccia1 kernel: re0: promiscuous mode disabled
Jan 10 16:19:12 ccia1 kernel: re1: promiscuous mode disabled
Jan 10 16:19:12 ccia1 php-fpm[367]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found'
Jan 10 16:19:13 ccia1 kernel: re0: promiscuous mode enabled
Jan 10 16:19:13 ccia1 kernel: re1: promiscuous mode enabled
Jan 10 16:25:00 ccia1 sshguard[22836]: Exiting on signal.
Jan 10 16:25:00 ccia1 sshguard[38778]: Now monitoring attacks.
Jan 10 16:32:00 ccia1 sshguard[38778]: Exiting on signal.
Jan 10 16:32:00 ccia1 sshguard[58980]: Now monitoring attacks.
Jan 10 17:08:00 ccia1 sshguard[99400]: Now monitoring attacks.
Jan 10 17:19:28 ccia1 php-fpm[368]: /pkg_edit.php: Configuration Change:
Jan 10 17:19:28 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 17:19:32 ccia1 kernel: re0: promiscuous mode disabled
Jan 10 17:19:32 ccia1 kernel: re1: promiscuous mode disabled
Jan 10 17:19:35 ccia1 php-fpm[368]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found'
Jan 10 17:19:35 ccia1 kernel: re0: promiscuous mode enabled
Jan 10 17:19:35 ccia1 kernel: re1: promiscuous mode enabled
Jan 10 17:19:35 ccia1 kernel: re2: promiscuous mode enabled
Jan 10 17:36:46 ccia1 php-fpm[367]: /pkg_edit.php: Configuration Change:
Jan 10 17:36:46 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 17:36:49 ccia1 kernel: re0: promiscuous mode disabled
Jan 10 17:36:49 ccia1 kernel: re1: promiscuous mode disabled
Jan 10 17:36:49 ccia1 kernel: re2: promiscuous mode disabled
Jan 10 17:36:52 ccia1 php-fpm[367]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found'
Jan 10 17:36:53 ccia1 kernel: re0: promiscuous mode enabled
Jan 10 17:36:53 ccia1 kernel: re1: promiscuous mode enabled
Jan 10 17:36:53 ccia1 kernel: re2: promiscuous mode enabled
Jan 10 17:37:06 ccia1 php-fpm[368]: /pkg_edit.php: Configuration Change:
Jan 10 17:37:06 ccia1 check_reload_status[397]: Syncing firewall
Jan 10 17:37:09 ccia1 kernel: re0: promiscuous mode disabled
Jan 10 17:37:09 ccia1 kernel: re1: promiscuous mode disabled
Jan 10 17:37:09 ccia1 kernel: re2: promiscuous mode disabled
Jan 10 17:37:12 ccia1 php-fpm[368]: /pkg_edit.php: The command '/usr/local/etc/rc.d/ntopng.sh stop' returned exit code '1', the output was 'No matching processes were found No matching processes were found'
Jan 10 17:37:13 ccia1 kernel: re0: promiscuous mode enabled
Jan 10 17:37:13 ccia1 kernel: re2: promiscuous mode enabled
Jan 10 17:43:00 ccia1 sshguard[99400]: Exiting on signal.
Jan 10 17:43:00 ccia1 sshguard[88504]: Now monitoring attacks.
, client: 10.1.0.2, server: 0.0.0.0:48562
Jan 10 18:26:00 ccia1 sshguard[88504]: Exiting on signal.
Jan 10 18:26:00 ccia1 sshguard[14898]: Now monitoring attacks.
Jan 10 19:15:00 ccia1 sshguard[14898]: Exiting on signal.
Jan 10 19:15:00 ccia1 sshguard[45834]: Now monitoring attacks.
Jan 10 20:04:00 ccia1 sshguard[45834]: Exiting on signal.
Jan 10 20:04:00 ccia1 sshguard[87400]: Now monitoring attacks.
Jan 10 20:04:04 ccia1 nginx: 2023/01/10 20:04:04 [crit] 601#100529: *2472 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 10.1.0.2, server: 0.0.0.0:48562
Jan 10 20:56:00 ccia1 sshguard[87400]: Exiting on signal.
Jan 10 20:56:00 ccia1 sshguard[16073]: Now monitoring attacks.
Jan 10 21:48:00 ccia1 sshguard[16073]: Exiting on signal.
Jan 10 21:48:00 ccia1 sshguard[47268]: Now monitoring attacks.
Jan 10 22:43:00 ccia1 sshguard[47268]: Exiting on signal.
Jan 10 22:43:00 ccia1 sshguard[82446]: Now monitoring attacks.
Jan 10 23:39:00 ccia1 sshguard[82446]: Exiting on signal.
Jan 10 23:39:00 ccia1 sshguard[18269]: Now monitoring attacks.
                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Hmm, so not really.

                        What packages do you have running there?

                        1 Reply Last reply Reply Quote 0
                        • hendiH
                          hendi
                          last edited by

                          ntopng net 0.8.13_10
                          openvpn-client-export security 1.6_9
                          pfBlockerNG-devel net 3.1.0_11

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Hmm, you might try disabling ntop-ng as a test. It can use resources and cause problems. Though I don't see any direct evidence of that here.

                            hendiH 1 Reply Last reply Reply Quote 0
                            • hendiH
                              hendi @stephenw10
                              last edited by

                              @stephenw10 said in Monitoring Gateway:

                              Hmm, you might try disabling ntop-ng as a test. It can use resources and cause problems. Though I don't see any direct evidence of that here.

                              I tried this from the beginning but without result.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post