Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    sg-3100 (2.4.4_3) wedges itself every 3 months -- suspect ntopng (0.8.13_3)

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    9 Posts 3 Posters 464 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • lohphatL
      lohphat
      last edited by lohphat

      About every 3 months my sg-3100 stops responding and rebooting shows a corrupt filesystem/config.

      I currently suspect ntopng is somehow consuming resources but the system is in such an unstable state when it fails that it needs to be reloaded via USB to recover. Currently I have it uninstalled to see what's going on.

      When it is wedged, the console reports missing dynamic libraries to the point the UI can't load, any KB input on the console port just results in rapidly scrolling error messages.

      These are the current modules I have running as I've tried to strip down to the bare minimum.

      Cron 0.3.7_3
      pfBlockerNG-devel 2.2.5_25
      Shellcmd 1.0.5_1
      snort 3.2.9.9_1
      Does anyone have any recommendations of how I should triage this?

      Update: I reinstalled ntopng to restart the investivation and the first reboot after the package install, it wedged. Logs follow. The consule UI is usless, can't even reboot from the menu.

      Here's the relevant errors during bootup:

      Welcome to pfSense 2.4.4-RELEASE (Patch 3)...


...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.26/mach/CORE
Soft Float compatibility ldconfig path:
done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/ecl.php(21): require_once('/etc/inc/functi...')
#2 {main}
  thrown in /etc/inc/certs.inc on line 46
Launching the init system...
Warning: Failed loading Zend extension 'opcache.so' (tried: /usr/local/lib/php/20131226/opcache.so (Cannot open "/usr/local/lib/php/20131226/opcache.so"), /usr/local/lib/php/20131226/opcache.so.so (Cannot open "/usr/local/lib/php/20131226/opcache.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'session.so' (tried: /usr/local/lib/php/20131226/session.so (Cannot open "/usr/local/lib/php/20131226/session.so"), /usr/local/lib/php/20131226/session.so.so (Cannot open "/usr/local/lib/php/20131226/session.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'bcmath.so' (tried: /usr/local/lib/php/20131226/bcmath.so (Cannot open "/usr/local/lib/php/20131226/bcmath.so"), /usr/local/lib/php/20131226/bcmath.so.so (Cannot open "/usr/local/lib/php/20131226/bcmath.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'bz2.so' (tried: /usr/local/lib/php/20131226/bz2.so (Cannot open "/usr/local/lib/php/20131226/bz2.so"), /usr/local/lib/php/20131226/bz2.so.so (Cannot open "/usr/local/lib/php/20131226/bz2.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'ctype.so' (tried: /usr/local/lib/php/20131226/ctype.so (Cannot open "/usr/local/lib/php/20131226/ctype.so"), /usr/local/lib/php/20131226/ctype.so.so (Cannot open "/usr/local/lib/php/20131226/ctype.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20131226/curl.so (Cannot open "/usr/local/lib/php/20131226/curl.so"), /usr/local/lib/php/20131226/curl.so.so (Cannot open "/usr/local/lib/php/20131226/curl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'dom.so' (tried: /usr/local/lib/php/20131226/dom.so (Cannot open "/usr/local/lib/php/20131226/dom.so"), /usr/local/lib/php/20131226/dom.so.so (Cannot open "/usr/local/lib/php/20131226/dom.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'filter.so' (tried: /usr/local/lib/php/20131226/filter.so (Cannot open "/usr/local/lib/php/20131226/filter.so"), /usr/local/lib/php/20131226/filter.so.so (Cannot open "/usr/local/lib/php/20131226/filter.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'gettext.so' (tried: /usr/local/lib/php/20131226/gettext.so (Cannot open "/usr/local/lib/php/20131226/gettext.so"), /usr/local/lib/php/20131226/gettext.so.so (Cannot open "/usr/local/lib/php/20131226/gettext.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'hash.so' (tried: /usr/local/lib/php/20131226/hash.so (Cannot open "/usr/local/lib/php/20131226/hash.so"), /usr/local/lib/php/20131226/hash.so.so (Cannot open "/usr/local/lib/php/20131226/hash.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20131226/intl.so (Cannot open "/usr/local/lib/php/20131226/intl.so"), /usr/local/lib/php/20131226/intl.so.so (Cannot open "/usr/local/lib/php/20131226/intl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'json.so' (tried: /usr/local/lib/php/20131226/json.so (Cannot open "/usr/local/lib/php/20131226/json.so"), /usr/local/lib/php/20131226/json.so.so (Cannot open "/usr/local/lib/php/20131226/json.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/local/lib/php/20131226/ldap.so (Cannot open "/usr/local/lib/php/20131226/ldap.so"), /usr/local/lib/php/20131226/ldap.so.so (Cannot open "/usr/local/lib/php/20131226/ldap.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'mbstring.so' (tried: /usr/local/lib/php/20131226/mbstring.so (Cannot open "/usr/local/lib/php/20131226/mbstring.so"), /usr/local/lib/php/20131226/mbstring.so.so (Cannot open "/usr/local/lib/php/20131226/mbstring.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'mcrypt.so' (tried: /usr/local/lib/php/20131226/mcrypt.so (Cannot open "/usr/local/lib/php/20131226/mcrypt.so"), /usr/local/lib/php/20131226/mcrypt.so.so (Cannot open "/usr/local/lib/php/20131226/mcrypt.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'openssl.so' (tried: /usr/local/lib/php/20131226/openssl.so (Cannot open "/usr/local/lib/php/20131226/openssl.so"), /usr/local/lib/php/20131226/openssl.so.so (Cannot open "/usr/local/lib/php/20131226/openssl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pcntl.so' (tried: /usr/local/lib/php/20131226/pcntl.so (Cannot open "/usr/local/lib/php/20131226/pcntl.so"), /usr/local/lib/php/20131226/pcntl.so.so (Cannot open "/usr/local/lib/php/20131226/pcntl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pdo.so' (tried: /usr/local/lib/php/20131226/pdo.so (Cannot open "/usr/local/lib/php/20131226/pdo.so"), /usr/local/lib/php/20131226/pdo.so.so (Cannot open "/usr/local/lib/php/20131226/pdo.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pfSense.so' (tried: /usr/local/lib/php/20131226/pfSense.so (Cannot open "/usr/local/lib/php/20131226/pfSense.so"), /usr/local/lib/php/20131226/pfSense.so.so (Cannot open "/usr/local/lib/php/20131226/pfSense.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'posix.so' (tried: /usr/local/lib/php/20131226/posix.so (Cannot open "/usr/local/lib/php/20131226/posix.so"), /usr/local/lib/php/20131226/posix.so.so (Cannot open "/usr/local/lib/php/20131226/posix.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'radius.so' (tried: /usr/local/lib/php/20131226/radius.so (Cannot open "/usr/local/lib/php/20131226/radius.so"), /usr/local/lib/php/20131226/radius.so.so (Cannot open "/usr/local/lib/php/20131226/radius.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'readline.so' (tried: /usr/local/lib/php/20131226/readline.so (Cannot open "/usr/local/lib/php/20131226/readline.so"), /usr/local/lib/php/20131226/readline.so.so (Cannot open "/usr/local/lib/php/20131226/readline.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'rrd.so' (tried: /usr/local/lib/php/20131226/rrd.so (Cannot open "/usr/local/lib/php/20131226/rrd.so"), /usr/local/lib/php/20131226/rrd.so.so (Cannot open "/usr/local/lib/php/20131226/rrd.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'shmop.so' (tried: /usr/local/lib/php/20131226/shmop.so (Cannot open "/usr/local/lib/php/20131226/shmop.so"), /usr/local/lib/php/20131226/shmop.so.so (Cannot open "/usr/local/lib/php/20131226/shmop.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'simplexml.so' (tried: /usr/local/lib/php/20131226/simplexml.so (Cannot open "/usr/local/lib/php/20131226/simplexml.so"), /usr/local/lib/php/20131226/simplexml.so.so (Cannot open "/usr/local/lib/php/20131226/simplexml.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sockets.so' (tried: /usr/local/lib/php/20131226/sockets.so (Cannot open "/usr/local/lib/php/20131226/sockets.so"), /usr/local/lib/php/20131226/sockets.so.so (Cannot open "/usr/local/lib/php/20131226/sockets.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sqlite3.so' (tried: /usr/local/lib/php/20131226/sqlite3.so (Cannot open "/usr/local/lib/php/20131226/sqlite3.so"), /usr/local/lib/php/20131226/sqlite3.so.so (Cannot open "/usr/local/lib/php/20131226/sqlite3.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvmsg.so' (tried: /usr/local/lib/php/20131226/sysvmsg.so (Cannot open "/usr/local/lib/php/20131226/sysvmsg.so"), /usr/local/lib/php/20131226/sysvmsg.so.so (Cannot open "/usr/local/lib/php/20131226/sysvmsg.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvsem.so' (tried: /usr/local/lib/php/20131226/sysvsem.so (Cannot open "/usr/local/lib/php/20131226/sysvsem.so"), /usr/local/lib/php/20131226/sysvsem.so.so (Cannot open "/usr/local/lib/php/20131226/sysvsem.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvshm.so' (tried: /usr/local/lib/php/20131226/sysvshm.so (Cannot open "/usr/local/lib/php/20131226/sysvshm.so"), /usr/local/lib/php/20131226/sysvshm.so.so (Cannot open "/usr/local/lib/php/20131226/sysvshm.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'tokenizer.so' (tried: /usr/local/lib/php/20131226/tokenizer.so (Cannot open "/usr/local/lib/php/20131226/tokenizer.so"), /usr/local/lib/php/20131226/tokenizer.so.so (Cannot open "/usr/local/lib/php/20131226/tokenizer.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xml.so' (tried: /usr/local/lib/php/20131226/xml.so (Cannot open "/usr/local/lib/php/20131226/xml.so"), /usr/local/lib/php/20131226/xml.so.so (Cannot open "/usr/local/lib/php/20131226/xml.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xmlwriter.so' (tried: /usr/local/lib/php/20131226/xmlwriter.so (Cannot open "/usr/local/lib/php/20131226/xmlwriter.so"), /usr/local/lib/php/20131226/xmlwriter.so.so (Cannot open "/usr/local/lib/php/20131226/xmlwriter.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'zlib.so' (tried: /usr/local/lib/php/20131226/zlib.so (Cannot open "/usr/local/lib/php/20131226/zlib.so"), /usr/local/lib/php/20131226/zlib.so.so (Cannot open "/usr/local/lib/php/20131226/zlib.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'zmq.so' (tried: /usr/local/lib/php/20131226/zmq.so (Cannot open "/usr/local/lib/php/20131226/zmq.so"), /usr/local/lib/php/20131226/zmq.so.so (Cannot open "/usr/local/lib/php/20131226/zmq.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pdo_sqlite.so' (tried: /usr/local/lib/php/20131226/pdo_sqlite.so (Cannot open "/usr/local/lib/php/20131226/pdo_sqlite.so"), /usr/local/lib/php/20131226/pdo_sqlite.so.so (Cannot open "/usr/local/lib/php/20131226/pdo_sqlite.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xmlreader.so' (tried: /usr/local/lib/php/20131226/xmlreader.so (Cannot open "/usr/local/lib/php/20131226/xmlreader.so"), /usr/local/lib/php/20131226/xmlreader.so.so (Cannot open "/usr/local/lib/php/20131226/xmlreader.so.so")) in Unknown on line 0

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.gui.inc(37): require_once('/etc/inc/notice...')
#3 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#4 /etc/inc/captiveportal.inc(27): require_once('/etc/inc/auth.i...')
#5 /etc/inc/service-utils.inc(23): require_once('/etc/inc/captiv...')
#6 /etc/inc/pkg-utils.inc(24): require_once('/etc/inc/servic...')
#7 /etc/rc.bootup(24): require_once('/etc/inc/pkg-ut...')
#8 {main}
  thrown in /etc/inc/certs.inc on line 46
PHP ERROR: Type: 1, File: /etc/inc/certs.inc, Line: 46, Message: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.gui.inc(37): require_once('/etc/inc/notice...')
#3 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#4 /etc/inc/captiveportal.inc(27): require_once('/etc/inc/auth.i...')
#5 /etc/inc/service-utils.inc(23): require_once('/etc/inc/captiv...')
#6 /etc/inc/pkg-utils.inc(24): require_once('/etc/inc/servic...')
#7 /etc/rc.bootup(24): require_once('/etc/inc/pkg-ut...')
#8 {main}
  thrown
Fatal error: Uncaught Error: Call to undefined function log_error() in /etc/inc/config.lib.inc:950
Stack trace:
#0 [internal function]: pfSense_clear_globals()
#1 {main}
  thrown in /etc/inc/config.lib.inc on line 950
Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(49): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(22): require_once('/etc/inc/config...')
#4 {main}
  thrown in /etc/inc/certs.inc on line 46
pfSense 2.4.4-RELEASE (Patch 3) arm Thu May 16 06:01:19 EDT 2019
Bootup complete

      SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_5)

      1 Reply Last reply Reply Quote 0
      • A
        AEITS_Inc
        last edited by

        How many lists are loaded in pfBlockerNG? I have had that consume all availble memory when it did a list update, and lock the system similar to what you are experiencing.

        lohphatL 1 Reply Last reply Reply Quote 0
        • lohphatL
          lohphat @AEITS_Inc
          last edited by

          @AEITS_Inc Just the PRI1 and EasyList feeds. Memory use stays around 30%.

          SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_5)

          1 Reply Last reply Reply Quote 0
          • lohphatL
            lohphat
            last edited by

            I've restored the unit to factory defaults and just installed pfblockerNG-devel to see if that's the culprit. I'll report back later to see if there's any change.

            SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_5)

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              When it stops responding does it stop entirely? Even at the console?

              If you can catch it doing that next time try entering Ctrl+t at the console. That can often respond when nothing else does if it's hung up on a process. It should show you what process is using CPU time.

              I would suspect Snort before the other packages there unless it's completely exhausting the RAM. You should notice that filling over the 3 months though if that was the case.

              Steve

              lohphatL 1 Reply Last reply Reply Quote 0
              • lohphatL
                lohphat @stephenw10
                last edited by

                @stephenw10 No. The console is there but when a key is hit I get the scroll above of the *.so file errors. No menu item will work not even reboot. I haveto hard power cycle. But upon reboot the *.so errors are still there and the unit needs a full USB reload.

                SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_5)

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  That's before you reboot? You spontaneously start seeing library errors without rebooting?
                  I have never seen that before if that's the case.

                  Are you running from SSD or eMMC? I would certainly want an SSD for those packages.

                  Steve

                  lohphatL 1 Reply Last reply Reply Quote 0
                  • lohphatL
                    lohphat @stephenw10
                    last edited by

                    @stephenw10 It's an sg-3100 it is what it is. Those errors appear after I either type anything on the console or after a reboot after the f/w and UI stop responding.

                    SG-3100 24.11-RELEASE (arm) | Avahi (2.2_6) | ntopng (5.6.0_1) | openvpn-client-export (1.9.5) | pfBlockerNG-devel (3.2.1_20) | System_Patches (2.2.20_5)

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      The SG-3100 is available with or without an m.2 SSD from the factory, or you may have fitted one afterwards.

                      The / slice will be ~8GB if it's running from eMMC.

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.