3. sg-3100 (2.4.4_3) wedges itself every 3 months -- suspect ntopng (0.8.13_3)

sg-3100 (2.4.4_3) wedges itself every 3 months -- suspect ntopng (0.8.13_3)

  • L

    About every 3 months my sg-3100 stops responding and rebooting shows a corrupt filesystem/config.

    I currently suspect ntopng is somehow consuming resources but the system is in such an unstable state when it fails that it needs to be reloaded via USB to recover. Currently I have it uninstalled to see what's going on.

    When it is wedged, the console reports missing dynamic libraries to the point the UI can't load, any KB input on the console port just results in rapidly scrolling error messages.

    These are the current modules I have running as I've tried to strip down to the bare minimum.

    Cron 0.3.7_3
    pfBlockerNG-devel 2.2.5_25
    Shellcmd 1.0.5_1
    snort 3.2.9.9_1
    Does anyone have any recommendations of how I should triage this?

    Update: I reinstalled ntopng to restart the investivation and the first reboot after the package install, it wedged. Logs follow. The consule UI is usless, can't even reboot from the menu.

    Here's the relevant errors during bootup:

    Welcome to pfSense 2.4.4-RELEASE (Patch 3)...


...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.26/mach/CORE
Soft Float compatibility ldconfig path:
done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/ecl.php(21): require_once('/etc/inc/functi...')
#2 {main}
  thrown in /etc/inc/certs.inc on line 46
Launching the init system...
Warning: Failed loading Zend extension 'opcache.so' (tried: /usr/local/lib/php/20131226/opcache.so (Cannot open "/usr/local/lib/php/20131226/opcache.so"), /usr/local/lib/php/20131226/opcache.so.so (Cannot open "/usr/local/lib/php/20131226/opcache.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'session.so' (tried: /usr/local/lib/php/20131226/session.so (Cannot open "/usr/local/lib/php/20131226/session.so"), /usr/local/lib/php/20131226/session.so.so (Cannot open "/usr/local/lib/php/20131226/session.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'bcmath.so' (tried: /usr/local/lib/php/20131226/bcmath.so (Cannot open "/usr/local/lib/php/20131226/bcmath.so"), /usr/local/lib/php/20131226/bcmath.so.so (Cannot open "/usr/local/lib/php/20131226/bcmath.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'bz2.so' (tried: /usr/local/lib/php/20131226/bz2.so (Cannot open "/usr/local/lib/php/20131226/bz2.so"), /usr/local/lib/php/20131226/bz2.so.so (Cannot open "/usr/local/lib/php/20131226/bz2.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'ctype.so' (tried: /usr/local/lib/php/20131226/ctype.so (Cannot open "/usr/local/lib/php/20131226/ctype.so"), /usr/local/lib/php/20131226/ctype.so.so (Cannot open "/usr/local/lib/php/20131226/ctype.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'curl.so' (tried: /usr/local/lib/php/20131226/curl.so (Cannot open "/usr/local/lib/php/20131226/curl.so"), /usr/local/lib/php/20131226/curl.so.so (Cannot open "/usr/local/lib/php/20131226/curl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'dom.so' (tried: /usr/local/lib/php/20131226/dom.so (Cannot open "/usr/local/lib/php/20131226/dom.so"), /usr/local/lib/php/20131226/dom.so.so (Cannot open "/usr/local/lib/php/20131226/dom.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'filter.so' (tried: /usr/local/lib/php/20131226/filter.so (Cannot open "/usr/local/lib/php/20131226/filter.so"), /usr/local/lib/php/20131226/filter.so.so (Cannot open "/usr/local/lib/php/20131226/filter.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'gettext.so' (tried: /usr/local/lib/php/20131226/gettext.so (Cannot open "/usr/local/lib/php/20131226/gettext.so"), /usr/local/lib/php/20131226/gettext.so.so (Cannot open "/usr/local/lib/php/20131226/gettext.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'hash.so' (tried: /usr/local/lib/php/20131226/hash.so (Cannot open "/usr/local/lib/php/20131226/hash.so"), /usr/local/lib/php/20131226/hash.so.so (Cannot open "/usr/local/lib/php/20131226/hash.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20131226/intl.so (Cannot open "/usr/local/lib/php/20131226/intl.so"), /usr/local/lib/php/20131226/intl.so.so (Cannot open "/usr/local/lib/php/20131226/intl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'json.so' (tried: /usr/local/lib/php/20131226/json.so (Cannot open "/usr/local/lib/php/20131226/json.so"), /usr/local/lib/php/20131226/json.so.so (Cannot open "/usr/local/lib/php/20131226/json.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/local/lib/php/20131226/ldap.so (Cannot open "/usr/local/lib/php/20131226/ldap.so"), /usr/local/lib/php/20131226/ldap.so.so (Cannot open "/usr/local/lib/php/20131226/ldap.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'mbstring.so' (tried: /usr/local/lib/php/20131226/mbstring.so (Cannot open "/usr/local/lib/php/20131226/mbstring.so"), /usr/local/lib/php/20131226/mbstring.so.so (Cannot open "/usr/local/lib/php/20131226/mbstring.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'mcrypt.so' (tried: /usr/local/lib/php/20131226/mcrypt.so (Cannot open "/usr/local/lib/php/20131226/mcrypt.so"), /usr/local/lib/php/20131226/mcrypt.so.so (Cannot open "/usr/local/lib/php/20131226/mcrypt.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'openssl.so' (tried: /usr/local/lib/php/20131226/openssl.so (Cannot open "/usr/local/lib/php/20131226/openssl.so"), /usr/local/lib/php/20131226/openssl.so.so (Cannot open "/usr/local/lib/php/20131226/openssl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pcntl.so' (tried: /usr/local/lib/php/20131226/pcntl.so (Cannot open "/usr/local/lib/php/20131226/pcntl.so"), /usr/local/lib/php/20131226/pcntl.so.so (Cannot open "/usr/local/lib/php/20131226/pcntl.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pdo.so' (tried: /usr/local/lib/php/20131226/pdo.so (Cannot open "/usr/local/lib/php/20131226/pdo.so"), /usr/local/lib/php/20131226/pdo.so.so (Cannot open "/usr/local/lib/php/20131226/pdo.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pfSense.so' (tried: /usr/local/lib/php/20131226/pfSense.so (Cannot open "/usr/local/lib/php/20131226/pfSense.so"), /usr/local/lib/php/20131226/pfSense.so.so (Cannot open "/usr/local/lib/php/20131226/pfSense.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'posix.so' (tried: /usr/local/lib/php/20131226/posix.so (Cannot open "/usr/local/lib/php/20131226/posix.so"), /usr/local/lib/php/20131226/posix.so.so (Cannot open "/usr/local/lib/php/20131226/posix.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'radius.so' (tried: /usr/local/lib/php/20131226/radius.so (Cannot open "/usr/local/lib/php/20131226/radius.so"), /usr/local/lib/php/20131226/radius.so.so (Cannot open "/usr/local/lib/php/20131226/radius.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'readline.so' (tried: /usr/local/lib/php/20131226/readline.so (Cannot open "/usr/local/lib/php/20131226/readline.so"), /usr/local/lib/php/20131226/readline.so.so (Cannot open "/usr/local/lib/php/20131226/readline.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'rrd.so' (tried: /usr/local/lib/php/20131226/rrd.so (Cannot open "/usr/local/lib/php/20131226/rrd.so"), /usr/local/lib/php/20131226/rrd.so.so (Cannot open "/usr/local/lib/php/20131226/rrd.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'shmop.so' (tried: /usr/local/lib/php/20131226/shmop.so (Cannot open "/usr/local/lib/php/20131226/shmop.so"), /usr/local/lib/php/20131226/shmop.so.so (Cannot open "/usr/local/lib/php/20131226/shmop.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'simplexml.so' (tried: /usr/local/lib/php/20131226/simplexml.so (Cannot open "/usr/local/lib/php/20131226/simplexml.so"), /usr/local/lib/php/20131226/simplexml.so.so (Cannot open "/usr/local/lib/php/20131226/simplexml.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sockets.so' (tried: /usr/local/lib/php/20131226/sockets.so (Cannot open "/usr/local/lib/php/20131226/sockets.so"), /usr/local/lib/php/20131226/sockets.so.so (Cannot open "/usr/local/lib/php/20131226/sockets.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sqlite3.so' (tried: /usr/local/lib/php/20131226/sqlite3.so (Cannot open "/usr/local/lib/php/20131226/sqlite3.so"), /usr/local/lib/php/20131226/sqlite3.so.so (Cannot open "/usr/local/lib/php/20131226/sqlite3.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvmsg.so' (tried: /usr/local/lib/php/20131226/sysvmsg.so (Cannot open "/usr/local/lib/php/20131226/sysvmsg.so"), /usr/local/lib/php/20131226/sysvmsg.so.so (Cannot open "/usr/local/lib/php/20131226/sysvmsg.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvsem.so' (tried: /usr/local/lib/php/20131226/sysvsem.so (Cannot open "/usr/local/lib/php/20131226/sysvsem.so"), /usr/local/lib/php/20131226/sysvsem.so.so (Cannot open "/usr/local/lib/php/20131226/sysvsem.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvshm.so' (tried: /usr/local/lib/php/20131226/sysvshm.so (Cannot open "/usr/local/lib/php/20131226/sysvshm.so"), /usr/local/lib/php/20131226/sysvshm.so.so (Cannot open "/usr/local/lib/php/20131226/sysvshm.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'tokenizer.so' (tried: /usr/local/lib/php/20131226/tokenizer.so (Cannot open "/usr/local/lib/php/20131226/tokenizer.so"), /usr/local/lib/php/20131226/tokenizer.so.so (Cannot open "/usr/local/lib/php/20131226/tokenizer.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xml.so' (tried: /usr/local/lib/php/20131226/xml.so (Cannot open "/usr/local/lib/php/20131226/xml.so"), /usr/local/lib/php/20131226/xml.so.so (Cannot open "/usr/local/lib/php/20131226/xml.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xmlwriter.so' (tried: /usr/local/lib/php/20131226/xmlwriter.so (Cannot open "/usr/local/lib/php/20131226/xmlwriter.so"), /usr/local/lib/php/20131226/xmlwriter.so.so (Cannot open "/usr/local/lib/php/20131226/xmlwriter.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'zlib.so' (tried: /usr/local/lib/php/20131226/zlib.so (Cannot open "/usr/local/lib/php/20131226/zlib.so"), /usr/local/lib/php/20131226/zlib.so.so (Cannot open "/usr/local/lib/php/20131226/zlib.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'zmq.so' (tried: /usr/local/lib/php/20131226/zmq.so (Cannot open "/usr/local/lib/php/20131226/zmq.so"), /usr/local/lib/php/20131226/zmq.so.so (Cannot open "/usr/local/lib/php/20131226/zmq.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pdo_sqlite.so' (tried: /usr/local/lib/php/20131226/pdo_sqlite.so (Cannot open "/usr/local/lib/php/20131226/pdo_sqlite.so"), /usr/local/lib/php/20131226/pdo_sqlite.so.so (Cannot open "/usr/local/lib/php/20131226/pdo_sqlite.so.so")) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xmlreader.so' (tried: /usr/local/lib/php/20131226/xmlreader.so (Cannot open "/usr/local/lib/php/20131226/xmlreader.so"), /usr/local/lib/php/20131226/xmlreader.so.so (Cannot open "/usr/local/lib/php/20131226/xmlreader.so.so")) in Unknown on line 0

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.gui.inc(37): require_once('/etc/inc/notice...')
#3 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#4 /etc/inc/captiveportal.inc(27): require_once('/etc/inc/auth.i...')
#5 /etc/inc/service-utils.inc(23): require_once('/etc/inc/captiv...')
#6 /etc/inc/pkg-utils.inc(24): require_once('/etc/inc/servic...')
#7 /etc/rc.bootup(24): require_once('/etc/inc/pkg-ut...')
#8 {main}
  thrown in /etc/inc/certs.inc on line 46
PHP ERROR: Type: 1, File: /etc/inc/certs.inc, Line: 46, Message: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.gui.inc(37): require_once('/etc/inc/notice...')
#3 /etc/inc/auth.inc(31): require_once('/etc/inc/config...')
#4 /etc/inc/captiveportal.inc(27): require_once('/etc/inc/auth.i...')
#5 /etc/inc/service-utils.inc(23): require_once('/etc/inc/captiv...')
#6 /etc/inc/pkg-utils.inc(24): require_once('/etc/inc/servic...')
#7 /etc/rc.bootup(24): require_once('/etc/inc/pkg-ut...')
#8 {main}
  thrown
Fatal error: Uncaught Error: Call to undefined function log_error() in /etc/inc/config.lib.inc:950
Stack trace:
#0 [internal function]: pfSense_clear_globals()
#1 {main}
  thrown in /etc/inc/config.lib.inc on line 950
Starting CRON... done.

Fatal error: Uncaught Error: Call to undefined function gettext() in /etc/inc/certs.inc:46
Stack trace:
#0 /etc/inc/functions.inc(32): require_once()
#1 /etc/inc/notices.inc(24): require_once('/etc/inc/functi...')
#2 /etc/inc/config.inc(49): require_once('/etc/inc/notice...')
#3 /etc/rc.start_packages(22): require_once('/etc/inc/config...')
#4 {main}
  thrown in /etc/inc/certs.inc on line 46
pfSense 2.4.4-RELEASE (Patch 3) arm Thu May 16 06:01:19 EDT 2019
Bootup complete
    0
  • A

    How many lists are loaded in pfBlockerNG? I have had that consume all availble memory when it did a list update, and lock the system similar to what you are experiencing.

    0
    L
     1 Reply
  • L

    @AEITS_Inc Just the PRI1 and EasyList feeds. Memory use stays around 30%.

    0
  • L

    I've restored the unit to factory defaults and just installed pfblockerNG-devel to see if that's the culprit. I'll report back later to see if there's any change.

    0
  • Netgate Administrator

    When it stops responding does it stop entirely? Even at the console?

    If you can catch it doing that next time try entering Ctrl+t at the console. That can often respond when nothing else does if it's hung up on a process. It should show you what process is using CPU time.

    I would suspect Snort before the other packages there unless it's completely exhausting the RAM. You should notice that filling over the 3 months though if that was the case.

    Steve

    0
    L
     1 Reply
  • L

    @stephenw10 No. The console is there but when a key is hit I get the scroll above of the *.so file errors. No menu item will work not even reboot. I haveto hard power cycle. But upon reboot the *.so errors are still there and the unit needs a full USB reload.

    0
  • Netgate Administrator

    That's before you reboot? You spontaneously start seeing library errors without rebooting?
    I have never seen that before if that's the case.

    Are you running from SSD or eMMC? I would certainly want an SSD for those packages.

    Steve

    0
    L
     1 Reply
  • L

    @stephenw10 It's an sg-3100 it is what it is. Those errors appear after I either type anything on the console or after a reboot after the f/w and UI stop responding.

    0
  • Netgate Administrator

    The SG-3100 is available with or without an m.2 SSD from the factory, or you may have fitted one afterwards.

    The / slice will be ~8GB if it's running from eMMC.

    Steve

    0
Log in to reply
 

Products

Services

Support

News

Resources

Company

Our Mission

We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2002 - 2019 Rubicon Communications, LLC | Privacy Policy