Suricata on the SG-3100 does not survive a firmware upgrade
-
Indeed I'm running dev and hence upgrading frequently which is when I spotted it.
This seems pretty conclusive as to why it's failing to start:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V Shared object "libnss3.so" not found, required by "suricata"
Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.
It does appear to be there though so some path missing?
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*' /usr/local/lib/nss/libnss3.so
Steve
-
Indeed I'm running dev and hence upgrading frequently which is when I spotted it.
This seems pretty conclusive as to why it's failing to start:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V Shared object "libnss3.so" not found, required by "suricata"
Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.
It does appear to be there though so some path missing?
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*' /usr/local/lib/nss/libnss3.so
Steve
It might be a required symlink is missing.
<rant on=""></rant>
I hate shared libraries! They cause this kind of stuff. Same as the old "DLL hell" experienced on Windows.We will probably need to run this one by Renato on the pfSense team to see if he can determine what's going on. I have not been following the latest DEV build. It's a different FreeBSD tree isn't it? If so, the FreeBSD-ports part of the pfSense fork may need an update to match upstream.
Bill
-
Indeed I'm running dev and hence upgrading frequently which is when I spotted it.
This seems pretty conclusive as to why it's failing to start:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V Shared object "libnss3.so" not found, required by "suricata"
Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.
It does appear to be there though so some path missing?
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*' /usr/local/lib/nss/libnss3.so
Steve
What is the output of the following commands?
pkg upgrade -n
ldd /usr/local/bin/suricata
-
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking for upgrades (6 candidates): 100% Processing candidates (6 candidates): 100% The following 6 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense] Number of packages to be upgraded: 6 37 MiB to be downloaded.
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata /usr/local/bin/suricata: libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000) libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000) libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000) libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000) libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000) libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000) libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000) libthr.so.3 => /lib/libthr.so.3 (0x203d8000) libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000) libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000) libnss3.so => not found (0) libsmime3.so => not found (0) libssl3.so => not found (0) libnssutil3.so => not found (0) libplds4.so => /usr/local/lib/libplds4.so (0x204dc000) libplc4.so => /usr/local/lib/libplc4.so (0x204e6000) libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000) libc.so.7 => /lib/libc.so.7 (0x20600000) libm.so.5 => /lib/libm.so.5 (0x2052e000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000) libz.so.6 => /lib/libz.so.6 (0x2056a000) libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)
-
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. Checking for upgrades (6 candidates): 100% Processing candidates (6 candidates): 100% The following 6 package(s) will be affected (of 0 checked): Installed packages to be UPGRADED: pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core] pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense] Number of packages to be upgraded: 6 37 MiB to be downloaded.
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata /usr/local/bin/suricata: libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000) libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000) libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000) libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000) libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000) libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000) libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000) libthr.so.3 => /lib/libthr.so.3 (0x203d8000) libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000) libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000) libnss3.so => not found (0) libsmime3.so => not found (0) libssl3.so => not found (0) libnssutil3.so => not found (0) libplds4.so => /usr/local/lib/libplds4.so (0x204dc000) libplc4.so => /usr/local/lib/libplc4.so (0x204e6000) libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000) libc.so.7 => /lib/libc.so.7 (0x20600000) libm.so.5 => /lib/libm.so.5 (0x2052e000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000) libz.so.6 => /lib/libz.so.6 (0x2056a000) libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)
There are 2 possibilities in this case. Or nss is not installed or /usr/local/lib/nss is not being tracked by ld. Please run the following commands and let me know the result.
pkg info nss
cat /usr/local/libdata/ldconfig/nss
-
Here you go:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss nss-3.35 Name : nss Version : 3.35 Installed on : Sun Feb 11 16:06:28 2018 GMT Origin : security/nss Architecture : FreeBSD:11:armv6 Prefix : /usr/local Categories : security Licenses : MPL20 Maintainer : gecko@FreeBSD.org WWW : http://www.mozilla.org/projects/security/pki/nss/ Comment : Libraries to support development of security-enabled applications Options : DEBUG : off Shared Libs required: libplc4.so libsqlite3.so.0 libnspr4.so libplds4.so Shared Libs provided: libnss3.so libfreeblpriv3.so libnssckbi.so libnssutil3.so libfreebl3.so libsmime3.so libsoftokn3.so libnssdbm3.so libssl3.so Annotations : cpe : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6 repo_type : binary repository : pfSense Flat size : 7.76MiB Description : Network Security Services (NSS) is a set of libraries designed to support cross-platform development of security-enabled server applications. Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7, PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security standards. WWW: http://www.mozilla.org/projects/security/pki/nss/
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: cat /usr/local/libdata/ldconfig/nss /usr/local/lib/nss
Steve
-
Here you go:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss nss-3.35 Name : nss Version : 3.35 Installed on : Sun Feb 11 16:06:28 2018 GMT Origin : security/nss Architecture : FreeBSD:11:armv6 Prefix : /usr/local Categories : security Licenses : MPL20 Maintainer : gecko@FreeBSD.org WWW : http://www.mozilla.org/projects/security/pki/nss/ Comment : Libraries to support development of security-enabled applications Options : DEBUG : off Shared Libs required: libplc4.so libsqlite3.so.0 libnspr4.so libplds4.so Shared Libs provided: libnss3.so libfreeblpriv3.so libnssckbi.so libnssutil3.so libfreebl3.so libsmime3.so libsoftokn3.so libnssdbm3.so libssl3.so Annotations : cpe : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6 repo_type : binary repository : pfSense Flat size : 7.76MiB Description : Network Security Services (NSS) is a set of libraries designed to support cross-platform development of security-enabled server applications. Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7, PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security standards. WWW: http://www.mozilla.org/projects/security/pki/nss/
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: cat /usr/local/libdata/ldconfig/nss /usr/local/lib/nss
Steve
ld database is out of date for some reason. Try to run
/etc/rc.d/ldconfig start
and thenldd /usr/local/bin/suricata
again to see if it fixes the issue -
Yup, looks like that resolved it:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE Soft Float compatibility ldconfig path: [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata /usr/local/bin/suricata: libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000) libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000) libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000) libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000) libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000) libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000) libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000) libthr.so.3 => /lib/libthr.so.3 (0x203d8000) libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000) libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000) libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000) libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000) libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000) libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000) libplds4.so => /usr/local/lib/libplds4.so (0x204e5000) libplc4.so => /usr/local/lib/libplc4.so (0x204ef000) libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000) libc.so.7 => /lib/libc.so.7 (0x20700000) libm.so.5 => /lib/libm.so.5 (0x20868000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000) libz.so.6 => /lib/libz.so.6 (0x2089b000) libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)
Suricata now starts normally. :)
Steve
-
Yup, looks like that resolved it:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE Soft Float compatibility ldconfig path: [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata /usr/local/bin/suricata: libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000) libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000) libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000) libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000) libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000) libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000) libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000) libthr.so.3 => /lib/libthr.so.3 (0x203d8000) libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000) libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000) libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000) libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000) libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000) libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000) libplds4.so => /usr/local/lib/libplds4.so (0x204e5000) libplc4.so => /usr/local/lib/libplc4.so (0x204ef000) libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000) libc.so.7 => /lib/libc.so.7 (0x20700000) libm.so.5 => /lib/libm.so.5 (0x20868000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000) libz.so.6 => /lib/libz.so.6 (0x2089b000) libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)
Suricata now starts normally. :)
Steve
Good. I don't know exactly what caused it but I'm glad we figured it out
-
I'm still seeing this and now on both x86 and ARM.
Opened a bug to track it: https://redmine.pfsense.org/issues/8716 -
@stephenw10
This one will need to be looked at by Renato at Netgate. I'm thinking it's related to a change in the FreeBSD patch files for Suricata. I noticed the binary package maintainer for FreeBSD removed a patch file with "nss3" in its name from the latest 4.0.5 source code on FreeBSD ports, so I am surmising that may be where the libnss3.so library error is coming from.Bill
-
Ah, thanks for that. We'll look into it here then.
Steve