Suricata on the SG-3100 does not survive a firmware upgrade

stephenw10 · Feb 15, 2018, 11:09 PM

It shows nothing at all.

The last entries there are from it stopping when I ran the upgrade:

13/2/2018 -- 20:58:23 - <notice> -- Signal Received.  Stopping engine.
13/2/2018 -- 20:58:23 - <info> -- time elapsed 190197.109s
13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Packets 1863609, bytes 241627739
13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Pcap Total:1863611 Recv:1863611 Drop:0 (0.0%).
13/2/2018 -- 20:58:24 - <info> -- Alerts: 0
13/2/2018 -- 20:58:24 - <info> -- cleaning up signature grouping structure... complete
13/2/2018 -- 20:58:24 - <notice> -- Stats for 'mvneta2':  pkts: 1863609, drop: 0 (0.00%), invalid chksum: 0</notice></info></info></info></info></info></notice>

It's like it never even tries to start.

Steve

dales · Feb 15, 2018, 11:58 PM

Does tracing (sh -x /usr/local/etc/rc.d/suricata.sh start) show anything interesting?

stephenw10 · Feb 16, 2018, 1:36 AM

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: sh -x /usr/local/etc/rc.d/suricata.sh start
+ [ ! -f /var/run/suricata_pkg_starting.lck ]
+ rc_start
+ /usr/bin/touch /var/run/suricata_pkg_starting.lck
+ [ ! -f /var/run/suricata_mvneta262562.pid ]
+ /bin/pgrep -fn 'suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml '
+ pid=''
+ [ -z ]
+ /usr/bin/logger -p daemon.info -i -t SuricataStartup 'Suricata START for WAN(62562_mvneta2)...'
+ /usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid
+ sleep 1
+ [ -f /var/run/barnyard2_mvneta262562.pid ]
+ /bin/pgrep -fn 'barnyard2 -r 62562 '
+ pid=''
+ [ ! -z ]
+ [ -f /var/run/suricata_pkg_starting.lck ]
+ /bin/rm /var/run/suricata_pkg_starting.lck

bmeeks · Feb 16, 2018, 3:34 AM

@stephenw10:

It shows nothing at all.

The last entries there are from it stopping when I ran the upgrade:

13/2/2018 -- 20:58:23 - <notice> -- Signal Received.  Stopping engine.
13/2/2018 -- 20:58:23 - <info> -- time elapsed 190197.109s
13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Packets 1863609, bytes 241627739
13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Pcap Total:1863611 Recv:1863611 Drop:0 (0.0%).
13/2/2018 -- 20:58:24 - <info> -- Alerts: 0
13/2/2018 -- 20:58:24 - <info> -- cleaning up signature grouping structure... complete
13/2/2018 -- 20:58:24 - <notice> -- Stats for 'mvneta2':  pkts: 1863609, drop: 0 (0.00%), invalid chksum: 0</notice></info></info></info></info></info></notice>

It's like it never even tries to start.

Steve

Very weird. And it now won't start from the GUI either?

Bill

dales · Feb 16, 2018, 4:26 AM

What happens if you run the command from the trace file manually?

/usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid

[Hoping that it dumps something interesting on stderr….]

bmeeks · Feb 17, 2018, 1:28 AM

I assume you are running DEV firmware since you are upgrading. I've not tested Suricata recently with the new pfSense development releases on the SG-3100. My single test unit is still running the RELEASE firmware.

Getting absolutely nothing in either log (the suricata.log file nor the system log) is strange. Try doing this from the command line –

/usr/local/bin/suricata -V    # should print the version information and exit

See if any console errors get output or if the command runs successfully. We can then go from there.

Bill

stephenw10 · Feb 17, 2018, 8:53 PM

Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

This seems pretty conclusive as to why it's failing to start:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
Shared object "libnss3.so" not found, required by "suricata"

Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

It does appear to be there though so some path missing?

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
/usr/local/lib/nss/libnss3.so

Steve

bmeeks · Feb 19, 2018, 10:34 PM

@stephenw10:

Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

This seems pretty conclusive as to why it's failing to start:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
Shared object "libnss3.so" not found, required by "suricata"
Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

It does appear to be there though so some path missing?
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
/usr/local/lib/nss/libnss3.so
Steve

It might be a required symlink is missing.

<rant on=""></rant>
I hate shared libraries! They cause this kind of stuff. Same as the old "DLL hell" experienced on Windows.

We will probably need to run this one by Renato on the pfSense team to see if he can determine what's going on. I have not been following the latest DEV build. It's a different FreeBSD tree isn't it? If so, the FreeBSD-ports part of the pfSense fork may need an update to match upstream.

Bill

rbgarga · Feb 20, 2018, 12:26 PM

@stephenw10:

Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

This seems pretty conclusive as to why it's failing to start:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
Shared object "libnss3.so" not found, required by "suricata"
Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

It does appear to be there though so some path missing?
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
/usr/local/lib/nss/libnss3.so
Steve

What is the output of the following commands?

pkg upgrade -n

ldd /usr/local/bin/suricata

stephenw10 · Feb 20, 2018, 2:51 PM

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (6 candidates): 100%
Processing candidates (6 candidates): 100%
The following 6 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense]

Number of packages to be upgraded: 6

37 MiB to be downloaded.

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
/usr/local/bin/suricata:
	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000)
	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000)
	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000)
	libnss3.so => not found (0)
	libsmime3.so => not found (0)
	libssl3.so => not found (0)
	libnssutil3.so => not found (0)
	libplds4.so => /usr/local/lib/libplds4.so (0x204dc000)
	libplc4.so => /usr/local/lib/libplc4.so (0x204e6000)
	libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000)
	libc.so.7 => /lib/libc.so.7 (0x20600000)
	libm.so.5 => /lib/libm.so.5 (0x2052e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000)
	libz.so.6 => /lib/libz.so.6 (0x2056a000)
	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)

rbgarga · Feb 20, 2018, 3:04 PM

@stephenw10:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (6 candidates): 100%
Processing candidates (6 candidates): 100%
The following 6 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
	pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense]

Number of packages to be upgraded: 6

37 MiB to be downloaded.

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
/usr/local/bin/suricata:
	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000)
	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000)
	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000)
	libnss3.so => not found (0)
	libsmime3.so => not found (0)
	libssl3.so => not found (0)
	libnssutil3.so => not found (0)
	libplds4.so => /usr/local/lib/libplds4.so (0x204dc000)
	libplc4.so => /usr/local/lib/libplc4.so (0x204e6000)
	libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000)
	libc.so.7 => /lib/libc.so.7 (0x20600000)
	libm.so.5 => /lib/libm.so.5 (0x2052e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000)
	libz.so.6 => /lib/libz.so.6 (0x2056a000)
	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)

There are 2 possibilities in this case. Or nss is not installed or /usr/local/lib/nss is not being tracked by ld. Please run the following commands and let me know the result.

pkg info nss

cat /usr/local/libdata/ldconfig/nss

stephenw10 · Feb 20, 2018, 10:07 PM

Here you go:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss
nss-3.35
Name           : nss
Version        : 3.35
Installed on   : Sun Feb 11 16:06:28 2018 GMT
Origin         : security/nss
Architecture   : FreeBSD:11:armv6
Prefix         : /usr/local
Categories     : security
Licenses       : MPL20
Maintainer     : gecko@FreeBSD.org
WWW            : http://www.mozilla.org/projects/security/pki/nss/
Comment        : Libraries to support development of security-enabled applications
Options        :
	DEBUG          : off
Shared Libs required:
	libplc4.so
	libsqlite3.so.0
	libnspr4.so
	libplds4.so
Shared Libs provided:
	libnss3.so
	libfreeblpriv3.so
	libnssckbi.so
	libnssutil3.so
	libfreebl3.so
	libsmime3.so
	libsoftokn3.so
	libnssdbm3.so
	libssl3.so
Annotations    :
	cpe            : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6
	repo_type      : binary
	repository     : pfSense
Flat size      : 7.76MiB
Description    :
Network Security Services (NSS) is a set of libraries designed to support
cross-platform development of security-enabled server applications.
Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7,
PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security
standards.

WWW: http://www.mozilla.org/projects/security/pki/nss/

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root:  cat /usr/local/libdata/ldconfig/nss
/usr/local/lib/nss

Steve

rbgarga · Feb 21, 2018, 11:18 AM

@stephenw10:

Here you go:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss
nss-3.35
Name           : nss
Version        : 3.35
Installed on   : Sun Feb 11 16:06:28 2018 GMT
Origin         : security/nss
Architecture   : FreeBSD:11:armv6
Prefix         : /usr/local
Categories     : security
Licenses       : MPL20
Maintainer     : gecko@FreeBSD.org
WWW            : http://www.mozilla.org/projects/security/pki/nss/
Comment        : Libraries to support development of security-enabled applications
Options        :
	DEBUG          : off
Shared Libs required:
	libplc4.so
	libsqlite3.so.0
	libnspr4.so
	libplds4.so
Shared Libs provided:
	libnss3.so
	libfreeblpriv3.so
	libnssckbi.so
	libnssutil3.so
	libfreebl3.so
	libsmime3.so
	libsoftokn3.so
	libnssdbm3.so
	libssl3.so
Annotations    :
	cpe            : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6
	repo_type      : binary
	repository     : pfSense
Flat size      : 7.76MiB
Description    :
Network Security Services (NSS) is a set of libraries designed to support
cross-platform development of security-enabled server applications.
Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7,
PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security
standards.

WWW: http://www.mozilla.org/projects/security/pki/nss/

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root:  cat /usr/local/libdata/ldconfig/nss
/usr/local/lib/nss

Steve

ld database is out of date for some reason. Try to run /etc/rc.d/ldconfig start and then ldd /usr/local/bin/suricata again to see if it fixes the issue

stephenw10 · Feb 21, 2018, 4:52 PM

Yup, looks like that resolved it:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE
Soft Float compatibility ldconfig path:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
/usr/local/bin/suricata:
	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000)
	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000)
	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000)
	libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000)
	libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000)
	libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000)
	libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000)
	libplds4.so => /usr/local/lib/libplds4.so (0x204e5000)
	libplc4.so => /usr/local/lib/libplc4.so (0x204ef000)
	libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000)
	libc.so.7 => /lib/libc.so.7 (0x20700000)
	libm.so.5 => /lib/libm.so.5 (0x20868000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000)
	libz.so.6 => /lib/libz.so.6 (0x2089b000)
	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)

Suricata now starts normally. :)

Steve

rbgarga · Feb 21, 2018, 4:54 PM

@stephenw10:

Yup, looks like that resolved it:

[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE
Soft Float compatibility ldconfig path:
[2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
/usr/local/bin/suricata:
	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000)
	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000)
	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000)
	libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000)
	libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000)
	libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000)
	libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000)
	libplds4.so => /usr/local/lib/libplds4.so (0x204e5000)
	libplc4.so => /usr/local/lib/libplc4.so (0x204ef000)
	libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000)
	libc.so.7 => /lib/libc.so.7 (0x20700000)
	libm.so.5 => /lib/libm.so.5 (0x20868000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000)
	libz.so.6 => /lib/libz.so.6 (0x2089b000)
	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)

Suricata now starts normally. :)

Steve

Good. I don't know exactly what caused it but I'm glad we figured it out

stephenw10 · Jul 29, 2018, 3:37 PM

I'm still seeing this and now on both x86 and ARM.
Opened a bug to track it: https://redmine.pfsense.org/issues/8716

bmeeks · Jul 31, 2018, 11:57 AM

@stephenw10
This one will need to be looked at by Renato at Netgate. I'm thinking it's related to a change in the FreeBSD patch files for Suricata. I noticed the binary package maintainer for FreeBSD removed a patch file with "nss3" in its name from the latest 4.0.5 source code on FreeBSD ports, so I am surmising that may be where the libnss3.so library error is coming from.

Bill

stephenw10 · Jul 31, 2018, 1:04 PM

Ah, thanks for that. We'll look into it here then.

Steve