Suricata on the SG-3100 does not survive a firmware upgrade


  • Netgate Administrator

    Like the subject says really. Because I am always testing fixes I usually follow current snapshots quire closely, updating every few days. Suricata runs just fine on the SG-3100 (thanks bmeeks!) but after updatong to the latest snapshot it does not start. Nor can it be manually started. Nothing useful is logged either. The system log shows:

    Jan 23 22:19:48 	SuricataStartup 	29909 	Suricata START for WAN(62562_mvneta2)... 
    

    The Suricata log shows nothing at all at that point.

    The service does not start.

    Removing the package, whilst keeping the settings, and installing it again restores fuctionality.

    It looks like something either doesn't get created at the upgrade or something is not removed as it is during a normal reinstall and that prevents it starting.

    Steve

    Edit: This does not happen on x86. Or at least I have failed to make it fail in my testing.



  • Since I have an SG-3100 to test with courtesy of Netgate, I will investigate.  I will have to change my appliance from the RELEASE build to the DEV build first, though.  And per your edit this is confirmed to happen only on the ARM builds and not x86 Intel stuff.  Weird!  My first impulse would be to point a finger at pkg, but I'm not sure why it would be messing around with Suricata at all if it is only the firmware being updated.  That does not reinstall packages does it?

    During a CLI-based install or reinstall sequence (and at regular boot up as well), Suricata is started by the system calling the /usr/local/etc/rc.d/suricata.sh script with a "start" argument.  That command assumes that a suitable suricata.yaml configuration file exists for each Suricata-enabled interface.  That shell script is created from code inside the /usr/local/pkg/suricata/suricata.inc file.  Next time you see this problem, drop to the CLI and take a look at that shell script and see what's in it and verify it even exists.  You could also try running the shell script from the command line like so:

    /usr/local/etc/rc.d/suricata.sh start
    

    It should start all the Suricata configured interfaces.  I would be curious what you see and what happens.

    Bill


  • Netgate Administrator

    Sorry for the delay!

    Oddly Suricata started after yesterdays update but failed again today.

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: cat  /usr/local/etc/rc.d/suricata.sh
    #!/bin/sh
    ########
    # This file was automatically generated
    # by the pfSense service handler.
    ######## Start of main suricata.sh
    
    rc_start() {
    
    	### Lock out other start signals until we are done
    	/usr/bin/touch /var/run/suricata_pkg_starting.lck
    
    	## Start suricata on WAN (mvneta2) ##
    	if [ ! -f /var/run/suricata_mvneta262562.pid ]; then
    		pid=`/bin/pgrep -fn "suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml "`
    	else
    		pid=`/bin/pgrep -F /var/run/suricata_mvneta262562.pid`
    	fi
    
    	if [ -z $pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata START for WAN(62562_mvneta2)..."
    		/usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid > /dev/null 2>&1
    	fi
    
    	sleep 1
    
    	if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Barnyard2 STOP for WAN(62562_mvneta2)..."
    		pid=`/bin/pgrep -F /var/run/barnyard2_mvneta262562.pid`
                    /bin/pkill -TERM -F /var/run/barnyard2_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    			/bin/rm /var/run/barnyard2_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "barnyard2 -r 62562 "`
    		if [ ! -z $pid ]; then
    			/bin/pkill -TERM -fn "barnyard2 -r 62562 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
            fi
    
    	### Remove the lock since we have started all interfaces
    	if [ -f /var/run/suricata_pkg_starting.lck ]; then
    		/bin/rm /var/run/suricata_pkg_starting.lck
    	fi
    }
    
    rc_stop() {
    
    	if [ -f /var/run/suricata_mvneta262562.pid ]; then
    		pid=`/bin/pgrep -F /var/run/suricata_mvneta262562.pid`
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WAN(62562_mvneta2)..."
    		/bin/pkill -TERM -F /var/run/suricata_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/suricata_mvneta262562.pid ]; then
    			/bin/rm /var/run/suricata_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml "`
    		if [ ! -z $pid ]; then
    			/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WAN(62562_mvneta2)..."
    			/bin/pkill -TERM -fn "suricata -i mvneta2 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
    	fi
    
    	sleep 1
    
    	if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Barnyard2 STOP for WAN(62562_mvneta2)..."
    		pid=`/bin/pgrep -F /var/run/barnyard2_mvneta262562.pid`
                    /bin/pkill -TERM -F /var/run/barnyard2_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    			/bin/rm /var/run/barnyard2_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "barnyard2 -r 62562 "`
    		if [ ! -z $pid ]; then
    			/bin/pkill -TERM -fn "barnyard2 -r 62562 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
            fi
    }
    
    case $1 in
    	start)
    		if [ ! -f /var/run/suricata_pkg_starting.lck ]; then
    			rc_start
    		else
    			/usr/bin/logger -p daemon.info -i -t SuricataStartup "Ignoring additional START command since Suricata is already starting..."
    		fi
    		;;
    	stop)
    		rc_stop
    		;;
    	restart)
    		rc_stop
    		rc_start
    		;;
    esac
    
    

    Running the start script manually returns clean:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/etc/rc.d/suricata.sh start
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: 
    
    

    Suricata does not start though. System log shows

    Feb 13 22:12:33 	SuricataStartup 	26109 	Suricata START for WAN(62562_mvneta2)... 
    

    /var/run/suricata_mvneta262562.pid does not exist.

    Steve



  • @stephenw10:

    Sorry for the delay!

    Oddly Suricata started after yesterdays update but failed again today.

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: cat  /usr/local/etc/rc.d/suricata.sh
    #!/bin/sh
    ########
    # This file was automatically generated
    # by the pfSense service handler.
    ######## Start of main suricata.sh
    
    rc_start() {
    
    	### Lock out other start signals until we are done
    	/usr/bin/touch /var/run/suricata_pkg_starting.lck
    	
    	## Start suricata on WAN (mvneta2) ##
    	if [ ! -f /var/run/suricata_mvneta262562.pid ]; then
    		pid=`/bin/pgrep -fn "suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml "`
    	else
    		pid=`/bin/pgrep -F /var/run/suricata_mvneta262562.pid`
    	fi
    
    	if [ -z $pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata START for WAN(62562_mvneta2)..."
    		/usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid > /dev/null 2>&1
    	fi
    
    	sleep 1
    	
    	if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Barnyard2 STOP for WAN(62562_mvneta2)..."
    		pid=`/bin/pgrep -F /var/run/barnyard2_mvneta262562.pid`
                    /bin/pkill -TERM -F /var/run/barnyard2_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    			/bin/rm /var/run/barnyard2_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "barnyard2 -r 62562 "`
    		if [ ! -z $pid ]; then
    			/bin/pkill -TERM -fn "barnyard2 -r 62562 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
            fi
    
    	### Remove the lock since we have started all interfaces
    	if [ -f /var/run/suricata_pkg_starting.lck ]; then
    		/bin/rm /var/run/suricata_pkg_starting.lck
    	fi
    }
    
    rc_stop() {
    	
    	if [ -f /var/run/suricata_mvneta262562.pid ]; then
    		pid=`/bin/pgrep -F /var/run/suricata_mvneta262562.pid`
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WAN(62562_mvneta2)..."
    		/bin/pkill -TERM -F /var/run/suricata_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/suricata_mvneta262562.pid ]; then
    			/bin/rm /var/run/suricata_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml "`
    		if [ ! -z $pid ]; then
    			/usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WAN(62562_mvneta2)..."
    			/bin/pkill -TERM -fn "suricata -i mvneta2 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
    	fi
    
    	sleep 1
    	
    	if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    		/usr/bin/logger -p daemon.info -i -t SuricataStartup "Barnyard2 STOP for WAN(62562_mvneta2)..."
    		pid=`/bin/pgrep -F /var/run/barnyard2_mvneta262562.pid`
                    /bin/pkill -TERM -F /var/run/barnyard2_mvneta262562.pid
    		time=0 timeout=30
    		while /bin/kill -TERM $pid 2>/dev/null; do
    			sleep 1
    			time=$((time+1))
    			if [ $time -gt $timeout ]; then
    				break
    			fi
    		done
    		if [ -f /var/run/barnyard2_mvneta262562.pid ]; then
    			/bin/rm /var/run/barnyard2_mvneta262562.pid
    		fi
    	else
    		pid=`/bin/pgrep -fn "barnyard2 -r 62562 "`
    		if [ ! -z $pid ]; then
    			/bin/pkill -TERM -fn "barnyard2 -r 62562 "
    			time=0 timeout=30
    			while /bin/kill -TERM $pid 2>/dev/null; do
    				sleep 1
    				time=$((time+1))
    				if [ $time -gt $timeout ]; then
    					break
    				fi
    			done
    		fi
            fi
    }
    
    case $1 in
    	start)
    		if [ ! -f /var/run/suricata_pkg_starting.lck ]; then
    			rc_start
    		else
    			/usr/bin/logger -p daemon.info -i -t SuricataStartup "Ignoring additional START command since Suricata is already starting..."
    		fi
    		;;
    	stop)
    		rc_stop
    		;;
    	restart)
    		rc_stop
    		rc_start
    		;;
    esac
    
    

    Running the start script manually returns clean:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/etc/rc.d/suricata.sh start
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: 
    
    

    Suricata does not start though. System log shows

    Feb 13 22:12:33 	SuricataStartup 	26109 	Suricata START for WAN(62562_mvneta2)... 
    

    /var/run/suricata_mvneta262562.pid does not exist.

    Steve

    What does the /var/log/suricata/suricata_mvneta262562/suricata.log file show?  Suricata puts most of its info into its own log rather than the system log – exact opposite of the way Snort behaves.

    Bill


  • Netgate Administrator

    It shows nothing at all.

    The last entries there are from it stopping when I ran the upgrade:

    13/2/2018 -- 20:58:23 - <notice> -- Signal Received.  Stopping engine.
    13/2/2018 -- 20:58:23 - <info> -- time elapsed 190197.109s
    13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Packets 1863609, bytes 241627739
    13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Pcap Total:1863611 Recv:1863611 Drop:0 (0.0%).
    13/2/2018 -- 20:58:24 - <info> -- Alerts: 0
    13/2/2018 -- 20:58:24 - <info> -- cleaning up signature grouping structure... complete
    13/2/2018 -- 20:58:24 - <notice> -- Stats for 'mvneta2':  pkts: 1863609, drop: 0 (0.00%), invalid chksum: 0</notice></info></info></info></info></info></notice>
    

    It's like it never even tries to start.

    Steve



  • Does tracing (sh -x /usr/local/etc/rc.d/suricata.sh start) show anything interesting?


  • Netgate Administrator

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: sh -x /usr/local/etc/rc.d/suricata.sh start
    + [ ! -f /var/run/suricata_pkg_starting.lck ]
    + rc_start
    + /usr/bin/touch /var/run/suricata_pkg_starting.lck
    + [ ! -f /var/run/suricata_mvneta262562.pid ]
    + /bin/pgrep -fn 'suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml '
    + pid=''
    + [ -z ]
    + /usr/bin/logger -p daemon.info -i -t SuricataStartup 'Suricata START for WAN(62562_mvneta2)...'
    + /usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid
    + sleep 1
    + [ -f /var/run/barnyard2_mvneta262562.pid ]
    + /bin/pgrep -fn 'barnyard2 -r 62562 '
    + pid=''
    + [ ! -z ]
    + [ -f /var/run/suricata_pkg_starting.lck ]
    + /bin/rm /var/run/suricata_pkg_starting.lck
    
    


  • @stephenw10:

    It shows nothing at all.

    The last entries there are from it stopping when I ran the upgrade:

    13/2/2018 -- 20:58:23 - <notice> -- Signal Received.  Stopping engine.
    13/2/2018 -- 20:58:23 - <info> -- time elapsed 190197.109s
    13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Packets 1863609, bytes 241627739
    13/2/2018 -- 20:58:24 - <info> -- (RX#01-mvneta2) Pcap Total:1863611 Recv:1863611 Drop:0 (0.0%).
    13/2/2018 -- 20:58:24 - <info> -- Alerts: 0
    13/2/2018 -- 20:58:24 - <info> -- cleaning up signature grouping structure... complete
    13/2/2018 -- 20:58:24 - <notice> -- Stats for 'mvneta2':  pkts: 1863609, drop: 0 (0.00%), invalid chksum: 0</notice></info></info></info></info></info></notice>
    

    It's like it never even tries to start.

    Steve

    Very weird.  And it now won't start from the GUI either?

    Bill



  • What happens if you run the command from the trace file manually?

    /usr/local/bin/suricata -i mvneta2 -D -c /usr/local/etc/suricata/suricata_62562_mvneta2/suricata.yaml --pidfile /var/run/suricata_mvneta262562.pid
    
    

    [Hoping that it dumps something interesting on stderr….]



  • I assume you are running DEV firmware since you are upgrading.  I've not tested Suricata recently with the new pfSense development releases on the SG-3100.  My single test unit is still running the RELEASE firmware.

    Getting absolutely nothing in either log (the suricata.log file nor the system log) is strange.  Try doing this from the command line –

    /usr/local/bin/suricata -V    # should print the version information and exit
    

    See if any console errors get output or if the command runs successfully.  We can then go from there.

    Bill


  • Netgate Administrator

    Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

    This seems pretty conclusive as to why it's failing to start:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
    Shared object "libnss3.so" not found, required by "suricata"
    
    

    Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

    It does appear to be there though so some path missing?

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
    /usr/local/lib/nss/libnss3.so
    
    

    Steve



  • @stephenw10:

    Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

    This seems pretty conclusive as to why it's failing to start:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
    Shared object "libnss3.so" not found, required by "suricata"
    
    

    Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

    It does appear to be there though so some path missing?

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
    /usr/local/lib/nss/libnss3.so
    
    

    Steve

    It might be a required symlink is missing.

    <rant on=""></rant>
    I hate shared libraries!  They cause this kind of stuff.  Same as the old "DLL hell" experienced on Windows.

    We will probably need to run this one by Renato on the pfSense team to see if he can determine what's going on.  I have not been following the latest DEV build.  It's a different FreeBSD tree isn't it?  If so, the FreeBSD-ports part of the pfSense fork may need an update to match upstream.

    Bill


  • Administrator

    @stephenw10:

    Indeed I'm running dev and hence upgrading frequently which is when I spotted it.

    This seems pretty conclusive as to why it's failing to start:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /usr/local/bin/suricata -V
    Shared object "libnss3.so" not found, required by "suricata"
    
    

    Interesting then that the upgrade before last it did not fail, presumably the correct library was included that time.

    It does appear to be there though so some path missing?

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: find / -name 'libnss3*'
    /usr/local/lib/nss/libnss3.so
    
    

    Steve

    What is the output of the following commands?

    pkg upgrade -n

    ldd /usr/local/bin/suricata


  • Netgate Administrator

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    Checking for upgrades (6 candidates): 100%
    Processing candidates (6 candidates): 100%
    The following 6 package(s) will be affected (of 0 checked):
    
    Installed packages to be UPGRADED:
    	pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense]
    
    Number of packages to be upgraded: 6
    
    37 MiB to be downloaded.
    
    
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
    /usr/local/bin/suricata:
    	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
    	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
    	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
    	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
    	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
    	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
    	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
    	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000)
    	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000)
    	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000)
    	libnss3.so => not found (0)
    	libsmime3.so => not found (0)
    	libssl3.so => not found (0)
    	libnssutil3.so => not found (0)
    	libplds4.so => /usr/local/lib/libplds4.so (0x204dc000)
    	libplc4.so => /usr/local/lib/libplc4.so (0x204e6000)
    	libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000)
    	libc.so.7 => /lib/libc.so.7 (0x20600000)
    	libm.so.5 => /lib/libm.so.5 (0x2052e000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000)
    	libz.so.6 => /lib/libz.so.6 (0x2056a000)
    	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)
    
    

  • Administrator

    @stephenw10:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg upgrade -n
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    Checking for upgrades (6 candidates): 100%
    Processing candidates (6 candidates): 100%
    The following 6 package(s) will be affected (of 0 checked):
    
    Installed packages to be UPGRADED:
    	pfSense-u-boot-sg3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-rc: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-kernel-pfSense-SG-3100: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-default-config-serial: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense-base: 2.4.3.a.20180213.0339 -> 2.4.3.a.20180219.1328 [pfSense-core]
    	pfSense: 2.4.3.a.20180212.0807 -> 2.4.3.a.20180219.1329 [pfSense]
    
    Number of packages to be upgraded: 6
    
    37 MiB to be downloaded.
    
    
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
    /usr/local/bin/suricata:
    	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
    	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
    	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
    	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
    	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
    	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
    	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
    	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x20406000)
    	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20429000)
    	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204b9000)
    	libnss3.so => not found (0)
    	libsmime3.so => not found (0)
    	libssl3.so => not found (0)
    	libnssutil3.so => not found (0)
    	libplds4.so => /usr/local/lib/libplds4.so (0x204dc000)
    	libplc4.so => /usr/local/lib/libplc4.so (0x204e6000)
    	libnspr4.so => /usr/local/lib/libnspr4.so (0x204f1000)
    	libc.so.7 => /lib/libc.so.7 (0x20600000)
    	libm.so.5 => /lib/libm.so.5 (0x2052e000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2055b000)
    	libz.so.6 => /lib/libz.so.6 (0x2056a000)
    	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x20768000)
    
    

    There are 2 possibilities in this case. Or nss is not installed or /usr/local/lib/nss is not being tracked by ld. Please run the following commands and let me know the result.

    pkg info nss

    cat /usr/local/libdata/ldconfig/nss


  • Netgate Administrator

    Here you go:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss
    nss-3.35
    Name           : nss
    Version        : 3.35
    Installed on   : Sun Feb 11 16:06:28 2018 GMT
    Origin         : security/nss
    Architecture   : FreeBSD:11:armv6
    Prefix         : /usr/local
    Categories     : security
    Licenses       : MPL20
    Maintainer     : gecko@FreeBSD.org
    WWW            : http://www.mozilla.org/projects/security/pki/nss/
    Comment        : Libraries to support development of security-enabled applications
    Options        :
    	DEBUG          : off
    Shared Libs required:
    	libplc4.so
    	libsqlite3.so.0
    	libnspr4.so
    	libplds4.so
    Shared Libs provided:
    	libnss3.so
    	libfreeblpriv3.so
    	libnssckbi.so
    	libnssutil3.so
    	libfreebl3.so
    	libsmime3.so
    	libsoftokn3.so
    	libnssdbm3.so
    	libssl3.so
    Annotations    :
    	cpe            : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6
    	repo_type      : binary
    	repository     : pfSense
    Flat size      : 7.76MiB
    Description    :
    Network Security Services (NSS) is a set of libraries designed to support
    cross-platform development of security-enabled server applications.
    Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7,
    PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security
    standards.
    
    WWW: http://www.mozilla.org/projects/security/pki/nss/
    
    
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root:  cat /usr/local/libdata/ldconfig/nss
    /usr/local/lib/nss
    
    

    Steve


  • Administrator

    @stephenw10:

    Here you go:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: pkg info nss
    nss-3.35
    Name           : nss
    Version        : 3.35
    Installed on   : Sun Feb 11 16:06:28 2018 GMT
    Origin         : security/nss
    Architecture   : FreeBSD:11:armv6
    Prefix         : /usr/local
    Categories     : security
    Licenses       : MPL20
    Maintainer     : gecko@FreeBSD.org
    WWW            : http://www.mozilla.org/projects/security/pki/nss/
    Comment        : Libraries to support development of security-enabled applications
    Options        :
    	DEBUG          : off
    Shared Libs required:
    	libplc4.so
    	libsqlite3.so.0
    	libnspr4.so
    	libplds4.so
    Shared Libs provided:
    	libnss3.so
    	libfreeblpriv3.so
    	libnssckbi.so
    	libnssutil3.so
    	libfreebl3.so
    	libsmime3.so
    	libsoftokn3.so
    	libnssdbm3.so
    	libssl3.so
    Annotations    :
    	cpe            : cpe:2.3:a:mozilla:network_security_services:3.35:::::freebsd11:armv6
    	repo_type      : binary
    	repository     : pfSense
    Flat size      : 7.76MiB
    Description    :
    Network Security Services (NSS) is a set of libraries designed to support
    cross-platform development of security-enabled server applications.
    Applications built with NSS can support SSL v2 and v3, TLS, PKCS #5, PKCS #7,
    PKCS #11, PKCS #12, S/MIME, X.509 v3 certificates, and other security
    standards.
    
    WWW: http://www.mozilla.org/projects/security/pki/nss/
    
    
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root:  cat /usr/local/libdata/ldconfig/nss
    /usr/local/lib/nss
    
    

    Steve

    ld database is out of date for some reason. Try to run /etc/rc.d/ldconfig start and then ldd /usr/local/bin/suricata again to see if it fixes the issue


  • Netgate Administrator

    Yup, looks like that resolved it:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start
    ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE
    Soft Float compatibility ldconfig path:
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
    /usr/local/bin/suricata:
    	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
    	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
    	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
    	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
    	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
    	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
    	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
    	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000)
    	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000)
    	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000)
    	libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000)
    	libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000)
    	libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000)
    	libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000)
    	libplds4.so => /usr/local/lib/libplds4.so (0x204e5000)
    	libplc4.so => /usr/local/lib/libplc4.so (0x204ef000)
    	libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000)
    	libc.so.7 => /lib/libc.so.7 (0x20700000)
    	libm.so.5 => /lib/libm.so.5 (0x20868000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000)
    	libz.so.6 => /lib/libz.so.6 (0x2089b000)
    	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)
    
    

    Suricata now starts normally.  :)

    Steve


  • Administrator

    @stephenw10:

    Yup, looks like that resolved it:

    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: /etc/rc.d/ldconfig start
    ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/nss /usr/local/lib/perl5/5.24/mach/CORE
    Soft Float compatibility ldconfig path:
    [2.4.3-DEVELOPMENT][admin@3100.stevew.lan]/root: ldd /usr/local/bin/suricata
    /usr/local/bin/suricata:
    	libhiredis.so.0.13 => /usr/local/lib/libhiredis.so.0.13 (0x20252000)
    	libGeoIP.so.1 => /usr/local/lib/libGeoIP.so.1 (0x20266000)
    	libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x202a4000)
    	libmagic.so.4 => /usr/lib/libmagic.so.4 (0x20329000)
    	libpcap.so.1 => /usr/local/lib/libpcap.so.1 (0x20351000)
    	libnet.so.1 => /usr/local/lib/libnet.so.1 (0x203a7000)
    	libjansson.so.4 => /usr/local/lib/libjansson.so.4 (0x203c3000)
    	libthr.so.3 => /lib/libthr.so.3 (0x203d8000)
    	libyaml-0.so.2 => /usr/local/lib/libyaml-0.so.2 (0x2040f000)
    	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x20432000)
    	libhtp.so.2 => /usr/local/lib/libhtp.so.2 (0x204c2000)
    	libnss3.so => /usr/local/lib/nss/libnss3.so (0x20500000)
    	libsmime3.so => /usr/local/lib/nss/libsmime3.so (0x20617000)
    	libssl3.so => /usr/local/lib/nss/libssl3.so (0x20642000)
    	libnssutil3.so => /usr/local/lib/nss/libnssutil3.so (0x2068c000)
    	libplds4.so => /usr/local/lib/libplds4.so (0x204e5000)
    	libplc4.so => /usr/local/lib/libplc4.so (0x204ef000)
    	libnspr4.so => /usr/local/lib/libnspr4.so (0x206b9000)
    	libc.so.7 => /lib/libc.so.7 (0x20700000)
    	libm.so.5 => /lib/libm.so.5 (0x20868000)
    	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2088c000)
    	libz.so.6 => /lib/libz.so.6 (0x2089b000)
    	libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x208ba000)
    
    

    Suricata now starts normally.  :)

    Steve

    Good. I don't know exactly what caused it but I'm glad we figured it out


  • Netgate Administrator

    I'm still seeing this and now on both x86 and ARM.
    Opened a bug to track it: https://redmine.pfsense.org/issues/8716



  • @stephenw10
    This one will need to be looked at by Renato at Netgate. I'm thinking it's related to a change in the FreeBSD patch files for Suricata. I noticed the binary package maintainer for FreeBSD removed a patch file with "nss3" in its name from the latest 4.0.5 source code on FreeBSD ports, so I am surmising that may be where the libnss3.so library error is coming from.

    Bill


  • Netgate Administrator

    Ah, thanks for that. We'll look into it here then.

    Steve