Another upgrade problem



  • Upgrading an SG-2440 from 2.3.3_1 to 2.3.4 if I remember correctly.

    Went through the upgrade process, got the message that it was rebooting, but it didn't come back up for ~ 10 minutes so I power-cycled it.  Same issue so I tried diagnosing the rest of the network and still no go.  Power-cycled again to same problem, now running off backup firewall while I have the laptop plugged into the SG-2440 hoping it will eventually figure out what it needs to do (I read some other reports that the upgrade is just sloooooow.)

    So right now it will issue DHCP addresses, respond to pings, but it will not pass traffic and it will not respond to web requests.  It's been a few hours.

    I'd log into the console and see what's up but I think my console cable is at the data center 8 hours away next to the pfSense boxes that actually help me make money.

    Anyone else seeing this?  Any ideas?

    Here's the text on the upgrade screen in case that helps:

    >>> Updating repositories metadata... 
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    >>> Unlocking package pfSense-kernel-pfSense... done.
    >>> Downloading upgrade packages... 
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    Checking for upgrades (135 candidates): .......... done
    Processing candidates (135 candidates): ... done
    The following 42 package(s) will be affected (of 0 checked):
    
    New packages to be INSTALLED:
    	libevent: 2.1.8 [pfSense]
    
    Installed packages to be UPGRADED:
    	unbound: 1.6.0 -> 1.6.1 [pfSense]
    	strongswan: 5.5.1 -> 5.5.1_1 [pfSense]
    	sqlite3: 3.15.1_1 -> 3.17.0 [pfSense]
    	smartmontools: 6.5_1 -> 6.5_2 [pfSense]
    	python27: 2.7.13_1 -> 2.7.13_3 [pfSense]
    	php-xdebug: 2.4.1_1 -> 2.5.0 [pfSense]
    	pftop: 0.7_7 -> 0.7_8 [pfSense]
    	pfSense-rc: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-kernel-pfSense: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-default-config: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-base: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense: 2.3.3_1 -> 2.3.4 [pfSense]
    	perl5: 5.24.1.r4_1 -> 5.24.1 [pfSense]
    	pcre: 8.39_1 -> 8.40 [pfSense]
    	ntp: 4.2.8p9_1 -> 4.2.8p10_2 [pfSense]
    	nginx: 1.10.2_3,2 -> 1.10.3_1,2 [pfSense]
    	lzo2: 2.09 -> 2.10_1 [pfSense]
    	libssh2: 1.8.0,2 -> 1.8.0,3 [pfSense]
    	libsodium: 1.0.11_1 -> 1.0.12 [pfSense]
    	libnghttp2: 1.18.0 -> 1.21.0 [pfSense]
    	ldns: 1.6.17_5 -> 1.7.0 [pfSense]
    	ipmitool: 1.8.17_1 -> 1.8.18 [pfSense]
    	gmp: 5.1.3_3 -> 6.1.2 [pfSense]
    	glib: 2.46.2_4 -> 2.46.2_5 [pfSense]
    	dhcp6: 20080615.1 -> 20080615.2 [pfSense]
    	curl: 7.53.0 -> 7.54.0 [pfSense]
    	ca_root_nss: 3.28.1 -> 3.30.2 [pfSense]
    	bind-tools: 9.11.0P3 -> 9.11.1 [pfSense]
    
    Installed packages to be REINSTALLED:
    	vstr-1.0.15_1 [pfSense]
    	nettle-3.3 [pfSense]
    	libxml2-2.9.4 [pfSense]
    	libidn-1.33_1 [pfSense]
    	libiconv-1.14_10 [pfSense]
    	libffi-3.2.1 [pfSense]
    	libedit-3.1.20150325_2,1 [pfSense]
    	json-c-0.12.1 [pfSense]
    	idnkit-1.0_6 [pfSense]
    	gettext-runtime-0.19.8.1_1 [pfSense]
    	expat-2.2.0_1 [pfSense]
    	dnsmasq-2.76,1 [pfSense] (options changed)
    	check_reload_status-0.0.7 [pfSense] (direct dependency changed: libevent)
    
    Number of packages to be installed: 1
    Number of packages to be upgraded: 28
    Number of packages to be reinstalled: 13
    
    The process will require 3 MiB more space.
    81 MiB to be downloaded.
    [1/42] Fetching vstr-1.0.15_1.txz: .......... done
    [2/42] Fetching unbound-1.6.1.txz: .......... done
    [3/42] Fetching strongswan-5.5.1_1.txz: .......... done
    [4/42] Fetching sqlite3-3.17.0.txz: .......... done
    [5/42] Fetching smartmontools-6.5_2.txz: .......... done
    [6/42] Fetching python27-2.7.13_3.txz: .......... done
    [7/42] Fetching php-xdebug-2.5.0.txz: .......... done
    [8/42] Fetching pftop-0.7_8.txz: ........ done
    [9/42] Fetching pfSense-rc-2.3.4.txz: . done
    [10/42] Fetching pfSense-kernel-pfSense-2.3.4.txz: .......... done
    [11/42] Fetching pfSense-default-config-2.3.4.txz: . done
    [12/42] Fetching pfSense-base-2.3.4.txz: .......... done
    [13/42] Fetching pfSense-2.3.4.txz: . done
    [14/42] Fetching perl5-5.24.1.txz: .......... done
    [15/42] Fetching pcre-8.40.txz: .......... done
    [16/42] Fetching ntp-4.2.8p10_2.txz: .......... done
    [17/42] Fetching nginx-1.10.3_1,2.txz: .......... done
    [18/42] Fetching nettle-3.3.txz: .......... done
    [19/42] Fetching lzo2-2.10_1.txz: .......... done
    [20/42] Fetching libxml2-2.9.4.txz: .......... done
    [21/42] Fetching libssh2-1.8.0,3.txz: .......... done
    [22/42] Fetching libsodium-1.0.12.txz: .......... done
    [23/42] Fetching libnghttp2-1.21.0.txz: .......... done
    [24/42] Fetching libidn-1.33_1.txz: .......... done
    [25/42] Fetching libiconv-1.14_10.txz: .......... done
    [26/42] Fetching libffi-3.2.1.txz: ..... done
    [27/42] Fetching libedit-3.1.20150325_2,1.txz: .......... done
    [28/42] Fetching ldns-1.7.0.txz: .......... done
    [29/42] Fetching json-c-0.12.1.txz: ..... done
    [30/42] Fetching ipmitool-1.8.18.txz: .......... done
    [31/42] Fetching idnkit-1.0_6.txz: .......... done
    [32/42] Fetching gmp-6.1.2.txz: .......... done
    [33/42] Fetching glib-2.46.2_5.txz: .......... done
    [34/42] Fetching gettext-runtime-0.19.8.1_1.txz: .......... done
    [35/42] Fetching expat-2.2.0_1.txz: .......... done
    [36/42] Fetching dnsmasq-2.76,1.txz: .......... done
    [37/42] Fetching dhcp6-20080615.2.txz: .......... done
    [38/42] Fetching curl-7.54.0.txz: .......... done
    [39/42] Fetching check_reload_status-0.0.7.txz: .... done
    [40/42] Fetching ca_root_nss-3.30.2.txz: .......... done
    [41/42] Fetching bind-tools-9.11.1.txz: .......... done
    [42/42] Fetching libevent-2.1.8.txz: .......... done
    Checking integrity... done (1 conflicting)
      - libevent-2.1.8 [pfSense] conflicts with libevent2-2.0.22_1 [installed] on /usr/local/bin/event_rpcgen.py
    Checking integrity... done (0 conflicting)
    Conflicts with the existing packages have been found.
    One more solver iteration is needed to resolve them.
    The following 43 package(s) will be affected (of 0 checked):
    
    Installed packages to be REMOVED:
    	libevent2-2.0.22_1
    
    New packages to be INSTALLED:
    	libevent: 2.1.8 [pfSense]
    
    Installed packages to be UPGRADED:
    	python27: 2.7.13_1 -> 2.7.13_3 [pfSense]
    	perl5: 5.24.1.r4_1 -> 5.24.1 [pfSense]
    	pcre: 8.39_1 -> 8.40 [pfSense]
    	libnghttp2: 1.18.0 -> 1.21.0 [pfSense]
    	ldns: 1.6.17_5 -> 1.7.0 [pfSense]
    	glib: 2.46.2_4 -> 2.46.2_5 [pfSense]
    	ca_root_nss: 3.28.1 -> 3.30.2 [pfSense]
    	unbound: 1.6.0 -> 1.6.1 [pfSense]
    	libsodium: 1.0.11_1 -> 1.0.12 [pfSense]
    	gmp: 5.1.3_3 -> 6.1.2 [pfSense]
    	curl: 7.53.0 -> 7.54.0 [pfSense]
    	strongswan: 5.5.1 -> 5.5.1_1 [pfSense]
    	sqlite3: 3.15.1_1 -> 3.17.0 [pfSense]
    	lzo2: 2.09 -> 2.10_1 [pfSense]
    	libssh2: 1.8.0,2 -> 1.8.0,3 [pfSense]
    	smartmontools: 6.5_1 -> 6.5_2 [pfSense]
    	php-xdebug: 2.4.1_1 -> 2.5.0 [pfSense]
    	pftop: 0.7_7 -> 0.7_8 [pfSense]
    	pfSense-rc: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	ntp: 4.2.8p9_1 -> 4.2.8p10_2 [pfSense]
    	nginx: 1.10.2_3,2 -> 1.10.3_1,2 [pfSense]
    	ipmitool: 1.8.17_1 -> 1.8.18 [pfSense]
    	dhcp6: 20080615.1 -> 20080615.2 [pfSense]
    	bind-tools: 9.11.0P3 -> 9.11.1 [pfSense]
    	pfSense-kernel-pfSense: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-default-config: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-base: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense: 2.3.3_1 -> 2.3.4 [pfSense]
    
    Installed packages to be REINSTALLED:
    	libffi-3.2.1 [pfSense]
    	gettext-runtime-0.19.8.1_1 [pfSense]
    	libiconv-1.14_10 [pfSense]
    	libxml2-2.9.4 [pfSense]
    	expat-2.2.0_1 [pfSense]
    	vstr-1.0.15_1 [pfSense]
    	nettle-3.3 [pfSense]
    	libidn-1.33_1 [pfSense]
    	libedit-3.1.20150325_2,1 [pfSense]
    	json-c-0.12.1 [pfSense]
    	idnkit-1.0_6 [pfSense]
    	dnsmasq-2.76,1 [pfSense] (options changed)
    	check_reload_status-0.0.7 [pfSense] (direct dependency changed: libevent)
    
    Number of packages to be removed: 1
    Number of packages to be installed: 1
    Number of packages to be upgraded: 28
    Number of packages to be reinstalled: 13
    
    The process will require 1 MiB more space.
    >>> Upgrading pfSense kernel... 
    Checking integrity... done (0 conflicting)
    The following 2 package(s) will be affected (of 0 checked):
    
    Installed packages to be UPGRADED:
    	pfSense-kernel-pfSense: 2.3.3_1 -> 2.3.4 [pfSense-core]
    	pfSense-rc: 2.3.3_1 -> 2.3.4 [pfSense-core]
    
    Number of packages to be upgraded: 2
    [1/2] Upgrading pfSense-rc from 2.3.3_1 to 2.3.4...
    [1/2] Extracting pfSense-rc-2.3.4: .... done
    [2/2] Upgrading pfSense-kernel-pfSense from 2.3.3_1 to 2.3.4...
    [2/2] Extracting pfSense-kernel-pfSense-2.3.4: .......... done
    ===> Keeping a copy of current kernel in /boot/kernel.old
    Upgrade is complete.  Rebooting in 10 seconds.
    >>> Locking package pfSense-kernel-pfSense... done.
    Success
    
    


  • In my case it rebooted fine and proceeded to extract and install what was downloaded prior to the reboot.  That extract and install is where it was really sloooooow.  So you could be in about the same place.  Took an hour or so.  I had the console available though, so could see that it was at least progressing.  Otherwise probably would have rebooted in the middle of that as well.

    So far don't recall anyone identifying cause or solution.



  • Well, shucks.  At least it wasn't my rebooting it that was the problem.

    Someone else linked a blog post that talked about bad components that would fail in a way that kept the firewall running, but prevented a reboot.

    Maybe that got me.



  • Tech support was helpful.  At the end of the day it turns out I rebooted at the wrong time and borked things up.  A fresh install and config upload and all is well.

    Still a bit frustrating, but in the future I'll know to wait a couple of hours to make sure it's running well.



  • @Derek:

    …in the future I'll know to wait a couple of hours to make sure it's running well.

    And that's the frustrating part.