Upgrade cluster to 2.4 failed on second node

-DesT- 0 · Oct 4, 2017, 11:17 AM

After the upgrade, node 1 works fine but node 2 failed with:

/boot/kernel/kernel text=0x16f5540 data=0xb66778+0x4cf968 syms=[0x8+0x1898d0+0x8+0x18a61b]
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p1 #75 r313908+5a6726eb541(RELENG_2_4_0): Tue Oct  3 12:13:26 CDT 2017
    root@buildbot2.netgate.com:/xbuilder/crossbuild-240/pfSense/tmp/obj/xbuilder/crossbuild-240/pfSense/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2400.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 10737418240 (10240 MB)
avail memory = 8212951040 (7832 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <core  coreboot="">WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
ioapic0 <version 2.0="">irqs 0-23 on motherboard
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1200029904 Hz quality 1000
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80683490, 0) error 1
random: entropy device external interface
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c570, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c620, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c6d0, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80683330, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806833e0, 0) error 1
kbd0 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0: <vt vga="" driver="">on motherboard
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <core coreboot="">on motherboard
acpi0: Power Button (fixed)
hpet0: <high precision="" event="" timer="">iomem 0xfed03000-0xfed033ff on acpi0
hpet0: invalid period
device_attach: hpet0 attach returned 6
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
hpet0: <high precision="" event="" timer="">iomem 0xfed03000-0xfed033ff on acpi0
hpet0: invalid period
device_attach: hpet0 attach returned 6
atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">mem 0xdfe00000-0xdfe1ffff irq 16 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
pcib2: <acpi pci-pci="" bridge="">mem 0xdfe20000-0xdfe3ffff irq 16 at device 2.0 on pci0
pci2: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">mem 0xdfe40000-0xdfe5ffff irq 20 at device 3.0 on pci0
pci3: <acpi pci="" bus="">on pcib3
igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 00:08:a2:09:74:7e
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib4: <acpi pci-pci="" bridge="">mem 0xdfe60000-0xdfe7ffff irq 23 at device 4.0 on pci0
pci4: <acpi pci="" bus="">on pcib4
igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x2000-0x201f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 19 at device 0.0 on pci4
igb1: Using MSIX interrupts with 3 vectors
igb1: Ethernet address: 00:08:a2:09:74:7f
igb1: Bound queue 0 to cpu 2
igb1: Bound queue 1 to cpu 3
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
pci0: <processor>at device 11.0 (no driver attached)
pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached)
igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3000-0x301f mem 0xdfea0000-0xdfebffff,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:08:a2:09:74:7a
igb2: Bound queue 0 to cpu 0
igb2: Bound queue 1 to cpu 1
igb2: Bound queue 2 to cpu 2
igb2: Bound queue 3 to cpu 3
igb2: netmap queues/slots: TX 4/1024, RX 4/1024
igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3020-0x303f mem 0xdfec0000-0xdfedffff,0xdff28000-0xdff2bfff irq 19 at device 20.1 on pci0
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 00:08:a2:09:74:7b
igb3: Bound queue 0 to cpu 0
igb3: Bound queue 1 to cpu 1
igb3: Bound queue 2 to cpu 2
igb3: Bound queue 3 to cpu 3
igb3: netmap queues/slots: TX 4/1024, RX 4/1024
igb4: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3040-0x305f mem 0xdfee0000-0xdfefffff,0xdff2c000-0xdff2ffff irq 20 at device 20.2 on pci0
igb4: Using MSIX interrupts with 5 vectors
igb4: Ethernet address: 00:08:a2:09:74:7c
igb4: Bound queue 0 to cpu 0
igb4: Bound queue 1 to cpu 1
igb4: Bound queue 2 to cpu 2
igb4: Bound queue 3 to cpu 3
igb4: netmap queues/slots: TX 4/1024, RX 4/1024
igb5: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3060-0x307f mem 0xdff00000-0xdff1ffff,0xdff30000-0xdff33fff irq 21 at device 20.3 on pci0
igb5: Using MSIX interrupts with 5 vectors
igb5: Ethernet address: 00:08:a2:09:74:7d
igb5: Bound queue 0 to cpu 0
igb5: Bound queue 1 to cpu 1
igb5: Bound queue 2 to cpu 2
igb5: Bound queue 3 to cpu 3
igb5: netmap queues/slots: TX 4/1024, RX 4/1024
ehci0: <intel avoton="" usb="" 2.0="" controller="">mem 0xdff35400-0xdff357ff irq 22 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
ahci0: <intel avoton="" ahci="" sata="" controller="">port 0x30c0-0x30c7,0x30e0-0x30e3,0x30c8-0x30cf,0x30e4-0x30e7,0x3080-0x309f mem 0xdff34000-0xdff347ff irq 23 at device 23.0 on pci0
ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich0: <ahci channel="">at channel 0 on ahci0
ahcich1: <ahci channel="">at channel 1 on ahci0
ahcich2: <ahci channel="">at channel 2 on ahci0
ahcich3: <ahci channel="">at channel 3 on ahci0
ahci1: <intel avoton="" ahci="" sata="" controller="">port 0x30d0-0x30d7,0x30e8-0x30eb,0x30d8-0x30df,0x30ec-0x30ef,0x30a0-0x30bf mem 0xdff34800-0xdff34fff irq 16 at device 24.0 on pci0
ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich4: <ahci channel="">at channel 0 on ahci1
ahcich5: <ahci channel="">at channel 1 on ahci1
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
orm0: <isa option="" rom="">at iomem 0xc0000-0xc0fff on isa0
ppc0: cannot reserve I/O port range
uart0: <16550 or compatible> at port 0x3f8 irq 4 on isa0
uart1: <16550 or compatible> at port 0x2f8 irq 3 flags 0x10 on isa0
uart1: console (115200,n,8,1)
est0: <enhanced speedstep="" frequency="" control="">on cpu0
est1: <enhanced speedstep="" frequency="" control="">on cpu1
est2: <enhanced speedstep="" frequency="" control="">on cpu2
est3: <enhanced speedstep="" frequency="" control="">on cpu3
Timecounters tick every 1.000 msec
nvme cam probe device init
ugen0.1: <intel ehci="" root="" hub="">at usbus0
uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
uhub0: 8 ports with 8 removable, self powered
ugen0.2: <vendor 0x8087="" product="" 0x07db="">at usbus0
uhub1 on uhub0
uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr="">on usbus0
uhub1: 4 ports with 4 removable, self powered
ugen0.3: <generic ultra="" fast="" media="">at usbus0
umass0 on uhub1
umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr="">on usbus0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <kingston sms200s3120g="" 608abbf0="">ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B726706C498
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors)
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <generic ultra="" hs-combo="" 1.98="">Removable Direct Access SCSI device
da0: Serial Number 000000225001
da0: 40.000MB/s transfers
da0: 3648MB (7471104 512 byte sectors)
da0: quirks=0x2 <no_6_byte>Trying to mount root from ufs:/dev/ufsid/580e76410c2ccf71 [rw]...
WARNING: /: TRIM flag on fs but disk does not support TRIM
Configuring crash dumps...
Using /dev/label/swap0 for dump device.
/dev/ufsid/580e76410c2ccf71: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufsid/580e76410c2ccf71: clean, 218856 free (2424 frags, 27054 blocks, 0.4% fragmentation)
Filesystems are clean, continuing...
Mounting filesystems...
random: unblocking device.

     ___
 ___/ f \
/ p \___/ Sense
\___/   \
    \___/

Welcome to pfSense 2.4.0-RC (Patch 1)...

No core dumps found.
...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
done.
pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
ERROR: It was not possible to identify which pfSense kernel is installed
Launching the init system...
Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0

Warning: require_once(Net/IPv6.php): failed to open stream: No such file or directory in /etc/inc/config.inc on line 51

Call Stack:
    0.0045     213256   1\. {main}() /etc/rc.bootup:0
    0.0311     214376   2\. require_once('/etc/inc/pkg-utils.inc') /etc/rc.bootup:24
    0.0528     236096   3\. require_once('/etc/inc/service-utils.inc') /etc/inc/pkg-utils.inc:24
    0.0901     237704   4\. require_once('/etc/inc/captiveportal.inc') /etc/inc/service-utils.inc:23
    0.0933     238448   5\. require_once('/etc/inc/config.inc') /etc/inc/captiveportal.inc:27

Fatal error: require_once(): Failed opening required 'Net/IPv6.php' (include_path='.:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear') in /etc/inc/config.inc on line 51

Call Stack:
    0.0045     213256   1\. {main}() /etc/rc.bootup:0
    0.0311     214376   2\. require_once('/etc/inc/pkg-utils.inc') /etc/rc.bootup:24
    0.0528     236096   3\. require_once('/etc/inc/service-utils.inc') /etc/inc/pkg-utils.inc:24
    0.0901     237704   4\. require_once('/etc/inc/captiveportal.inc') /etc/inc/service-utils.inc:23
    0.0933     238448   5\. require_once('/etc/inc/config.inc') /etc/inc/captiveportal.inc:27

Starting CRON... done.

Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
ERROR: It was not possible to identify which pfSense kernel is installed

Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
pfSense 2.4.0-RC (Patch 1) amd64 Tue Oct 03 12:08:12 CDT 2017
Bootup complete

FreeBSD/amd64 (Amnesiac) (ttyu1)

login: igb0: link state changed to UP
igb1: link state changed to UP
igb2: link state changed to UP
igb3: link state changed to UP
igb4: link state changed to UP
igb5: link state changed to UP</no_6_byte></generic></kingston></generic></generic></vendor></vendor></intel></intel></enhanced></enhanced></enhanced></enhanced></isa></isa></pci-isa></ahci></ahci></intel></ahci></ahci></ahci></ahci></intel></intel></intel(r)></intel(r)></intel(r)></intel(r)></processor></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></at></at></high></acpi></acpi></acpi></acpi></high></core></software></vt></version></core ></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>

redguy · Oct 12, 2017, 7:17 PM

Same for me.. Upgraded my box to 2.4.0 and got loads and loads of PHP warnings (Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/<blah>.so'- Share object "libssl.so.7" not found, required by "blah" in unkown on line 0

After this i can login on the console, but that's it..</blah>

jimp · Oct 12, 2017, 8:05 PM

Did you happen to see and/or keep any of the output from the upgrade process before it rebooted?

Can you look in /conf/ and see if there is an upgrade_log.txt or upgrade_log.latest.txt and post the contents?

musicwizard · Oct 12, 2017, 8:22 PM

i tried to update via command line pkg update and saw there was a mayor update en should use a other command that is recommend so i typed that line.

it seemed to download 1 file and nothing else.

so now all plugs are empty. and when i try to update it says

Shared object "libssl.so.8" not found, required by "pkg"

WARNING: Current pkg repository has a new OS major version.
pfSense should be upgraded before doing any other
operation
__RC=1

i cant restore anything because there are already some files updated for the 2.4.

and in the webgui it also says now " Unable to check for updates"

everything still works. my addons also still work but i cant see them in the addon list. and i cant upgrade now.

jimp · Oct 12, 2017, 8:28 PM

You should not update with "pkg upgrade", use "pfSense-upgrade" or the GUI to update.

musicwizard · Oct 12, 2017, 8:34 PM

@jimp:

You should not update with "pkg upgrade", use "pfSense-upgrade" or the GUI to update.

when i try that now i get

pfSense-upgrade
Shared object "libssl.so.8" not found, required by "pkg"
Shared object "libssl.so.8" not found, required by "pkg"
ERROR: It was not possible to identify which pfSense kernel is installed

can i still upgrade my installation? or does it need a clean install ?

jimp · Oct 12, 2017, 8:36 PM

No, it is most likely too damaged to correct. Reinstall and use the "recover config.xml" option on the first screen of the installer to get your current config back. You'll be up and running in only a few moments.

musicwizard · Oct 12, 2017, 9:45 PM

@jimp:

No, it is most likely too damaged to correct. Reinstall and use the "recover config.xml" option on the first screen of the installer to get your current config back. You'll be up and running in only a few moments.

when i did the pkg-static update –f it all happend but i forgot to do pkg-static upgrade –f i cant remember if i saw that somewhere. After that was done i did the pfsense-upgrade and it all updated correctly. Didn't have to reinstall.

only had to do the reinstall packaged from the Gui otherwise i would not access the addons via the Menu.