Upgrade cluster to 2.4 failed on second node



  • After the upgrade, node 1 works fine but node 2 failed with:

    /boot/kernel/kernel text=0x16f5540 data=0xb66778+0x4cf968 syms=[0x8+0x1898d0+0x8+0x18a61b]
    Booting...
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1992-2017 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.1-RELEASE-p1 #75 r313908+5a6726eb541(RELENG_2_4_0): Tue Oct  3 12:13:26 CDT 2017
        root@buildbot2.netgate.com:/xbuilder/crossbuild-240/pfSense/tmp/obj/xbuilder/crossbuild-240/pfSense/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
    VT(vga): resolution 640x480
    CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2400.06-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
      TSC: P-state invariant, performance statistics
    real memory  = 10737418240 (10240 MB)
    avail memory = 8212951040 (7832 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <core  coreboot="">WARNING: L1 data cache covers less APIC IDs than a core
    0 < 1
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s)
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #1 Launched!
    Timecounter "TSC-low" frequency 1200029904 Hz quality 1000
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80683490, 0) error 1
    random: entropy device external interface
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c570, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c620, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c6d0, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80683330, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806833e0, 0) error 1
    kbd0 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff81158310, 0) error 19
    random: registering fast source Intel Secure Key RNG
    random: fast provider: "Intel Secure Key RNG"
    nexus0
    vtvga0: <vt vga="" driver="">on motherboard
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <core coreboot="">on motherboard
    acpi0: Power Button (fixed)
    hpet0: <high precision="" event="" timer="">iomem 0xfed03000-0xfed033ff on acpi0
    hpet0: invalid period
    device_attach: hpet0 attach returned 6
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    hpet0: <high precision="" event="" timer="">iomem 0xfed03000-0xfed033ff on acpi0
    hpet0: invalid period
    device_attach: hpet0 attach returned 6
    atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">mem 0xdfe00000-0xdfe1ffff irq 16 at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    pcib2: <acpi pci-pci="" bridge="">mem 0xdfe20000-0xdfe3ffff irq 16 at device 2.0 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    pcib3: <acpi pci-pci="" bridge="">mem 0xdfe40000-0xdfe5ffff irq 20 at device 3.0 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3
    igb0: Using MSIX interrupts with 3 vectors
    igb0: Ethernet address: 00:08:a2:09:74:7e
    igb0: Bound queue 0 to cpu 0
    igb0: Bound queue 1 to cpu 1
    igb0: netmap queues/slots: TX 2/1024, RX 2/1024
    pcib4: <acpi pci-pci="" bridge="">mem 0xdfe60000-0xdfe7ffff irq 23 at device 4.0 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x2000-0x201f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 19 at device 0.0 on pci4
    igb1: Using MSIX interrupts with 3 vectors
    igb1: Ethernet address: 00:08:a2:09:74:7f
    igb1: Bound queue 0 to cpu 2
    igb1: Bound queue 1 to cpu 3
    igb1: netmap queues/slots: TX 2/1024, RX 2/1024
    pci0: <processor>at device 11.0 (no driver attached)
    pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached)
    igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3000-0x301f mem 0xdfea0000-0xdfebffff,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
    igb2: Using MSIX interrupts with 5 vectors
    igb2: Ethernet address: 00:08:a2:09:74:7a
    igb2: Bound queue 0 to cpu 0
    igb2: Bound queue 1 to cpu 1
    igb2: Bound queue 2 to cpu 2
    igb2: Bound queue 3 to cpu 3
    igb2: netmap queues/slots: TX 4/1024, RX 4/1024
    igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3020-0x303f mem 0xdfec0000-0xdfedffff,0xdff28000-0xdff2bfff irq 19 at device 20.1 on pci0
    igb3: Using MSIX interrupts with 5 vectors
    igb3: Ethernet address: 00:08:a2:09:74:7b
    igb3: Bound queue 0 to cpu 0
    igb3: Bound queue 1 to cpu 1
    igb3: Bound queue 2 to cpu 2
    igb3: Bound queue 3 to cpu 3
    igb3: netmap queues/slots: TX 4/1024, RX 4/1024
    igb4: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3040-0x305f mem 0xdfee0000-0xdfefffff,0xdff2c000-0xdff2ffff irq 20 at device 20.2 on pci0
    igb4: Using MSIX interrupts with 5 vectors
    igb4: Ethernet address: 00:08:a2:09:74:7c
    igb4: Bound queue 0 to cpu 0
    igb4: Bound queue 1 to cpu 1
    igb4: Bound queue 2 to cpu 2
    igb4: Bound queue 3 to cpu 3
    igb4: netmap queues/slots: TX 4/1024, RX 4/1024
    igb5: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0x3060-0x307f mem 0xdff00000-0xdff1ffff,0xdff30000-0xdff33fff irq 21 at device 20.3 on pci0
    igb5: Using MSIX interrupts with 5 vectors
    igb5: Ethernet address: 00:08:a2:09:74:7d
    igb5: Bound queue 0 to cpu 0
    igb5: Bound queue 1 to cpu 1
    igb5: Bound queue 2 to cpu 2
    igb5: Bound queue 3 to cpu 3
    igb5: netmap queues/slots: TX 4/1024, RX 4/1024
    ehci0: <intel avoton="" usb="" 2.0="" controller="">mem 0xdff35400-0xdff357ff irq 22 at device 22.0 on pci0
    usbus0: EHCI version 1.0
    usbus0 on ehci0
    usbus0: 480Mbps High Speed USB v2.0
    ahci0: <intel avoton="" ahci="" sata="" controller="">port 0x30c0-0x30c7,0x30e0-0x30e3,0x30c8-0x30cf,0x30e4-0x30e7,0x3080-0x309f mem 0xdff34000-0xdff347ff irq 23 at device 23.0 on pci0
    ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
    ahcich0: <ahci channel="">at channel 0 on ahci0
    ahcich1: <ahci channel="">at channel 1 on ahci0
    ahcich2: <ahci channel="">at channel 2 on ahci0
    ahcich3: <ahci channel="">at channel 3 on ahci0
    ahci1: <intel avoton="" ahci="" sata="" controller="">port 0x30d0-0x30d7,0x30e8-0x30eb,0x30d8-0x30df,0x30ec-0x30ef,0x30a0-0x30bf mem 0xdff34800-0xdff34fff irq 16 at device 24.0 on pci0
    ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
    ahcich4: <ahci channel="">at channel 0 on ahci1
    ahcich5: <ahci channel="">at channel 1 on ahci1
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    orm0: <isa option="" rom="">at iomem 0xc0000-0xc0fff on isa0
    ppc0: cannot reserve I/O port range
    uart0: <16550 or compatible> at port 0x3f8 irq 4 on isa0
    uart1: <16550 or compatible> at port 0x2f8 irq 3 flags 0x10 on isa0
    uart1: console (115200,n,8,1)
    est0: <enhanced speedstep="" frequency="" control="">on cpu0
    est1: <enhanced speedstep="" frequency="" control="">on cpu1
    est2: <enhanced speedstep="" frequency="" control="">on cpu2
    est3: <enhanced speedstep="" frequency="" control="">on cpu3
    Timecounters tick every 1.000 msec
    nvme cam probe device init
    ugen0.1: <intel ehci="" root="" hub="">at usbus0
    uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
    uhub0: 8 ports with 8 removable, self powered
    ugen0.2: <vendor 0x8087="" product="" 0x07db="">at usbus0
    uhub1 on uhub0
    uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr="">on usbus0
    uhub1: 4 ports with 4 removable, self powered
    ugen0.3: <generic ultra="" fast="" media="">at usbus0
    umass0 on uhub1
    umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr="">on usbus0
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    ada0: <kingston sms200s3120g="" 608abbf0="">ATA8-ACS SATA 3.x device
    ada0: Serial Number 50026B726706C498
    ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
    ada0: Command Queueing enabled
    ada0: 114473MB (234441648 512 byte sectors)
    da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
    da0: <generic ultra="" hs-combo="" 1.98="">Removable Direct Access SCSI device
    da0: Serial Number 000000225001
    da0: 40.000MB/s transfers
    da0: 3648MB (7471104 512 byte sectors)
    da0: quirks=0x2 <no_6_byte>Trying to mount root from ufs:/dev/ufsid/580e76410c2ccf71 [rw]...
    WARNING: /: TRIM flag on fs but disk does not support TRIM
    Configuring crash dumps...
    Using /dev/label/swap0 for dump device.
    /dev/ufsid/580e76410c2ccf71: FILE SYSTEM CLEAN; SKIPPING CHECKS
    /dev/ufsid/580e76410c2ccf71: clean, 218856 free (2424 frags, 27054 blocks, 0.4% fragmentation)
    Filesystems are clean, continuing...
    Mounting filesystems...
    random: unblocking device.
    
         ___
     ___/ f \
    / p \___/ Sense
    \___/   \
        \___/
    
    Welcome to pfSense 2.4.0-RC (Patch 1)...
    
    No core dumps found.
    ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
    32-bit compatibility ldconfig path: /usr/lib32
    done.
    pkg: Unable to determine ABI
    pkg: Cannot parse configuration file!
    pkg: Unable to determine ABI
    pkg: Cannot parse configuration file!
    ERROR: It was not possible to identify which pfSense kernel is installed
    Launching the init system...
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0
    
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
    
    Warning: require_once(Net/IPv6.php): failed to open stream: No such file or directory in /etc/inc/config.inc on line 51
    
    Call Stack:
        0.0045     213256   1\. {main}() /etc/rc.bootup:0
        0.0311     214376   2\. require_once('/etc/inc/pkg-utils.inc') /etc/rc.bootup:24
        0.0528     236096   3\. require_once('/etc/inc/service-utils.inc') /etc/inc/pkg-utils.inc:24
        0.0901     237704   4\. require_once('/etc/inc/captiveportal.inc') /etc/inc/service-utils.inc:23
        0.0933     238448   5\. require_once('/etc/inc/config.inc') /etc/inc/captiveportal.inc:27
    
    Fatal error: require_once(): Failed opening required 'Net/IPv6.php' (include_path='.:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear') in /etc/inc/config.inc on line 51
    
    Call Stack:
        0.0045     213256   1\. {main}() /etc/rc.bootup:0
        0.0311     214376   2\. require_once('/etc/inc/pkg-utils.inc') /etc/rc.bootup:24
        0.0528     236096   3\. require_once('/etc/inc/service-utils.inc') /etc/inc/pkg-utils.inc:24
        0.0901     237704   4\. require_once('/etc/inc/captiveportal.inc') /etc/inc/service-utils.inc:23
        0.0933     238448   5\. require_once('/etc/inc/config.inc') /etc/inc/captiveportal.inc:27
    
    Starting CRON... done.
    
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0
    
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
    pkg: Unable to determine ABI
    pkg: Cannot parse configuration file!
    pkg: Unable to determine ABI
    pkg: Cannot parse configuration file!
    ERROR: It was not possible to identify which pfSense kernel is installed
    
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/mbstring.so' - /usr/local/lib/php/20131226/mbstring.so: Shared object has no run-time symbol table in Unknown on line 0
    
    Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
    pfSense 2.4.0-RC (Patch 1) amd64 Tue Oct 03 12:08:12 CDT 2017
    Bootup complete
    
    FreeBSD/amd64 (Amnesiac) (ttyu1)
    
    login: igb0: link state changed to UP
    igb1: link state changed to UP
    igb2: link state changed to UP
    igb3: link state changed to UP
    igb4: link state changed to UP
    igb5: link state changed to UP</no_6_byte></generic></kingston></generic></generic></vendor></vendor></intel></intel></enhanced></enhanced></enhanced></enhanced></isa></isa></pci-isa></ahci></ahci></intel></ahci></ahci></ahci></ahci></intel></intel></intel(r)></intel(r)></intel(r)></intel(r)></processor></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></at></at></high></acpi></acpi></acpi></acpi></high></core></software></vt></version></core ></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    


  • Same for me.. Upgraded my box to 2.4.0 and got loads and loads of PHP warnings (Warning: PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/<blah>.so'- Share object "libssl.so.7" not found, required by "blah" in unkown on line 0

    After this i can login on the console, but that's it..</blah>


  • Rebel Alliance Developer Netgate

    Did you happen to see and/or keep any of the output from the upgrade process before it rebooted?

    Can you look in /conf/ and see if there is an upgrade_log.txt or upgrade_log.latest.txt and post the contents?



  • i tried to update via command line pkg update and saw there was a mayor update en should  use a other command that is recommend so i typed that line.

    it seemed to download 1 file and nothing else.

    so now all plugs are empty. and when i try to update it says

    Shared object "libssl.so.8" not found, required by "pkg"

    WARNING: Current pkg repository has a new OS major version.
            pfSense should be upgraded before doing any other
            operation
    __RC=1

    i cant restore anything because there are already some files updated for the 2.4.

    and in the webgui it also says now " Unable to check for updates"

    everything still works. my addons also still work but i cant see them in the addon list. and i cant upgrade now.


  • Rebel Alliance Developer Netgate

    You should not update with "pkg upgrade", use "pfSense-upgrade" or the GUI to update.



  • @jimp:

    You should not update with "pkg upgrade", use "pfSense-upgrade" or the GUI to update.

    when i try that now i get

    pfSense-upgrade
    Shared object "libssl.so.8" not found, required by "pkg"
    Shared object "libssl.so.8" not found, required by "pkg"
    ERROR: It was not possible to identify which pfSense kernel is installed

    can i still upgrade my installation? or does it need a clean install ?


  • Rebel Alliance Developer Netgate

    No, it is most likely too damaged to correct. Reinstall and use the "recover config.xml" option on the first screen of the installer to get your current config back. You'll be up and running in only a few moments.



  • @jimp:

    No, it is most likely too damaged to correct. Reinstall and use the "recover config.xml" option on the first screen of the installer to get your current config back. You'll be up and running in only a few moments.

    when i did the pkg-static update –f it all happend but i forgot to do  pkg-static upgrade –f  i cant remember if i saw that somewhere. After that was done i did the pfsense-upgrade and it all updated correctly. Didn't have to reinstall.

    only had to do the reinstall packaged from the Gui otherwise i would not access the addons via the Menu.