Error going from 2.3 to 2.3.1_1



  • Hi all,

    Having an issue updating one of my 5 boxes to 2.3.1_1, on the dashboard it says: 'unable to check for updates', the actual System Updages page gives me: 'Unable to retrieve system versions.'

    SSH in and use option 13, returns this:
    PHP:  syntax error, unexpected '(' in /usr/local/etc/php/extensions.ini on line 2
    /usr/local/sbin/pkg: Shared object has no run-time symbol table
    /usr/local/sbin/pkg: Shared object has no run-time symbol table
    ERROR: It was not possible to identify which pfSense kernel is installed
    PHP:  syntax error, unexpected '(' in /usr/local/etc/php/extensions.ini on line 2
    PHP:  syntax error, unexpected '(' in /usr/local/etc/php/extensions.ini on line 2

    The update settings page is even stranger, instead of Stable and Dev, it shows 'FreeBSD: {enable', and 'FreeBSD: {enabled:no'

    Any ideas?



  • There's something screwy in your /usr/local/etc/php/extensions.ini

    What are the contents of that file?



  • ok, file was full of garbage, copied it over from a working unit, now I get the same on webconfigurator, and the below on SSH using 13.

    /usr/local/sbin/pkg: Shared object has no run-time symbol table
    /usr/local/sbin/pkg: Shared object has no run-time symbol table
    ERROR: It was not possible to identify which pfSense kernel is installed

    /usr/local/sbin/pkg is not readable on this system, nor a working one.

    • M


  • pkg is a compiled binary, not something that would be legible. But that system that isn't working clearly has a variety of issues. Sounds like it corrupted a number of files (which might be indicative of a hardware problem). I'd get a config backup from that system, reinstall it clean, and restore the backup. And be wary of the system given that level of weird file corruption.



  • You already identified yourself the real problem:
    @Marc:

    ok, file was full of garbage, copied it over ….

    I propose that you run
    fsck
    and you will see what up really : file system damaged …. your disk is probably dying (they so do all the time).

    @Marc:

    /usr/local/sbin/pkg is not readable on this system, nor a working one.

    What do you mean by not readable ?
    'pkg' is a program and as such, before it exectues in memory, it has to be read into memory first.
    It's just a file, and is readable.

    Proof :
    [2.3.1-RELEASE][admin@pfsense.brit-hotel-fumel.net]/etc/inc: hexdump /usr/local/sbin/pkg | more
    0000000 457f 464c 0102 0901 0000 0000 0000 0000
    0000010 0002 003e 0001 0000 6340 0040 0000 0000
    0000020 0040 0000 0000 0000 a320 0002 0000 0000
    0000030 0000 0000 0040 0038 0008 0040 001c 001b
    0000040 0006 0000 0005 0000 0040 0000 0000 0000
    0000050 0040 0040 0000 0000 0040 0040 0000 0000
    0000060 01c0 0000 0000 0000 01c0 0000 0000 0000
    0000070 0008 0000 0000 0000 0003 0000 0004 0000
    0000080 0200 0000 0000 0000 0200 0040 0000 0000
    0000090 0200 0040 0000 0000 0015 0000 0000 0000
    00000a0 0015 0000 0000 0000 0001 0000 0000 0000
    00000b0 0001 0000 0005 0000 0000 0000 0000 0000
    00000c0 0000 0040 0000 0000 0000 0040 0000 0000
    00000d0 8a5c 0002 0000 0000 8a5c 0002 0000 0000
    00000e0 0000 0020 0000 0000 0001 0000 0006 0000
    00000f0 9000 0002 0000 0000 9000 0062 0000 0000
    0000100 9000 0062 0000 0000 1068 0000 0000 0000
    0000110 2758 0000 0000 0000 0000 0020 0000 0000
    0000120 0002 0000 0006 0000 9028 0002 0000 0000
    0000130 9028 0062 0000 0000 9028 0062 0000 0000
    0000140 0270 0000 0000 0000 0270 0000 0000 0000
    0000150 0008 0000 0000 0000 0004 0000 0004 0000
    0000160 0218 0000 0000 0000 0218 0040 0000 0000
    …...

    The message
    /usr/local/sbin/pkg: Shared object has no run-time symbol table
    means : file f*cked up.

    So : check your drive.



  • I'm surprisingly happy with that, it's the only 32bit machine in the fleet.  3 are actual pfSense SG appliances, and my home unit is a VM running on my vsphere host.  Might be that final kick to get rid of that old machine.



  • I've had a couple of issues trying to upgrade my pfSense installation from 2.3 to 2.3.1_1 on Hyper V that sound similar to this issue.

    At first, my original installation would simply break when the update was applied in a completely repeatable manner (restore the VM from backup, rinse, repeat). Basically, I would update using the standard update feature in the UI, which would download, install, then reboot the system. At that point it'd install a couple more things, then continue to boot up the point where it said "Configuring WAN interface…" and immediately reboot. At that point it would then display a bunch of things after the boot prompt and "BTX halted" (as attached).

    At that point, I thought that it'd be a good idea to simply start fresh, so I made a config backup in the web interface and created a whole new VM and installed 2.3.1 on it from scratch. What I found was that I could successfully update it to 2.3.1_1 after making the minimal configuration required to get it onto the Internet, but if I tried to restore my config.xml file prior to upgrading, then I'd have issues during the upgrade process similar to the above (with the same "crash" at "Configuring WAN interface…" and reboot after installation), although it would at least attempt to boot the system, but with various issues (which seemed to change each time I tried it). For example, here is an excerpt of the console I captured while booting one of these attempted upgrades:

    KDB: current backend: ddb
    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.3-RELEASE-p3 #2 1988fec(RELENG_2_3_1): Wed May 25 14:14:46 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-231/tmp/obj/builder/pfsense-231/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (1664.84-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x306a9  Family=0x6  Model=0x3a  Stepping=9
      Features=0x1f83fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt>Features2=0xfe982203 <sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,popcnt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>Structured Extended Features=0x200 <erms>XSAVE Features=0x1 <xsaveopt>Hypervisor: Origin = "Microsoft Hv"
    real memory  = 536870912 (512 MB)
    avail memory = 478343168 (456 MB)
    
    ...
    
    SMP: AP CPU #1 Launched!
    Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
    Event timer "HyperV" frequency 10000000 Hz quality 1000
    storvsc0 on vmbus0
    storvsc1 on vmbus0
    da0 at blkvsc0 bus 0 scbus1 target 0 lun 0
    hyperv-utils0 on vmbus0
    hyperv-utils0: Hyper-V Service attaching: Hyper-V Heartbeat Service
    
    da0: hyperv-utils1 on vmbus0
    hyperv-utils1: Hyper-V Service attaching: Hyper-V KVP Service
    
     <msft virtual="" disk="" 1.0="">Fixed Direct Access SPC-2 SCSI device
    da0: 300.000MB/s transfershyperv-utils2 on vmbus0
    hyperv-utils2: Hyper-V Service attaching: Hyper-V Shutdown Service
    
    hyperv-utils3 on vmbus0
    hyperv-utils3: Hyper-V Service attaching: Hyper-V Time Synch Service
    
    da0: Command Queueing enabled
    hn0: <synthetic network="" interface=""> on vmbus0
    da0: 5120MB (10485760 512 byte sectors)
    hn0: unknown status 1073872902 received
    hn0: hv send offload request succeeded
    hn0: Using defaults for TSO: 65518/35/2048
    hn0: Ethernet address: 00:15:5d:00:04:17
    hn1: <synthetic network="" interface=""> on vmbus0
    hn1: unknown status 1073872902 received
    hn1: hv send offload request succeeded
    hn1: Using defaults for TSO: 65518/35/2048
    hn1: Ethernet address: 00:15:5d:00:04:18
    hn2: <synthetic network="" interface=""> on vmbus0
    hn2: unknown status 1073872902 received
    hn2: hv send offload request succeeded
    hn2: Using defaults for TSO: 65518/35/2048
    hn2: Ethernet address: 00:15:5d:00:04:19
    hn3: <synthetic network="" interface=""> on vmbus0
    hn3: unknown status 1073872902 received
    hn3: hv send offload request succeeded
    hn3: Using defaults for TSO: 65518/35/2048
    hn3: Ethernet address: 00:15:5d:00:04:1a
    Trying to mount root from ufs:/dev/ufsid/574a68fd984d1bc6 [rw]...
    May 29 15:48:35 init: login_getclass: unknown class 'daemon'
    
    May 29 15:48:35 init: login_getclass: no default/fallback class 'default'
    
    Configuring crash dumps...
    Filesystems are clean, continuing...
    Mounting filesystems...
    rm: /conf: Read-only file system
    ln: /conf/conf: Read-only file system
    
         ___
     ___/ f \
    / p \___/ Sense
    \___/   \
        \___/
    
    Welcome to  2.3.1-RELEASE on the 'pfSense' platform...
    
    rm: /COPYRIGHT: Read-only file system
    rm: /bin/cat: Read-only file system
    
    ...
    
    /etc/rc: cannot create /dev/null: Operation not supported
    Creating symlinks...ln: /tmp/tmp: Read-only file system
    rm: /tmp/config.cache: Read-only file system
    rm: /tmp/config.lock: Read-only file system
    rm: /tmp/mnt/cf: Read-only file system
    rm: /tmp/mnt: Read-only file system
    rm: /tmp/php_errors.txt: Read-only file system
    ..cp: /dev/null: No such file or directory
    ./etc/rc: cannot create /dev/null: Operation not supported
    done.
    /etc/rc: /usr/local/sbin/-upgrade: not found
    /etc/rc: cannot create /tmp/php_errors.txt: Read-only file system
    /etc/rc: cannot create /dev/null: Operation not supported
    Launching the init system...rm: /cf/conf/backup/backup.cache: Read-only file system
     done.
    Initializing.................. done.
    Error: cannot open dmesg.boot in system_dmesg_save().
    cannot create /dev/null: Operation not supported
    cannot create /dev/null: Operation not supported
    Starting device manager (devd)...done.
    Loading configuration.....PHP Fatal error:  Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205
    
    Fatal error: Call to undefined function xml_parser_create() in /etc/inc/xmlparse.inc on line 205
    PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 205, Message: Call to undefined function xml_parser_create()Starting CRON... /etc/rc: cannot create /dev/null: Operation not supported
    done.
    fcgicli: Could not connect to server(/var/run/php-fpm.socket).
    /etc/rc: /usr/local/sbin/-upgrade: not found
     (pfSense) 2.3.1-RELEASE  Tue May 17 18:46:53 CDT 2016
    Bootup complete
    /etc/rc: cannot create /dev/null: Operation not supported
    rm: /tmp/config.cache: Read-only file system</synthetic></synthetic></synthetic></synthetic></msft></xsaveopt></erms></lahf></syscall,nx,lm></sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,popcnt,aesni,xsave,osxsave,avx,f16c,rdrand,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt>
    

    In this particular case, it seems it's trying to rm -rf itself, but it hasn't mounted the filesystem in r/w mode in order to do that.

    My next attempt (i.e. start again with a fresh install, then restore config.xml, then try upgrade) did this on boot (which looks similar to the OP's behavior where it's trying to parse an invalid file):

    start_init: trying /sbin/init
    May 29 19:07:12 init: login_getclass: unknown class 'daemon'
    
    May 29 19:07:12 init: login_getclass: no default/fallback class 'default'
    
    Configuring crash dumps...
    Using /dev/label/swap0 for dump device.
    Filesystems are clean, continuing...
    Mounting filesystems...
    [: -eq: unexpected operator
    
    Welcome to   (Patch ) on the '' platform...
    
    Dump device does not exist.  Savecore not run.
    Creating symlinks......done.
    /etc/rc: /usr/local/sbin/-upgrade: not found
    /etc/rc: cannot create /tmp/php_errors.txt: Read-only file system
    pid 89 (php-fpm), uid 0: exited on signal 11
    Segmentation fault
    fcgicli: Could not connect to server(/var/run/php-fpm.socket).
    Launching the init system...pid 94 (php-cgi), uid 0: exited on signal 11
    Segmentation fault
    Starting CRON... done.
    fcgicli: Could not connect to server(/var/run/php-fpm.socket).
    /etc/rc: /usr/local/sbin/-upgrade: not found
     ()  (Patch )  
    Bootup complete[/code]
    
    I did boot this machine back up off the installation disc and found that most of the files in /sbin and /bin were 0 bytes in length - even fsck and ls were not usable in this state.
    
    If I restored my config.xml file [i]after[/i] performing the upgrade (which I did try multiple times to see if it was definitely not a fluke), the upgrade itself would always work correctly (and it wasn't crashing at the "Configuring WAN interface..." step of the boot process either). Since this has worked, I haven't really worried about it and most of the testing I did was more from a curiosity point of view, since my main gateway was working again after I did it the aforementioned way. In my case, the disk isn't a suspect as this disk is used by both other VMs and the host for storing data; and the consistency at which I can replicate the issue by restoring my config.xml, [i]then[/i] upgrading.
    
    ![pfSenseBTXHalt.png](/public/_imported_attachments_/1/pfSenseBTXHalt.png)
    ![pfSenseBTXHalt.png_thumb](/public/_imported_attachments_/1/pfSenseBTXHalt.png_thumb)