2.3-RELEASE -> 2.3.1 upgrade hangs; 100%CPU



  • Hey,

    When I run the autoupgrader through the GUI it never completes; I SSH'd in and ps aux to see that it's sitting doing:
    pkg upgrade pkg
    at 100% cpu

    If I shell in and run the command manually it too gets stuck:

    
    [2.3-RELEASE][admin@router.home]/root: pkg -d upgrade pkg
    DBG(1)[12923]> pkg initialized
    Updating pfSense-core repository catalogue...
    DBG(1)[12923]> PkgRepo: verifying update for pfSense-core
    DBG(1)[12923]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
    DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-core/meta.txz with opts "i"
    DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-core/packagesite.txz with opts "i"
    pfSense-core repository is up-to-date.
    Updating pfSense repository catalogue...
    DBG(1)[12923]> PkgRepo: verifying update for pfSense
    DBG(1)[12923]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
    DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-pfSense_v2_3_0/meta.txz with opts "i"
    DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-pfSense_v2_3_0/packagesite.txz with opts "i"
    pfSense repository is up-to-date.
    All repositories are up-to-date.
    DBG(1)[12923]> want to get an advisory lock on a database
    DBG(1)[12923]> found stale pid 87017 in lock database, my pid is: 12923
    DBG(1)[12923]> no concurrent processes found, cleanup the lock
    New version of pkg detected; it needs to be installed first.
    The following 1 package(s) will be affected (of 0 checked):
    
    Installed packages to be UPGRADED:
            pkg: 1.6.2 -> 1.7.2_2 [pfSense]
    
    The process will require 91 KiB more space.
    2 MiB to be downloaded.
    
    Proceed with this action? [y/N]: y
    
    

    Running top I see:

    
    last pid: 69543;  load averages:  0.93,  0.49,  0.36                                                                                                        up 0+00:14:59  22:29:43
    45 processes:  2 running, 43 sleeping
    CPU:  2.2% user,  0.0% nice, 24.9% system,  0.1% interrupt, 72.8% idle
    Mem: 65M Active, 42M Inact, 151M Wired, 64M Buf, 1693M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    12923 root        1 103    0 45144K  9508K CPU1    1   2:25 100.00% pkg
    15713 root        1  21    0   262M 28184K accept  0   0:01   0.98% php-fpm
    
    

    I'm not really sure what to do from here, any suggestions?

    Thanks in advance,
    Rob



  • I just had a thought and ran truss against the pid, it appears to be stuck in some insane loop:

    
    statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
    mkdir("/var",0777)                               ERR#17 'File exists'
    mkdir("/var/cache",0777)                         ERR#17 'File exists'
    mkdir("/var/cache/pkg",0777)                     ERR#17 'File exists'
    statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
    mkdir("/var",0777)                               ERR#17 'File exists'
    mkdir("/var/cache",0777)                         ERR#17 'File exists'
    mkdir("/var/cache/pkg",0777)                     ERR#17 'File exists'
    statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
    ^C
    [2.3-RELEASE][admin@router.home]/root: 
    
    

    I renamed pkg to pkg.old and voila; the pkg upgrade worked. I don't understand how this happened though. This box was upgraded 2.1.x -> 2.2.x -> 2.3-RELEASE without any issues o.0

    The file that was there was a symlink too:

    
    lrwxr-xr-x   1 root  wheel     24 May  5 21:02 pkg.old -> ../../root/var/cache/pkg
    
    

Log in to reply