2.3-RELEASE -> 2.3.1 upgrade hangs; 100%CPU

[RobEmery]

Hey,

When I run the autoupgrader through the GUI it never completes; I SSH'd in and ps aux to see that it's sitting doing:
pkg upgrade pkg
at 100% cpu

If I shell in and run the command manually it too gets stuck:

[2.3-RELEASE][admin@router.home]/root: pkg -d upgrade pkg
DBG(1)[12923]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[12923]> PkgRepo: verifying update for pfSense-core
DBG(1)[12923]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-core/meta.txz with opts "i"
DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-core/packagesite.txz with opts "i"
pfSense-core repository is up-to-date.
Updating pfSense repository catalogue...
DBG(1)[12923]> PkgRepo: verifying update for pfSense
DBG(1)[12923]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-pfSense_v2_3_0/meta.txz with opts "i"
DBG(1)[12923]> Fetch: fetching from: http://pkg.pfsense.org/pfSense_v2_3_0_amd64-pfSense_v2_3_0/packagesite.txz with opts "i"
pfSense repository is up-to-date.
All repositories are up-to-date.
DBG(1)[12923]> want to get an advisory lock on a database
DBG(1)[12923]> found stale pid 87017 in lock database, my pid is: 12923
DBG(1)[12923]> no concurrent processes found, cleanup the lock
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.6.2 -> 1.7.2_2 [pfSense]

The process will require 91 KiB more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y

Running top I see:

last pid: 69543;  load averages:  0.93,  0.49,  0.36                                                                                                        up 0+00:14:59  22:29:43
45 processes:  2 running, 43 sleeping
CPU:  2.2% user,  0.0% nice, 24.9% system,  0.1% interrupt, 72.8% idle
Mem: 65M Active, 42M Inact, 151M Wired, 64M Buf, 1693M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
12923 root        1 103    0 45144K  9508K CPU1    1   2:25 100.00% pkg
15713 root        1  21    0   262M 28184K accept  0   0:01   0.98% php-fpm

I'm not really sure what to do from here, any suggestions?

Thanks in advance,
Rob

[RobEmery]

I just had a thought and ran truss against the pid, it appears to be stuck in some insane loop:

statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
mkdir("/var",0777)                               ERR#17 'File exists'
mkdir("/var/cache",0777)                         ERR#17 'File exists'
mkdir("/var/cache/pkg",0777)                     ERR#17 'File exists'
statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
mkdir("/var",0777)                               ERR#17 'File exists'
mkdir("/var/cache",0777)                         ERR#17 'File exists'
mkdir("/var/cache/pkg",0777)                     ERR#17 'File exists'
statfs("/var/cache/pkg",0x7fffffffdfa8)          ERR#2 'No such file or directory'
^C
[2.3-RELEASE][admin@router.home]/root: 

I renamed pkg to pkg.old and voila; the pkg upgrade worked. I don't understand how this happened though. This box was upgraded 2.1.x -> 2.2.x -> 2.3-RELEASE without any issues o.0

The file that was there was a symlink too:

lrwxr-xr-x   1 root  wheel     24 May  5 21:02 pkg.old -> ../../root/var/cache/pkg