Upgrade to 2.1.2: Stuck on 2.1

kaput · Apr 14, 2014, 3:27 PM

I'm seeing the same issue on an ALIX board. The upgrade goes through with no errors reported on screen/console, but when it reboots, it boots back into 2.1. The upgrade log has only the following that appears notable.

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
shutdown: [pid 75726]

sashka · Apr 14, 2014, 3:30 PM

@kaput are you on 4g image?

kaput · Apr 14, 2014, 3:51 PM

Yes.

sashka · Apr 14, 2014, 4:06 PM

Can someone test 4g image to see if it's not broken? I had no problems upgrading 2g images, but 4g gives problem.

jimp · Apr 14, 2014, 4:32 PM

Provide the whole upgrade log. The part being quoted isn't the problem.

We tested the 4GB version more than others, as it's the most common one in use.

kaput · Apr 14, 2014, 4:43 PM

http://pastebin.com/YX8vK7C1

jimp · Apr 14, 2014, 4:45 PM

Here's your problem:

gpart set -a active -i 1 ad0
gpart: table 'ad0' is corrupt: Operation not permitted

Try this:

gpart bootcode -b /boot/mbr ad0

Then try the upgrade again.

kaput · Apr 14, 2014, 4:49 PM

Thanks, jimp. Missed that perm issue in the log.

Unfortunately, running "gpart bootcode -b /boot/mbr ad0" also results in the exact same error, noting that ad0 is corrupt.

prohidium · Apr 14, 2014, 5:12 PM

I have the exact same issue with same error in my upgrade log on a 2g install on an Alix box.

I also had the issue on a 4g install but I opted to flash a new image and swap cards. The 2g box though is 4 hours round trip away so I would love to fix the issue and update remotely.

I also had another 4g box that updated just fine. So, 2 out of 3 are broken and would not update with autoupdate or manual update.

jimp · Apr 14, 2014, 5:19 PM

Other things to try:

Set the boot slice on Diagnostics > NanoBSD, see if it actually changes.

or

gpart recover ad0

or

sysctl kern.geom.debugflags=16
boot0cfg -v -s 1 ad0
fdisk -a /dev/ad0
sysctl kern.geom.debugflags=0

(For fdisk make sure to specify the values it asks and press 'y' to confirm!)

We have yet to be able to reproduce this effectively, but several people have hit it. It would be helpful to know some more about the original setup that failed. Was it your own CF or did it come with the device? Had it ever been upgraded before, and if so how many times? What was its original version of pfSense on that CF card before upgrading?

sashka · Apr 14, 2014, 5:23 PM

Slice does not change; nor I can copy ad0s2 -> ad0s1.

kaput · Apr 14, 2014, 5:35 PM

 gpart bootcode -b /boot/mbr ad0
gpart: table 'ad0' is corrupt: Operation not permitted

gpart recover ad0
gpart: recovering 'ad0' failed: Function not implemented

sashka · Apr 14, 2014, 5:43 PM


fdisk -a /dev/ad0

I get an Intput/Output error.

This is compact flash card. It was installed as 2.0 and then upgraded to 2.0.3 and later 2.1.

robi · Apr 14, 2014, 7:31 PM

In my case I had pfSense nanobsd v1.2.3 upgraded to v2.0, that one upgraded to v2.1 - all of them upgraded OK. Now when I wanted to upgrade to v2.1.2, It seems corrupted. I'm preparing a fresly flashed CF card with config already put in.

me too · Apr 15, 2014, 4:31 AM

I am having the same issue. Hopefully my situation will help shed some light.

I have upgraded at least two other NanoBSD 4G (ALIX) system without issue.
Today I am upgrading three identical boxes - ALIX boards running 4G CF card
One existing install was already 2.1 and I successfully upgraded to 2.1.2
The other two were at 2.0.3 and I successfully upgraded to 2.1, but they are both now stuck on 2.1 after auto-upgrade and manual upgrade attempts.

robi · Apr 15, 2014, 7:30 AM

Also a side note, I'm not on Alix, but on a Jetway JNF99 with 5 intel nics, Atom CPU and 4GB of RAM, i386 build. The new CF card will carry a 64-bit version this time.

jimp · Apr 15, 2014, 1:27 PM

If you're getting an "i/o error" or "device not configured", reboot and try again. At least for the "i/o error" it seems to lean more toward maybe a bad CF but it's still tough to say for sure.

Also worth a try:

sysctl kern.geom.debugflags=16
fdisk -B -b /boot/boot0 ad0

mjohnson · Apr 17, 2014, 4:58 AM

I'm in the same boat with 40 plus devices scattered across 4 provinces. I actually haven't managed to get a single one to upgrade except the 3 in our local offices that I performed a clean install on, 2G and 4G, current versions 2.03 and 2.1 on all remotes attempting auto upgrade and manual firmware uploads. Not sure what to do, since I don't want to risk chopping them up and losing remote connectivity. I don't own a helicopter to get around that quickly ;D

All Alix boards. Errors in the upgrade log

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
tar: Failed to set default locale
tar: Failed to set default locale
shutdown: [pid 24752]

gazoo · Apr 18, 2014, 3:30 PM

I have same issue apparently. No answer. Here's my log:

NanoBSD Firmware upgrade in progress...

Installing /root/latest.tgz.
SLICE         1
OLDSLICE      2
TOFLASH       ad0s1
COMPLETE_PATH ad0s1a
GLABEL_SLICE  pfSense0
Sun Apr 13 02:37:42 EDT 2014

total 8
dr-xr-xr-x   7 root  wheel         512B Apr 12 15:03 .
drwxr-xr-x  26 root  wheel         1.0k Apr 12 15:03 ..
crw-r-----   1 root  operator    0,  56 Apr 12 15:03 ad0
crw-r-----   1 root  operator    0,  57 Apr 12 15:03 ad0s1
crw-r-----   1 root  operator    0,  60 Apr 12 15:03 ad0s1a
crw-r-----   1 root  operator    0,  58 Apr 12 15:03 ad0s2
crw-r-----   1 root  operator    0,  61 Apr 12 15:03 ad0s2a
crw-r-----   1 root  operator    0,  59 Apr 12 15:03 ad0s3
crw-------   1 root  operator    0,  28 Apr 12 15:03 ata
crw-------   1 root  wheel       0,  11 Apr 13 01:29 bpf
lrwxr-xr-x   1 root  wheel           3B Apr 12 15:03 bpf0 -> bpf
crw-------   1 root  tty         0,   4 Apr 13 02:37 console
crw-rw-rw-   1 root  wheel       0,  44 Apr 12 15:03 crypto
crw-rw-rw-   1 root  wheel       0,  10 Apr 12 15:03 ctty
crw-rw----   1 uucp  dialer      0,  35 Apr 12 15:03 cuau0
crw-rw----   1 uucp  dialer      0,  36 Apr 12 15:03 cuau0.init
crw-rw----   1 uucp  dialer      0,  37 Apr 12 15:03 cuau0.lock
crw-rw----   1 uucp  dialer      0,  41 Apr 12 15:03 cuau1
crw-rw----   1 uucp  dialer      0,  42 Apr 12 15:03 cuau1.init
crw-rw----   1 uucp  dialer      0,  43 Apr 12 15:03 cuau1.lock
crw-------   1 root  wheel       0,   5 Apr 12 15:03 devctl
cr--------   1 root  wheel       0,  54 Apr 12 15:03 devstat
dr-xr-xr-x   2 root  wheel         512B Apr 12 15:03 fd
crw-------   1 root  wheel       0,  13 Apr 12 15:03 fido
crw-r-----   1 root  operator    0,   3 Apr 12 15:03 geom.ctl
crw-------   1 root  wheel       0,  23 Apr 12 15:03 io
crw-------   1 root  wheel       0,   8 Apr 12 15:03 klog
crw-r-----   1 root  kmem        0,  15 Apr 12 15:03 kmem
dr-xr-xr-x   2 root  wheel         512B Apr 12 15:03 led
crw-r-----   1 root  operator    0,  62 Apr 12 15:03 md0
crw-r-----   1 root  operator    0,  66 Apr 12 15:03 md1
crw-------   1 root  wheel       0,  47 Apr 12 15:03 mdctl
crw-r-----   1 root  kmem        0,  14 Apr 12 15:03 mem
crw-------   1 root  kmem        0,  16 Apr 12 15:03 nfslock
crw-rw-rw-   1 root  wheel       0,  25 Apr 13 02:37 null
crw-r--r--   1 root  wheel       0,  27 Apr 12 15:03 pci
crw-rw----   1 root  proxy       0,  45 Apr 12 15:03 pf
crw-rw-rw-   1 root  wheel       0,   9 Apr 12 15:03 ptmx
crw-rw-rw-   1 root  wheel       0,   6 Apr 12 15:03 random
crw-------   1 root  wheel       0,  24 Apr 12 15:03 speaker
lrwxr-xr-x   1 root  wheel           4B Apr 12 15:03 stderr -> fd/2
lrwxr-xr-x   1 root  wheel           4B Apr 12 15:03 stdin -> fd/0
lrwxr-xr-x   1 root  wheel           4B Apr 12 15:03 stdout -> fd/1
crw-------   1 root  wheel       0,  32 Apr 12 15:03 ttyu0
crw-------   1 root  wheel       0,  33 Apr 12 15:03 ttyu0.init
crw-------   1 root  wheel       0,  34 Apr 12 15:03 ttyu0.lock
crw-------   1 root  wheel       0,  38 Apr 12 15:03 ttyu1
crw-------   1 root  wheel       0,  39 Apr 12 15:03 ttyu1.init
crw-------   1 root  wheel       0,  40 Apr 12 15:03 ttyu1.lock
crw-------   1 uucp  dialer      0,  73 Apr 12 15:04 tun1
dr-xr-xr-x   2 root  wheel         512B Apr 12 15:03 ufs
dr-xr-xr-x   2 root  wheel         512B Apr 12 15:03 ufsid
lrwxr-xr-x   1 root  wheel           9B Apr 12 15:03 ugen0.1 -> usb/0.1.0
lrwxr-xr-x   1 root  wheel           9B Apr 12 15:03 ugen1.1 -> usb/1.1.0
lrwxr-xr-x   1 root  wheel           6B Apr 12 15:03 urandom -> random
dr-xr-xr-x   2 root  wheel         512B Apr 12 15:03 usb
crw-r--r--   1 root  operator    0,  46 Apr 12 15:03 usbctl
crw-------   1 root  operator    0,  55 Apr 12 15:03 xpt0
crw-rw-rw-   1 root  wheel       0,  26 Apr 12 15:03 zero

-rw-r--r--  1 root  wheel    76M Apr 13 02:34 /root/latest.tgz

MD5 (/root/latest.tgz) = b914649dd6be90a461a99615c07882c3

/dev/ufs/pfSense1 on / (ufs, local, noatime, synchronous)
devfs on /dev (devfs, local)
/dev/ufs/cf on /cf (ufs, local, noatime, synchronous)
/dev/md0 on /tmp (ufs, local)
/dev/md1 on /var (ufs, local)
devfs on /var/dhcpd/dev (devfs, local)

last pid: 52248;  load averages:  3.09,  1.78,  1.20  up 0+11:34:50    02:38:14
67 processes:  7 running, 60 sleeping

Mem: 92M Active, 50M Inact, 74M Wired, 6780K Cache, 33M Buf, 7052K Free
Swap: 

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
92215 root        1  76    0 40308K 20156K accept   4:21  0.00% php
86955 root        1  76    0 26484K 15784K accept   2:37  0.00% php
41671 root        1  44    0  8004K  4984K kqread   1:04  0.00% lighttpd
93099 root        1  76   20  3644K  1332K piperd   0:20  0.00% sh
  292 root        1  76   20  3352K  1112K kqread   0:15  0.00% check_reload_status
45599 nobody      1  44    0  5512K  2316K select   0:12  0.00% dnsmasq
31777 root        1  44    0  3264K  1208K select   0:12  0.00% apinger
97171 root        1  44    0  3416K  1400K select   0:10  0.00% syslogd
66552 root        1  44    0  7284K  5872K select   0:07  0.00% bsnmpd
30425 root       21  44    0 21400K  3996K ucond    0:05  0.00% filterdns
78170 root        1  64   20  6280K  6300K select   0:05  0.00% ntpd
52603 dhcpd       1  44    0 11456K  7584K select   0:05  0.00% dhcpd
25251 root        1  44    0  3264K   848K piperd   0:03  0.00% logger
90659 proxy       1  64   20 10356K  5928K kqread   0:03  0.00% squid
25151 root        1  44    0  5868K  1916K bpf      0:02  0.00% tcpdump
61760 root        1  44    0  5952K  2324K kqread   0:01  0.00% lighttpd
72531 root        1  64   20  5432K  3224K select   0:01  0.00% openvpn
19617 root        1  44    0  5144K  1544K select   0:01  0.00% hostapd

NanoBSD upgrade starting

dd if=/dev/zero of=/dev/ad0s1 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.344171 secs (3046671 bytes/sec)

/usr/bin/gzip -dc /root/latest.tgz | /bin/dd of=/dev/ad0s1 obs=64k
3844449+0 records in
30034+1 records out
1968357888 bytes transferred in 394.695594 secs (4987028 bytes/sec)
After upgrade fdisk/bsdlabel

/sbin/fsck_ufs -y /dev/ad0s1a
** /dev/ad0s1a
** Last Mounted on /tmp/netgatemnt
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
5506 files, 317470 used, 3462558 free (542 frags, 432752 blocks, 0.0% fragmentation)

***** FILE SYSTEM IS CLEAN *****

/sbin/tunefs -L pfSense0 /dev/ad0s1a
Checking for post_upgrade_command...
Found post_upgrade_command, executing (pfSense0)...
tar: Failed to set default locale
Checking for /tmp/pfSense0/tmp/post_upgrade_command.php... 
Running /tmp/pfSense0/tmp/post_upgrade_command.php pfSense0
Adding serial port settings (/tmp/pfSense0)...
Reading /tmp/pfSense0/boot/loader.conf...

/dev/ufs/pfSense0 / ufs ro,sync,noatime 1 1
/dev/ufs/cf /cf ufs ro,sync,noatime 1 1

gpart set -a active -i 1 ad0
gpart: table 'ad0' is corrupt: Operation not permitted

/usr/sbin/boot0cfg -s 1 -v /dev/ad0
#   flag     start chs   type       end chs       offset         size
1   0x00      0:  1: 1   0xa5    751: 15:63           63      3854529
2   0x80    752:  1: 1   0xa5    479: 15:63      3854655      3854529
3   0x00    480:  0: 1   0xa5    581: 15:63      7709184       102816

version=2.0  drive=0x80  mask=0x3  ticks=182  bell=# (0x23)
options=packet,update,nosetdrv
volume serial ID 9090-9090
default_selection=F1 (Slice 1)
Sun Apr 13 02:48:27 EDT 2014

NanoBSD Firmware upgrade is complete.  Rebooting in 10 seconds.

File list:
/tmp/pfSense0
/tmp/pfSense0/.snap
/tmp/pfSense0/COPYRIGHT
/tmp/pfSense0/bin
/tmp/pfSense0/bin/cat
/tmp/pfSense0/bin/chflags
/tmp/pfSense0/bin/chmod
/tmp/pfSense0/bin/cp
/tmp/pfSense0/bin/csh
/tmp/pfSense0/bin/tcsh
/tmp/pfSense0/bin/date
/tmp/pfSense0/bin/dd
/tmp/pfSense0/bin/df
/tmp/pfSense0/bin/domainname
/tmp/pfSense0/bin/echo
/tmp/pfSense0/bin/expr
/tmp/pfSense0/bin/hostname
/tmp/pfSense0/bin/kenv
/tmp/pfSense0/bin/kill
/tmp/pfSense0/bin/ln
/tmp/pfSense0/bin/link
.....a bunch more tmp files..
Misc log:

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
tar: Failed to set default locale
tar: Failed to set default locale
shutdown: [pid 82135]

fdisk/bsdlabel log:

Before upgrade fdisk/bsdlabel
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 3854529 (1882 Meg), flag 0
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 751/ head 15/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 3854655, size 3854529 (1882 Meg), flag 80 (active)
	beg: cyl 752/ head 1/ sector 1;
	end: cyl 479/ head 15/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 7709184, size 102816 (50 Meg), flag 0
	beg: cyl 480/ head 0/ sector 1;
	end: cyl 581/ head 15/ sector 63
The data for partition 4 is:
 <unused># /dev/ad0s1:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
# /dev/ad0s2:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
---------------------------------------------------------------

******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 3854529 (1882 Meg), flag 0
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 751/ head 15/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 3854655, size 3854529 (1882 Meg), flag 80 (active)
	beg: cyl 752/ head 1/ sector 1;
	end: cyl 479/ head 15/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 7709184, size 102816 (50 Meg), flag 0
	beg: cyl 480/ head 0/ sector 1;
	end: cyl 581/ head 15/ sector 63
The data for partition 4 is:
 <unused># /dev/ad0s1:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
# /dev/ad0s2:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
---------------------------------------------------------------

Final upgrade fdisk/bsdlabel
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 3854529 (1882 Meg), flag 0
	beg: cyl 0/ head 1/ sector 1;
	end: cyl 751/ head 15/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 3854655, size 3854529 (1882 Meg), flag 80 (active)
	beg: cyl 752/ head 1/ sector 1;
	end: cyl 479/ head 15/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 7709184, size 102816 (50 Meg), flag 0
	beg: cyl 480/ head 0/ sector 1;
	end: cyl 581/ head 15/ sector 63
The data for partition 4 is:
 <unused># /dev/ad0s1:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
# /dev/ad0s2:
type: unknown
disk: amnesiac
label: 
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 3813
sectors/unit: 3844449
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0		# milliseconds
track-to-track seek: 0	# milliseconds
drivedata: 0 

8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:    3844433         16    unused        0     0   
  c:    3844449          0    unused        0     0     # "raw" part, don't edit
---------------------------------------------------------------</unused></unused></unused>

Darkk · Apr 19, 2014, 7:04 PM

@mjohnson:

I'm in the same boat with 40 plus devices scattered across 4 provinces. I actually haven't managed to get a single one to upgrade except the 3 in our local offices that I performed a clean install on, 2G and 4G, current versions 2.03 and 2.1 on all remotes attempting auto upgrade and manual firmware uploads. Not sure what to do, since I don't want to risk chopping them up and losing remote connectivity. I don't own a helicopter to get around that quickly ;D

All Alix boards. Errors in the upgrade log

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
bsdlabel: /dev/ad0s3: no valid label found
tar: Failed to set default locale
tar: Failed to set default locale
shutdown: [pid 24752]

I hear ya about all those remote devices and no time to travel there if there is a problem. For me if there is a risk of breaking it at a remote site I would send them another firewall box and tell them to swap it when time permits. This way the new firewall is working at the corporate office with the remote site's configuration file and should work fine when it reaches at the remote site. Then have them send the old one back to redo for another site.

I know it's a PITA. Might be good for critical sites that can't go down for any period of time. Hopefully soon we can get these upgrade issues sorted out.