Upgrade to 2.1.2: Stuck on 2.1

RCS-Michael

Pmiller,

That's nice that you were able to do that, but it's not an option for some of us. I have pfSense firewalls distributed around the world. Am I supposed to hop on my jet and get this all done? Some sites don't have people onsite to ship it to me.

I, for one, am still waiting on a workable solution.

Michael

VampBoy

Finally found the right googleFU to find this thread ;)

I'm in the same boat with ad0 corrupt during the upgrade whether or not auto or manual, going from 2.1 to 2.1.3 (tried 2.1.2 for giggles with the same result)

Running a netgate fw-7535 with a 4gb CF card that has been upgraded successfully in the past.

fdisk output looks the same as others have posted.


$ fdisk -p /dev/ad0
# /dev/ad0
g c7745 h16 s63
p 1 0xa5 63 3854529
p 2 0xa5 3854655 3854529
a 2
p 3 0xa5 7709184 102816

Hours later
Ended up flashing to a new card which was a massive PITA as I had forgotten how to do it and the new card kept hanging on boot.

In case it helps here's a fdisk dump of the newly working 4g card I have in there now. (running 4g nano amd64 2.1.3)


$ fdisk -p /dev/ad0
# /dev/ad0
g c7773 h16 s63
p 1 0xa5 63 3844449
a 1
p 2 0xa5 3844575 3844449
p 3 0xa5 7689024 102816

jimp

Can someone try setting this tunable:

sysctl kern.geom.part.check_integrity=0

And then perform an upgrade, see if that lets it get by.

njlinuxmike

@jimp:

Can someone try setting this tunable:
sysctl kern.geom.part.check_integrity=0
And then perform an upgrade, see if that lets it get by.

I tried the above. No joy.

This is a Soekris Net 5501-60 running pfSense 2.1-RELEASE NanoBSD 4G trying to up to 2.1.3

Additionally, after the first or second attempt pfBlocker was not re-installed.

/dev/ad0

g c7745 h16 s63
p 1 0xa5 63 3861585
a 1
p 2 0xa5 3861711 3861585
p 3 0xa5 7723296 102816

Upgrade log available here:
http://pastebin.com/tq6Y3gPq

Multiple attempts to upgrade using manual and auto.

eweri

Hello!

Do not know it this helps but I could not update my APU with SanDisk 4GB card and I started testing the sd-card.

And this is what I got:
dd if=/dev/random of=/dev/rdisk5 bs=64k
dd: /dev/rdisk5: Input/output error
60505+0 records in
60504+0 records out
3965190144 bytes transferred in 596.342498 secs (6649183 bytes/sec)

This means that the 4GB card seams to have a bad block there, so I tested the next card of the same type and was very surprised that the second card had a bad block at the exact same block number. :o

Maybe it is a fake 4GB card?
It is printed with "SanDisk Ultra 30MB/s SDHC I 4GB"

At the moment I am testing the third card …. ups ... same result - I am going to call my Supplier >:(

Bye,
eweri

P.S. Okay - spoke to my supplier - this is expected - a 4GB SanDisk SD-Card has a capacity of 3.965 billion bytes and not 4 billion bytes as printed on the label :-\

kpa

I/O error means that either there is a real bad block on the card or there's some odd compatibility issue with the card/controller combination with FreeBSD 8.3. I would test the cards on a different machine to see if the issue can be repeated there.

luckman212

I've got a few Netgate FW-7535's that are "stuck" with 2.1 or 2.1.3 and trying to upgrade them to 2.1.4 seems to go OK but after the reboot they are back on their original version. Both stuck units that I happened to be banging my head against tonight are running a nanobsd 2g (i386) image burned to a SanDisk Extreme 2GB CF P/N# SDCFH-002G

Here's some fdisk output (both units give the identical output)

fdisk -p /dev/ad0
# /dev/ad0
g c3875 h16 s63
p 1 0xa5 63 1902033
p 2 0xa5 1902159 1902033
a 2
p 3 0xa5 3804192 102816

I'm reluctant to roll out there to swap CF cards as it would be both time consuming and cause noticeable downtime. If anyone knows of a way to fix this conundrum remotely I would be so grateful! ::)

mickrussom

I'm stuck too! 2.1 is stuck in place, and I cannot go to anything past 2.1-Release.

I have a router with a 6 interfaces, serial, with a 4GB card.

I upgraded to 2.1 no issues.

However, no upgrades (2.1.1, .2, .3, .4) can be applied thereafter.

I am attaching a log of the attempt at an upgrade, within there is a dmi/smbios dump, an lspci, the upgrade log and the fdisk output.

The two main errors I see are:

gpart set -a active -i 2 ad0
gpart: table 'ad0' is corrupt: Operation not permitted

fdisk: invalid fdisk partition table found

This unit is remote and difficult to reach, I would like to try to fix it without going there.

The unit is a LANNER FW-7535 .

Please help.

upgradelog.txt

mickrussom

@jimp:

I tried all the remediations suggested.

1)
dd if=/dev/ad0 of=/tmp/mbr_part_bkup.img bs=512 count=1
dd of=/dev/ad0 if=/tmp/mbr_part_bkup.img bs=512 count=1

still "corrupt"

2)
[2.1-RELEASE][admin@pg-router-5/root(24): fdisk -B -b /boot/boot0 ad0

******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 3854529 (1882 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 751/ head 15/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 3854655, size 3854529 (1882 Meg), flag 0
beg: cyl 752/ head 1/ sector 1;
end: cyl 479/ head 15/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 7709184, size 102816 (50 Meg), flag 0
beg: cyl 480/ head 0/ sector 1;
end: cyl 581/ head 15/ sector 63
The data for partition 4 is:
<unused>Do you want to change the boot code? [n] y

We haven't changed the partition table yet. This is your last chance.
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Information from DOS bootblock is:
1: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 3854529 (1882 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 751/ head 15/ sector 63
2: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 3854655, size 3854529 (1882 Meg), flag 0
beg: cyl 752/ head 1/ sector 1;
end: cyl 479/ head 15/ sector 63
3: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 7709184, size 102816 (50 Meg), flag 0
beg: cyl 480/ head 0/ sector 1;
end: cyl 581/ head 15/ sector 63
4: <unused>Should we write new partition table? [n] y

3)
[2.1-RELEASE][admin@pg-router-5]/root(23): gpart bootcode -b /boot/mbr ad0
gpart: table 'ad0' is corrupt: Operation not permitted

4)
[2.1-RELEASE][admin@pg-router-5]/root(26): gpart recover ad0
gpart: recovering 'ad0' failed: Function not implemented

5)
[2.1-RELEASE][admin@pg-router-5.]/root(27): boot0cfg -v -s 1 ad0
# flag start chs type end chs offset size
1 0x80 0: 1: 1 0xa5 751: 15:63 63 3854529
2 0x00 752: 1: 1 0xa5 479: 15:63 3854655 3854529
3 0x00 480: 0: 1 0xa5 581: 15:63 7709184 102816

version=2.0 drive=0x80 mask=0xf ticks=182 bell=# (0x23)
options=packet,update,nosetdrv
volume serial ID a8a8-a8a8
default_selection=F1 (Slice 1)

[2.1-RELEASE][admin@pg-router-5.]/root(28): fdisk -a /dev/ad0
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 3854529 (1882 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 751/ head 15/ sector 63
The data for partition 2 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 3854655, size 3854529 (1882 Meg), flag 0
beg: cyl 752/ head 1/ sector 1;
end: cyl 479/ head 15/ sector 63
The data for partition 3 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 7709184, size 102816 (50 Meg), flag 0
beg: cyl 480/ head 0/ sector 1;
end: cyl 581/ head 15/ sector 63
The data for partition 4 is:
<unused>Partition 1 is marked active
Do you want to change the active partition? [n]

We haven't changed the partition table yet. This is your last chance.
parameters extracted from in-core disklabel are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=7745 heads=16 sectors/track=63 (1008 blks/cyl)

Information from DOS bootblock is:
1: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 3854529 (1882 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 751/ head 15/ sector 63
2: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 3854655, size 3854529 (1882 Meg), flag 0
beg: cyl 752/ head 1/ sector 1;
end: cyl 479/ head 15/ sector 63
3: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 7709184, size 102816 (50 Meg), flag 0
beg: cyl 480/ head 0/ sector 1;
end: cyl 581/ head 15/ sector 63
4: <unused>Should we write new partition table? [n] y

I still have this issue:
[2.1-RELEASE][admin@router-5]/root(22): gpart status
Name Status Components
ad0s1 CORRUPT ad0
ad0s2 CORRUPT ad0
ad0s3 CORRUPT ad0
ad0s1a OK ad0s1
ad0s2a OK ad0s2

(33): gpart show
=> 63 7806897 ad0 MBR (3.7G) [CORRUPT]
63 3854529 1 freebsd [active] (1.9G)
3854592 63 - free - (31k)
3854655 3854529 2 freebsd (1.9G)
7709184 102816 3 freebsd (50M)

=> 0 3854529 ad0s1 BSD (1.9G)
0 16 - free - (8.0k)
16 3844433 1 !0 (1.9G)
3844449 10080 - free - (4.9M)
Is there a way to fix this? Seems I need to repair whatever it is that gpart looks at if this is ever going to be upgradeable again.

There was an upgrade to 2.1 before this, all upgrades were done via the GUI.

Would this work?

DISK=ad0

offset=diskinfo $DISK | awk '{ print $4 - 131072 }'
dd if=/dev/zero of=/dev/$DISK bs=64k count=1
dd if=/dev/zero of=/dev/$DISK bs=64k seek=$offset

gpart create -s gpt ${DISK}</unused></unused></unused></unused>

mickrussom

@jimp:

Can someone try setting this tunable:
sysctl kern.geom.part.check_integrity=0
And then perform an upgrade, see if that lets it get by.

I tried this; it came back with 2.1 again.

[2.1-RELEASE][admin@pg-router-5.]/root(4): sysctl -a | grep -i geom
kern.geom.part.check_integrity: 0

[2.1-RELEASE][admin@pg-router-5.]/root(5): exit

[2.1-RELEASE][admin@pg-router-5.]/root(2): exit
exit
*** Welcome to pfSense 2.1-RELEASE-nanobsd (i386) on pg-router-5 ***

WAN (wan) -> em5 -> v4/DHCP4: 192.168.7.139/24
LAN (lan) -> em4 -> v4: 192.168.14.1/24
OPT1 (opt1) -> em2 -> v4: 192.168.16.1/24
OPT2 (opt2) -> em0 ->
OPT3 (opt3) -> em1 -> v4: 192.168.17.1/24
OPT4 (opt4) -> em3 -> v4: 192.168.15.1/24
OPT5 (opt5) -> em0_vlan20 ->

Logout (SSH only) 8) Shell
Assign Interfaces 9) pfTop
Set interface(s) IP address 10) Filter Logs
Reset webConfigurator password 11) Restart webConfigurator
Reset to factory defaults 12) pfSense Developer Shell
Reboot system 13) Upgrade from console
Halt system 14) Disable Secure Shell (sshd)
Ping host 15) Restore recent configuration

Enter an option: 13

Starting the pfSense console firmware update system..

Update from a URL
Update from a local file
Q) Quit

Please select an option to continue: 2

Enter the complete path to the .tgz or .img.gz update file: /up/pfSense-2.1.4-RELEASE-4g-i386-nanobsd-upgrade.img.gz

One moment please…

Broadcast Message from admin@pg-router-5.
(no tty) at 22:54 PDT...

NanoBSD Firmware upgrade in progress...

Broadcast Message from admin@pg-router-5.
(no tty) at 22:54 PDT...

Installing /up/pfSense-2.1.4-RELEASE-4g-i386-nanobsd-upgrade.img.gz.

One moment please...

Broadcast Message from admin@pg-router-5.
(no tty) at 22:54 PDT...

NanoBSD Firmware upgrade in progress...

Broadcast Message from admin@pg-router-5.
(no tty) at 22:54 PDT...

Installing /up/pfSense-2.1.4-RELEASE-4g-i386-nanobsd-upgrade.img.gz.

...

NanoBSD Firmware upgrade is complete. Rebooting in 10 seconds.

...........Done. Rebooting...

*** Welcome to pfSense 2.1-RELEASE-nanobsd (i386) on pg-router-5 ***

mickrussom

@jimp:

.

I noticed that /boot/mbr and /boot/pmgr files are different, is that correct?

[2.1-RELEASE][admin@pg-router-5.]/boot(17): md5 mbr
MD5 (mbr) = db3f526667d01f5851ef3d0ddafb86db
[2.1-RELEASE][admin@pg-router-5.]/boot(18): md5 pmbr
MD5 (pmbr) = 6daee450f256507904e0aebe78187cf6

Also, from gpart man page (Im not sure what CORRUPT means, even after this reading)

RECOVERING
The GEOM PART class supports recovering of partition tables only for GPT.
The GPT primary metadata is stored at the beginning of the device. For
redundancy, a secondary (backup) copy of the metadata is stored at the
end of the device. As a result of having two copies, some corruption of
metadata is not fatal to the working of GPT. When the kernel detects
corrupt metadata, it marks this table as corrupt and reports the problem.
destroy and recover are the only operations allowed on corrupt tables.

If the first sector of a provider is corrupt, the kernel can not detect
GPT even if the partition table itself is not corrupt. The protective
MBR can be rewritten using the dd(1) command, to restore the ability to
detect the GPT. The copy of the protective MBR is usually located in the
/boot/pmbr file.

If one GPT header appears to be corrupt but the other copy remains
intact, the kernel will log the following:

GEOM: provider: the primary GPT table is corrupt or invalid.
GEOM: provider: using the secondary instead – recovery strongly advised.

or

GEOM: provider: the secondary GPT table is corrupt or invalid.
GEOM: provider: using the primary only -- recovery suggested.

Also gpart commands such as show, status and list will report about cor-
rupt tables.

If the size of the device has changed (e.g., volume expansion) the sec-
ondary GPT header will no longer be located in the last sector. This is
not a metadata corruption, but it is dangerous because any corruption of
the primary GPT will lead to loss of the partition table. This problem
is reported by the kernel with the message:

GEOM: provider: the secondary GPT header is not in the last LBA.

This situation can be recovered with the recover command. This command
reconstructs the corrupt metadata using known valid metadata and relo-
cates the secondary GPT to the end of the device.

NOTE: The GEOM PART class can detect the same partition table visible
through different GEOM providers, and some of them will be marked as cor-
rupt. Be careful when choosing a provider for recovery. If you choose
incorrectly you can destroy the metadata of another GEOM class, e.g.,
GEOM MIRROR or GEOM LABEL.

Any help recovering the ad0 would be cool to know.

jimp

Despite hacking and slashing at things in various ways I have yet to see any installation actually recover from this condition without reflashing the CF card (or using a new CF card)

trunix

@trunix:

jimp, I've got the same problem on a 4gb CF. Output of fdisk -p /dev/ad0:

/dev/ad0

g c7745 h16 s63
p 1 0xa5 63 3854529
p 2 0xa5 3854655 3854529
a 2
p 3 0xa5 7709184 102816

I've tried method #1 and #2, but neither worked. The output of fdisk -if /tmp/fdisk_bkup.txt /dev/ad0 from method #2 is below in case it's notable. I didn't get any errors from method #1, the system just booted back into 2.1 on the same slice. The same thing happened after method #2. I'm also not able to switch the bootup slice for whatever reason.

fdisk: WARNING line 2: number of cylinders (7745) may be out-of-range
(must be within 1-1024 for normal BIOS operation, unless the entire disk
is dedicated to FreeBSD)
******* Working on device /dev/ad0 *******

This system and CF card have been in stable operation for awhile now and I've successfully installed all the updates from 2.0.1 to 2.1. I never got a chance to install 2.1.1, I've had similar problems attempting to install 2.1.2.

I was onsite and got the opportunity to re-image the CF card for this build in mid-May to 2.1.3. Last week on a whim I decided to give the 2.1.4 update a shot. It's located a few states away so remote updates are definitely handy. I'm happy to report the update went fine, pfBlocker and the few other packages were reinstalled without issue. Whatever problem I had with 2.1 was solved with 2.1.3.

robi

Yep, one can re-flash safely the same card.

mickrussom

@jimp:

Despite hacking and slashing at things in various ways I have yet to see any installation actually recover from this condition without reflashing the CF card (or using a new CF card)

Any ideas what caused it? I was on a 2.0.x release, upped to 2.1, and then it got bonked. I guess I need a reflash - is there a howto to bootstrap the CF in another machine available so I can just goto the DC, swap and restore?

mickrussom

@jimp:

Despite hacking and slashing at things in various ways I have yet to see any installation actually recover from this condition without reflashing the CF card (or using a new CF card)

Does the replacement CF have to be 4GB, or can it be 16GB?

I have one of these:
SDCFXPS-016G

Would that work, also, how to install this form another machine .

stephenw10

You can use that card or any card bigger than 4GB. Seems like a bit of a waste though, that's an expensive CF card.

Write the Nano image to the card as described here:
https://doc.pfsense.org/index.php/Installing_pfSense#Writing_the_image

Backup your config file first remember.

Steve

mickrussom

@jimp:

Despite hacking and slashing at things in various ways I have yet to see any installation actually recover from this condition without reflashing the CF card (or using a new CF card)

Is there any way to upgrade the install in-place (kernel + userland) and just keep the corrupted labels for the time being.

jimp

@mickrussom:

@jimp:

Despite hacking and slashing at things in various ways I have yet to see any installation actually recover from this condition without reflashing the CF card (or using a new CF card)

Is there any way to upgrade the install in-place (kernel + userland) and just keep the corrupted labels for the time being.

Not any way that would be feasible/workable/supportable.

People have tried it, but it's not something I'd recommend or for which I'd provide any guidance.

luckman212

On my stuck unit I wound up just taking it apart and re-flashing the CF with a fresh 2.1.5 - problem solved.