Upgrade to 2.2.4 failed
-
Error on boot:
"can't find
kernel"See screenshot….
-
Don't see how that could happen outside a hardware failure, like sometimes a dead drive doesn't show as such until you reboot. But that looks like a Xen VM, in which case that seems unlikely unless the host OS is having issues.
-
I got the same error when upgrading 2.2.3 to 2.2.4 inside a Hyper-V VM. Since the machine is in my test environment I went to just wipe and re-set it up. The VM had a 1GB drive and it failed to install with something about not enough space for the swap partition (can't remember the exact message). I remade the drive as a 2GB drive and it installed without issue.
If I get time I'll try and setup another 2.2.3 box and try the upgrade again.
-
We've seen that happen a number of times with VMs that run out of disk space on upgrade. Provisioning a VM with only a 1GB disk is asking for trouble…
The 2.2.4 OS image is a few MB larger than previous versions so it will take a little more room, you're probably just lucky it never happened before.
-
ok so I should be fine in production as it's already 2GB in size. Is there a recommended minimum disk size?
-
Ah yes, forgot about the running out of disk space possibility when I posted yesterday. How much free disk space did you have pre-upgrade? We have a ticket open to add a check there to not try to upgrade if there isn't enough free space.
-
no idea how much free space there was as the machine was dead and is now rebuilt. The production machine that I have still to update only has 700MB available. Will that 700MB be enough?
-
Well, as said above, if upgrading with one GB space left is asking for trouble, then guess what "700 MB" will be ….
-
Well, as said above, if upgrading with one GB space left is asking for trouble, then guess what "700 MB" will be ….
@jimp said "a 1GB disk" with no mention of free space. Sounds like we'll have a free space check next time.
-
the upgrade with 700MB free space went without issue.
-
– Update 1: Problem identified --
/boot/kernel
/boot/
– Update 2: I got it working, details below --
-
I booted off the LiveCD in to Recovery mode.
-
I saw the OS partition I run on mounted under /tmp/rescue
-
Within my OS partition, I found I had several full system backups from previous upgrades, the latest of which likely tipped the balance and left the drive with very little free space. I deleted about 10GB of older backup images.
-
Since my LiveCD was the exact version of pfsense I was upgrading to, I copied its /boot/kernel/kernel.gz to /tmp/rescue/boot/kernel/
-
I made note that I could come back in to /tmp/rescue and grab my full system backup and config.xml if all this didn't work out.
-
I rebooted. The updated pfsense install booted properly, all appears well. So ultimately I was at most 13MB short (kernel.gz size) for the upgrade to work on its own.
So I guess the moral of the story is those full system backups can kill ya. I'm inclined to recall the wisdom of running with many well defined partition sizes for core functionality and having transient data exist safely on other volumes. Suffice to say, I'm glad I found a solution. Hopefully these notes help someone else.
– Update 3: Needed a little more --
Yes Update 2 above got me up and running again, but many things weren't working at all, my webui was down, I couldn't ssh in, etc. Many things were also working fine, like dhcp. From the console on the router I invoked a console upgrade, thankfully it didn't seem to care about versions it just ran the script. After that, now I think I'm 100%.
-
-
Reinstall frersh (don't upgrade) and restore your config.xml. Should take under 5 minutes.
-
Reinstall frersh (don't upgrade) and restore your config.xml. Should take under 5 minutes.
I bet config.xml doesn't capture minor under-the-hood changes I've made: Adding a secondary dedicated hard drive for a squid cache.
Generally speaking, I think your advice is really good though. Upgrading is generally inherently tricky.
-
Nope. Upgrading in the first place will clobber most of that anyway. Pretty much on you to have a log of the out-of-gui changes you have made so you can duplicate them on a reinstall.
-
Nope. Upgrading in the first place will clobber most of that anyway. Pretty much on you to have a log of the out-of-gui changes you have made so you can duplicate them on a reinstall.
Yup, fair enough for the cowboy that goes off-road, such as myself. I wanted to see if I could salvage the situation without blowing the escape hatch, it's been a couple days, so far so good. :)
