Upgrade from 2.2.6 to 2.3 Release fails - can't find kernel, can't mount filesys



  • Hi,

    just tried to upgrade my running 2.2.6 pfsense Box (Amd Sempron 200U, 1 GB DDR2).
    After rebooting it failed too boot complaining no kernel found….

    Then I tried do boot by (USB) CD-Rom and the next problem occured... neither the compact flash IDE drive, nor the cd drive was mountable - even mounting in manually
    ended with a "error 19"

    Mounting from ufs:/dev/ada0xxx failed with error 19
    mountroot>

    a ? did only show the cd rom...

    Now I am trying to reinstall 2.2.6 again - seems to work fine, now...

    No Idea what 2.3 does there...?

    After dozens of flawless updates, now this..:-(

    Any Ideas?

    Cheers,

    4920441



  • Check BIOS / DMA settings for the compact flash IDE drive.



  • Hi,

    I'll try that.. but that does absolutely doesn't explain why it also fails too boot the installer from an USB CDrom….

    When I am reading the Upgrade problems to Version 2.3 in the forum, I think I stick with the 2.2.6 release for  a couple of weeks... What a bummer - the first upgrade since 1.2 to 2.x which didn't work at all...:-( (at least for my system)

    Cheers,

    4920441


  • Rebel Alliance Developer Netgate

    What is the exact, specific error?

    If it cannot load the kernel, it's possible it ran out of disk space during the upgrade. Safest and quickest fix is to reinstall.



  • Hello!
    Have the same error, after the upgrading on first boot:
    kernel not found
    can't load kernel

    pfSense installed on ProxmoX

    How I can fix this problem?



  • I don't get it, people. When you see such errors, what's the first thing you can think about, to fix it???
    Isn't that if it's a physical machine, re-format and re-install? Or if it's a VM, just simply create a new VM with the same params and re-install there from ISO? Or revert from snapshot if you have one?
    Is it worth the time to ask in a forum????



  • Great idea:
    reinstall & configure…update
    reinstall & configure...update
    reinstall & configure...update

    Maybe updates must be more stable?!


  • Rebel Alliance Developer Netgate

    Without more info about why it happened, we can't address the issue. Every other time we've seen this happen it was because the disk was filled during the update (so it was installed on a too-small disk or allocated too little VM disk space, etc). That will be better from 2.3 forward because of how the updates work.

    If someone keeps installing to the same too-small disk, it's quite possible they could keep shooting their foot in the same way every time…



  • @BlackDragon381:

    Great idea:
    reinstall & configure…update
    reinstall & configure...update
    reinstall & configure...update

    Honestly? Yes!

    I never update in-place. I always do a fresh install and restore file from backup. On a separate hardware, and when assuring 100% is done, swap the cables. On a VM, it's even more easier.



  • I'm seeing something similar when attempting to upgrade from 2.2.6 amd64 running under Hyper-V 2012 R2.  I removed all packages from pfSense prior to the upgrade.  I had 4.9 GB of free HDD space before performing the upgrade.

    Nothing was output to the console during the upgrade process (other than the standard "firmware upgrade in process") until the end of the process, just before the machine reboots.  The first two screenshots reflect the errors.  The third screenshot is after the machine reboots and can no longer mount the filesystem.

    Hope it helps.







  • Rebel Alliance Developer Netgate

    @Boolah:

    I'm seeing something similar when attempting to upgrade from 2.2.6 amd64 running under Hyper-V 2012 R2.  I removed all packages from pfSense prior to the upgrade.  I had 4.9 GB of free HDD space before performing the upgrade.

    Nothing was output to the console during the upgrade process (other than the standard "firmware upgrade in process") until the end of the process, just before the machine reboots.  The first two screenshots reflect the errors.  The third screenshot is after the machine reboots and can no longer mount the filesystem.

    First two errors are normal, on 2.2 as the VM shuts down it (briefly) tries to load 2.3's PHP files on 2.2, which won't work, but after reboot it would be fine)

    Last one is different. That may be an issue with HyperV's disk controller setting. There's another thread for that already ( can't find the link at the moment ), likely related to https://forum.pfsense.org/index.php?topic=109952.0 – at least one report in another thread said they had to recreate the VM, but if that's the case, it's possible just fixing the disk controller settings in HyperV may be sufficient. I don't have any Hyper-V that I can test on myself, so I can't offer any more concrete suggestions for that situation.



  • @jimp:

    Without more info about why it happened, we can't address the issue. Every other time we've seen this happen it was because the disk was filled during the update (so it was installed on a too-small disk or allocated too little VM disk space, etc). That will be better from 2.3 forward because of how the updates work.

    If someone keeps installing to the same too-small disk, it's quite possible they could keep shooting their foot in the same way every time…

    Have about 5GiB free space before performing the upgrade (10GiB on VM, no additional packages).
    What information do you need?
    And how can I get it ?


  • Rebel Alliance Developer Netgate

    In general, "Mounting from ufs:/dev/ <mumble>failed with error 19" means the OS tried to mount the disk from that specific device, but the device wasn't there.

    Why wasn't it there? Two main potential answers:

    1. The disk controller driver didn't attach to the hardware. Rare that it would happen on a new version and not an old, but if for some reason a driver was dropped, it could happen. Unlikely here, since it's bare metal and was using AHCI, but still possible.

    2. The disk changed locations and is no longer at "/dev/<mumble>". This isn't a factor on modern installs since we go by ufs ID and not a specific device name, but if you upgraded over the years and the driver name changed, it could hit that. If you can find the right device name and enter it at the prompt, it can then be made to work by manually editing /etc/fstab. A reinstall is probably quicker if you don't know your way around FreeBSD though.</mumble></mumble>


  • Rebel Alliance Developer Netgate

    @BlackDragon381:

    @jimp:

    Without more info about why it happened, we can't address the issue. Every other time we've seen this happen it was because the disk was filled during the update (so it was installed on a too-small disk or allocated too little VM disk space, etc). That will be better from 2.3 forward because of how the updates work.

    If someone keeps installing to the same too-small disk, it's quite possible they could keep shooting their foot in the same way every time…

    Have about 5GiB free space before performing the upgrade (10GiB on VM, no additional packages).
    What information do you need?
    And how can I get it ?

    If it's a VM, probably not the same problem as the OP in this thread, there are other threads for VM issues (especially Hyper-V), you're better off moving there.



  • @jimp:

    If it's a VM, probably not the same problem as the OP in this thread, there are other threads for VM issues (especially Hyper-V), you're better off moving there.

    Actually, your second point in the above post (https://forum.pfsense.org/index.php?topic=109802.msg612250#msg612250) solves the issue.  pfSense v2.3 was trying to load the root from /dev/ada0s1a, when it should have been using /dev/da0s1a.  Entering "ufs:/dev/da0s1a" at the mountroot> prompt boots v2.3 correctly.  After the booting, /etc/fstab shows it's trying to mount the device directly, not by ufs id.  Editing the fstab and rebooting solves the issue and allows the machine to reboot in the future without issue.

    Also, the two posts linked in the Hyper-V thread both say the issue is with Hyper-V 2008 R2 and is fixed in Hyper-V 2012.  I'm running Hyper-V 2012 R2, so don't believe they're related.


  • Rebel Alliance Developer Netgate

    @Boolah:

    @jimp:

    If it's a VM, probably not the same problem as the OP in this thread, there are other threads for VM issues (especially Hyper-V), you're better off moving there.

    Actually, your second point in the above post (https://forum.pfsense.org/index.php?topic=109802.msg612250#msg612250) solves the issue.  pfSense v2.3 was trying to load the root from /dev/ada0s1a, when it should have been using /dev/da0s1a.  Entering "ufs:/dev/da0s1a" at the mountroot> prompt boots v2.3 correctly.  After the booting, /etc/fstab shows it's trying to mount the device directly, not by ufs id.  Editing the fstab and rebooting solves the issue and allows the machine to reboot in the future without issue.

    Also, the two posts linked in the Hyper-V thread both say the issue is with Hyper-V 2008 R2 and is fixed in Hyper-V 2012.  I'm running Hyper-V 2012 R2, so don't believe they're related.

    Great! Once you have a correct /etc/fstab that works after rebooting, you can run /usr/local/sbin/ufslabels.sh to permanently adjust fstab to use UFS labels rather than the device name



  • @jimp:

    Great! Once you have a correct /etc/fstab that works after rebooting, you can run /usr/local/sbin/ufslabels.sh to permanently adjust fstab to use UFS labels rather than the device name

    Out of the dozen or so pfSense v2.2 instances I manage (most are physical, not virtual), all but one are using device references in their fstab, rather than UFS labels.  Running /usr/local/sbin/ufslabels.sh and rebooting prior to upgrading to v2.3 seems to obviate this issue entirely.

    Note: I've only tried this on one VM that was failing to upgrade correctly, but upgraded without issue by running /usr/local/sbin/ufslabels.sh and rebooting prior to attempting to upgrade to v2.3.  Perhaps after someone else can confirm this, the pfSense v2.3 Upgrade Guide (https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.3_Upgrade_Guide) could be updated to reflect this?


  • Rebel Alliance Developer Netgate

    It's already mentioned on that page, under 2.2, but perhaps needs repeated ( https://doc.pfsense.org/index.php/UpgradeGuide#Disk_Driver_Changes )



  • @jimp:

    Great! Once you have a correct /etc/fstab that works after rebooting, you can run /usr/local/sbin/ufslabels.sh to permanently adjust fstab to use UFS labels rather than the device name

    How can I edit fstab?

    Now I have in /etc/fstab
    /dev/vtbd0s1a for /
    /dev/vtbd0s1b for swap
    and in /dev/
    ad0s1a
    ad0s1
    and no da0s1a

    And how to run /usr/local/sbin/ufslabels.sh?










  • @Boolah:

    Actually, your second point in the above post (https://forum.pfsense.org/index.php?topic=109802.msg612250#msg612250) solves the issue.  pfSense v2.3 was trying to load the root from /dev/ada0s1a, when it should have been using /dev/da0s1a.  Entering "ufs:/dev/da0s1a" at the mountroot> prompt boots v2.3 correctly.  After the booting, /etc/fstab shows it's trying to mount the device directly, not by ufs id.  Editing the fstab and rebooting solves the issue and allows the machine to reboot in the future without issue.

    Also, the two posts linked in the Hyper-V thread both say the issue is with Hyper-V 2008 R2 and is fixed in Hyper-V 2012.  I'm running Hyper-V 2012 R2, so don't believe they're related.

    Just to confirm I had exactly the same problem on my HyperV 2012R2 pfSense VM. It's also a pretty old install so was using device names rather than the UFS label.
    After a brief panic I got round it in exactly the same way, will take a look at ufslabels.sh later.



  • @BlackDragon381:

    How can I edit fstab?

    Somebody can help me?
    There is no editor in this shell!


  • Rebel Alliance Developer Netgate

    vi and ee both exist in the shell, and both can be used.

    If they won't run, try using the full path: /usr/bin/vi or /usr/bin/ee



  • Mounting from ufs:/dev/ada0xxx failed with error 19
    mountroot>

    IS this a 1 GB CF Card? Then you should go with a 2 GB or 4 GB since the version 2.3 is no more for 1 GB
    CF Cards!

    a ? did only show the cd rom…

    And on the other side with 1 GB RAM you could be easily run out of RAM space or in other words
    1 GB RAM is perhaps to small! Insert as a minimum 2 GB or better 4 GB if needed and pending on the
    the installed packets or used services.

    Now I am trying to reinstall 2.2.6 again - seems to work fine, now…

    Could be a short work around until the version 2.3.1 is out, but then perhaps read once more the lines above
    with 1 GB CF Card and 1 GB RAM it would be perhaps underpowered or not right sorted.

    No Idea what 2.3 does there…?

    Is this a 32Bit or 64Bit install?

    After dozens of flawless updates, now this..:-(

    There will be each version one day where the user or customer should be thinking on to do something
    by its own intense because his hardware could be outdated at one day. So if you are using a 1 GB CF card
    change it against a 2 GB or 4 GB model, the 1 GB CF Cards are to small, and if then all other things must be
    running in the RAM als the 1 GB of RAM is to small too.



  • @jimp:

    vi and ee both exist in the shell, and both can be used.

    If they won't run, try using the full path: /usr/bin/vi or /usr/bin/ee

    How do I use the editor to edit the file /etc/fstab?
    I can't open vi or ee, see attachment!




  • Sounds almost like you don't have vi installed on your box.
    It should come in from a standard install.

    What happens if you type:

    ls -al /usr/bin/vi

    At a command prompt?

    Are you running a bare metal or VM install?
    How did you load pfSense?



  • @divsys:

    Sounds almost like you don't have vi installed on your box.
    It should come in from a standard install.

    What happens if you type:

    ls -al /usr/bin/vi

    At a command prompt?

    Are you running a bare metal or VM install?
    How did you load pfSense?

    illegal option – a

    My pfSense broken after upgrading to 2.3 via gui.
    System can't find kernel (see attachment).
    PfSense installed on Proxmox (kvm).




  • That's the boot loader prompt, not the single user mode shell.



  • Sorry, I looked back through the thread and realized your not getting a properly mounted system drive, but you're getting farther than the OP in this thread.

    Normally you need to mount the required disk and prove that the pfSense install is up and running before making any of those fstab changes.

    As far as the "ls" command, try it without the "-al" -> "ls /usr/bin/vi".

    You may be able to recover your install, but it's probably far easier just to reinstall and reload a config backup.



  • @kpa:

    That's the boot loader prompt, not the single user mode shell.

    How enter to user mode shell?



  • @BlackDragon381:

    @kpa:

    That's the boot loader prompt, not the single user mode shell.

    How enter to user mode shell?

    https://doc.pfsense.org/index.php/Locked_out_of_the_WebGUI

    That is for a different problem but it documents how to get to the single user mode shell.