2.4.0 -> 2.4.1 Update Failed

MiamiDan

Same here. Update failed either Web Interface or console.

jimp

That's a bit strange, that directory must exist or it wouldn't be booted at all.

Are all of your filesystems live and mounted? Check this:

: mount 
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var on /var (zfs, local, noatime, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
/dev/md0 on /var/run (ufs, local)
devfs on /var/dhcpd/dev (devfs, local)
: zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                812M  24.2G    88K  /zroot
zroot/ROOT           553M  24.2G    88K  none
zroot/ROOT/default   553M  24.2G   553M  /
zroot/tmp            424K  24.2G   424K  /tmp
zroot/var            256M  24.2G   256M  /var

And then see if the /boot/kernel directory looks normal


: ls -ld /boot
drwxr-xr-x  9 root  wheel  57 Oct 26 14:15 /boot
: ls -ld /boot/kernel
drwxr-xr-x  2 root  wheel  48 Oct 26 14:11 /boot/kernel
: ls -l /boot/kernel
total 21127
-r-xr-xr-x  1 root  wheel     46928 Oct 26 11:42 aesni.ko
-r-xr-xr-x  1 root  wheel     20200 Oct 26 11:42 alpm.ko
-r-xr-xr-x  1 root  wheel     20136 Oct 26 11:42 amdpm.ko
-r-xr-xr-x  1 root  wheel     16736 Oct 26 11:42 amdsmb.ko
-r-xr-xr-x  1 root  wheel     15656 Oct 26 11:42 amdtemp.ko
-r-xr-xr-x  1 root  wheel     19424 Oct 26 11:42 coretemp.ko
-r-xr-xr-x  1 root  wheel     32448 Oct 26 11:42 cryptodev.ko
-r-xr-xr-x  1 root  wheel     38176 Oct 26 11:42 cyapa.ko
-r-xr-xr-x  1 root  wheel    156624 Oct 26 11:42 dummynet.ko
-r-xr-xr-x  1 root  wheel     33216 Oct 26 11:42 fdescfs.ko
-r-xr-xr-x  1 root  wheel    185344 Oct 26 11:42 hwpmc.ko
-r-xr-xr-x  1 root  wheel     24864 Oct 26 11:42 ichsmb.ko
-r-xr-xr-x  1 root  wheel     26160 Oct 26 11:42 ichwd.ko
-r-xr-xr-x  1 root  wheel    118304 Oct 26 11:42 if_ena.ko
-r-xr-xr-x  1 root  wheel     17920 Oct 26 11:42 if_ic.ko
-r-xr-xr-x  1 root  wheel    341176 Oct 26 11:42 if_ix.ko
-r-xr-xr-x  1 root  wheel    281856 Oct 26 11:42 if_ixv.ko
-r-xr-xr-x  1 root  wheel     35704 Oct 26 11:42 if_stf.ko
-r-xr-xr-x  1 root  wheel     32424 Oct 26 11:42 ig4.ko
-r-xr-xr-x  1 root  wheel     16688 Oct 26 11:42 iic.ko
-r-xr-xr-x  1 root  wheel     20912 Oct 26 11:42 iicbb.ko
-r-xr-xr-x  1 root  wheel     27416 Oct 26 11:42 iicbus.ko
-r-xr-xr-x  1 root  wheel     14976 Oct 26 11:42 iicsmb.ko
-r-xr-xr-x  1 root  wheel     23824 Oct 26 11:42 intpm.ko
-r-xr-xr-x  1 root  wheel     33984 Oct 26 11:42 ipdivert.ko
-r-xr-xr-x  1 root  wheel    279152 Oct 26 11:42 ipfw.ko
-r-xr-xr-x  1 root  wheel     66784 Oct 26 11:42 ipmi.ko
-r-xr-xr-x  1 root  wheel     12952 Oct 26 11:42 ipmi_linux.ko
-r-xr-xr-x  1 root  wheel     14104 Oct 26 11:42 isl.ko
-r-xr-xr-x  1 root  wheel     24984 Oct 26 11:42 ismt.ko
-r-xr-xr-x  1 root  wheel     10032 Oct 26 11:42 jedec_ts.ko
-r-xr-xr-x  1 root  wheel  18232207 Oct 26 11:42 kernel.gz
-rw-r--r--  1 root  wheel    119948 Oct 26 11:42 linker.hints
-r-xr-xr-x  1 root  wheel     13776 Oct 26 11:42 lpbb.ko
-r-xr-xr-x  1 root  wheel    222568 Oct 26 11:42 ndis.ko
-r-xr-xr-x  1 root  wheel     21544 Oct 26 11:42 nfsmb.ko
-r-xr-xr-x  1 root  wheel     20040 Oct 26 11:42 nmdm.ko
-r-xr-xr-x  1 root  wheel     23440 Oct 26 11:42 opensolaris.ko
-r-xr-xr-x  1 root  wheel     11864 Oct 26 11:42 pcf.ko
-r-xr-xr-x  1 root  wheel   1589176 Oct 26 11:42 sfxge.ko
-r-xr-xr-x  1 root  wheel     15584 Oct 26 11:42 smb.ko
-r-xr-xr-x  1 root  wheel     11888 Oct 26 11:42 smbus.ko
-r-xr-xr-x  1 root  wheel     55672 Oct 26 11:42 udf.ko
-r-xr-xr-x  1 root  wheel     32512 Oct 26 11:42 viapm.ko
-r-xr-xr-x  1 root  wheel    325512 Oct 26 11:42 vmm.ko
-r-xr-xr-x  1 root  wheel   2544056 Oct 26 11:42 zfs.ko

And lastly see what the kernel package looks like. This is from a 2.4.2 box but 2.4.1 should be similar

: pkg info pfSense-kernel-pfSense
pfSense-kernel-pfSense-2.4.2.a.20171026.1037
Name           : pfSense-kernel-pfSense
Version        : 2.4.2.a.20171026.1037
Installed on   : Thu Oct 26 14:11:49 2017 EDT
Origin         : security/pfSense-kernel
Architecture   : FreeBSD:11:amd64
Prefix         : /
Categories     : security
Licenses       : APACHE20
Maintainer     : development@pfsense.org
WWW            : https://www.pfsense.org/
Comment        : pfSense kernel (pfSense)
Annotations    :
	repo_type      : binary
	repository     : pfSense-core
Flat size      : 24.1MiB
Description    :
pfSense kernel (pfSense)

WWW: https://www.pfsense.org/

rmkilc

She boots just fine. My hardware is an APU1D with an mSATA SSD if that helps.

I have the same filesystems.


: mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var on /var (zfs, local, noatime, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
/dev/md0 on /var/run (ufs, local)
devfs on /var/dhcpd/dev (devfs, local)
: zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot                480M  4.71G    88K  /zroot
zroot/ROOT           397M  4.71G    88K  none
zroot/ROOT/default   397M  4.71G   397M  /
zroot/tmp            264K  4.71G   264K  /tmp
zroot/var           80.8M  4.71G  80.8M  /var

Well this is different. Why would it be like this? I did a normal 2.4.0 install.


: ls -ld /boot
lrwxr-xr-x  1 root  wheel  13 Oct 10 05:04 /boot -> bootpool/boot
: ls -ld /boot/kernel
ls: /boot/kernel: No such file or directory
ls -l /boot/kernel
ls: /boot/kernel: No such file or directory

Looks similar.


: pkg info pfSense-kernel-pfSense
pfSense-kernel-pfSense-2.4.0
Name           : pfSense-kernel-pfSense
Version        : 2.4.0
Installed on   : Tue Oct 10 05:04:47 2017 PDT
Origin         : security/pfSense-kernel
Architecture   : FreeBSD:11:amd64
Prefix         : /
Categories     : security
Licenses       : APACHE20
Maintainer     : development@pfsense.org
WWW            : https://www.pfsense.org/
Comment        : pfSense kernel (pfSense)
Annotations    :
Flat size      : 23.9MiB
Description    :
pfSense kernel (pfSense)

WWW: https://www.pfsense.org/

jimp

Do you happen to remember which ZFS partition scheme you selected (e.g. GPT, BIOS, etc)?

rmkilc

Yes, GPT and BIOS. Should I have chosen a different partition scheme? As for BIOS vs UEFI, I just assumed my hardware didn't support UEFI.

jimp

No, that's the same one I used on that box, I'm fairly certain. I spot checked a few systems here running ZFS and I don't see that on any of them. One of them is a single disk the other is a mirror.

What other ZFS options did you choose? Do you remember?

jimp

I don't have any setup for it, but it looks like maybe that can end up that way with GELI encryption for the disk

rmkilc

You might have to refresh my memory on what options there were. I deleted the swap partition if I remember correctly. I didn't enable encryption.

jimp

Looks like MBR can also result in a bootpool (looking at the source, haven't tested it, no time today)

What does "zpool list" show? Do you see bootpool there?

rmkilc

: zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  5.34G   481M  4.87G         -     8%     8%  1.00x  ONLINE  -

jimp

I finally had some time to look into this today and it turns out that in those cases (Using MBR or geli), the zfs pool called "bootpool" isn't getting imported due to a fun chicken-and-egg scenario with the zfs cache file. The zfs pool cache tells the OS what pools to mount, but that cache file is in /boot/, so it doesn't get read at boot time because the file isn't there.

https://redmine.pfsense.org/issues/8063

I just pushed a fix. Once that is in snapshots you can upgrade to a 2.4.2 snapshot, you will need to run "zpool import bootpool" first and then it should proceed from there.

rmkilc

Am I running with install options that are untested and/or in beta? I just want to be able to click the update button and have it update without issues every time. What do you recommend? This isn't the first time I have had updating issues.

jimp

ZFS support as a whole is new and still considered experimental. People are excited about it, but it's not something we recommend yet. That said, it works fine for the majority of those using it.

Even more rare are those that need MBR support in ZFS, most people that can run 2.4 can run GPT which does not have this problem.

ZFS Encryption is also new and rarely used, and not practical to test since it requires a lot of manual intervention and direct console access.

If you install using the default ZFS options (which is GPT, no encryption) then you'll be OK.

rmkilc

Like I said before, the options I used were GPT and no encryption.

jimp

If you ended up with /bootpool, that cannot be true. It only gets activated for MBR or encrypted drives in the installer code.

rmkilc

Is there a way to verify without reinstalling?

jimp

gpart list | grep scheme

If it says MBR/BSD then it's MBR, if it's says GPT/GPT then it's GPT

Either way though, once the bootpool fix is in snapshots it will be fine no matter what you chose.

rmkilc

Ok, you are right, it's MBR. I had installed it twice and the first go-around I had chosen GPT. I thought I had the second time too but I guess not.