Encrypted ZFS no keyboard during boot
-
I bought a Zotac CI323 to build my firewall with pfSense. I installed it with UFS initially, did the config and got everything working properly. During boot and setup, the keyboard works fine.
I hadn't noticed though that 2.4.0 supported ZFS and I very much prefer to use that so I can more easily make use of an encrypted FS, make snapshots and do (encrypted) ZFS send/receive to my ZFS fileserver over SSH.
I made a config backup and downloaded it. I reinstalled pfSense, chose ZFS, encrypted and typed in the passphrase. All went well and it rebooted. It asked for the password to mount the encrypted partition, and the keyboard was completely dead.
I pulled the power and plugged it back in, and it asked for the early boot password. This worked but after that it wanted to mount and it got stuck again. No keyboard.
So I figured I reinstall it again without encryption but during the USB thumb drive boot (pfsense installer), it also asked for the password and got stuck again.
I ended up booting from my Arch Linux thumb drive and wipe the internal SSD. The reinstall of pfSense worked and its running on ZFS but without encryption.
So, basically, USB keyboard works from all ports (2.0 and 3.0) except when I choose encryption, then it gets stuck on the mount part. Could it be that it's not a USB issue (unplugging and replugging the keyboard didn't trigger any console messages, nor do the leds light up). I did notice that all keyboard LEDs light up once during boot up of pfSense (I dont mean during system poweron or POST) after which it appears dead.
Last night I was unprepared and this problem caught me by surprise. I have a 14-year old internet monitoring system walking around the house (ie, my son) who reminds me every couple of seconds that the internet is down and asking if I'm done yet. I'd like to give it another try this week but be a little better prepared. I am not familiar with BSD, I'm more of a Linux guy ;).
Maybe this is a familiar issue (I searched but didn't find anything) or someone can give me some handles on how to troubleshoot this?
Can I use boot parameters to make it boot to console without attempting to mount ZFS? Single user didn't work. I assume the early password unlocks the efi/boot partition, the second one is for the root partition? The passphrase is the same so I'm not sure why it is asking for it twice?
Any help is appreciated. Thanks!
PS. I just tried the same in a VM under Virtualbox and what happened is that the first prompt after the install did not accept the passphrase. keyboard however worked. I shut down the VM, rebooted it manually and it asked for the passphrase pre-boot. This was the only prompt and pfSense booted and ran fine. So my Zotac box had something wrong in that it asked for a passphrase twice. The second prompt looked the same as the first prompt in the VM right after the installation.
-
Really no one has any suggestions on the keyboard issue? :-\
-
Really no one has any suggestions on the keyboard issue? :-\
I feel your pain on the crickets. Every now and then threads will get no traction, and it's just the OP going "Anyone? Help?" I've been in those shoes a few times, including a thread I've had the past couple months with no bites. Yarrrgh. :)
I don't have Zotac experience, nor have I run into your exact issue, so I can only shoot blindly in the dark.
I know back years ago when I was tinkering with FreeNAS (based on FreeBSD), that early version of FreeBSD did not like USB 3.0 at all, so I had to go into the Supermicro BIOS and disable all USB 3.0 ports to make them behave like USB 2.0 just to play it safe.
In your issue, you say the keyboard works during POST (meaning, can you get into the BIOS settings with the keyboard?) but then the keyboard stops working once the pfSense 2.4.0/FreeBSD GELI prompt appears? And you also say there's an early boot password: is that one of those passwords you set up in BIOS? I actually have that on many of my laptops: 1) Power-on BIOS password then the Full Disk Encryption password prompt, THEN the Windows password prompt. I assume the keyboard works so you can enter the BIOS boot password before the bootloader kicks in…
So none of the LEDs respond on the keyboard, Caps Lock, Num Lock, etc.? Have you tried different keyboards and also different USB ports? Have you disabled USB 3.0 in the Zotac BIOS? Is the BIOS on the latest version? I'm just grasping at straws here.
I'm tinkering with 2.4.0-RC in VirtualBox as well, including GELI Full Disk Encryption for funsies. It works just fine, but I realize this is apples and oranges to your baremetal issue with the Zotac.
Good luck, and I feel ya on the silence.
-
The number of people running ZFS on 2.4 with encryption is still very, very low. You're in what is still uncharted territory, so don't be surprised at a lack of response.
This is a case where, because it's all the OS at that point (FreeBSD), you'll have much better luck searching for the problem as it pertains to FreeBSD. There are several promising leads there, such as:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202703
https://forums.freebsd.org/threads/40965/
http://www.schmidp.com/2014/01/07/zfs-full-disk-encryption-with-freebsd-10-part-2/ -
Thanks both of you for the reply and willingness, appreciate it!
@Finger79: I'm doing more troubleshooting this weekend and I will document my steps and make pictures if I have to. THe first prompt was the bootloader password prompt, which worked. Keyboard continues to work during BSD boot until it tries to mount zroot at which point it asks for another password. At this point, the keyboard is completely dead, no numlock, no control-alt-delete. Pulling the keyboard and putting it back in (in another usb port) doesn't trigger any console messages stating a new usb device has been detected. At this point, only pulling the power and plugging it back in will get the box back to life.
I tried it in Virtualbox as well and it works fine. Difference is that in my case it only asks for a password once, during bootloader phase, no other password prompt popped up while on the Zotac, it asks for it twice.
@Jimp: Thanks! I think this may actually solve my issue. Way down in one of those threads you linked there is this:
I found a setting called "Port 60/64 Emulation which when disabled allowed me to enter the password for encryption. (Woo-hoo!)
The description in the BIOS for this setting is:
"Enables I/O Port 60h/64h emulation support"
When enabled, the BIOS will emulate I/O ports 64h and 60h for your USB keyboard and mouse. This enables PS/2 functionality like keyboard lock, password setting and scan code selection.
The option is present on my system and enabled by default.
I just booted from the installer memstick but it doesn't even boot up at the moment: it gets to the part where it says re0 link is up, re1 link is up and then nothing. It just hangs. Ctrl-alt-del makes it shutdown and reboot though. This is regardless of whether I enable or disable port 60/64 emulation so unrelated. I guess it's something in the latest installer because the previous one didn't do that. Or, its because its now running ZFS on the internal SSD and the installer somehow doesn't like that. Right before the interface messages there is something about ada0, forgot the exact message but I'm sure it has to do with how ZFS partitions (I've seen something similar on Linux with ZFS).
Anyway, no time to fiddle with it any further tonight, so this weekend I'll take some extra time and troubleshoot this further. I'll make notes and pictures.
Again, thanks for the help guys, I hadn't thought of searching for generic BSD problems, I simply assumed it was a pfSense thingy and I have no idea how much you actually change (or didn't :P) on a BSD level, kernel, modules, etc.
To be continued! It would be totally awesome to get ZFS with native encryption working! ZFS is awesome with snapshots, bookmarks and zfs send/receive for offbox backups of the entire system.
-
I experienced intermittend problems between attempts to setup zfs with encryption. Turns out that in my case Port 60/64 Emulation enabled/disabled doesn't really make any difference.
The GELI prompt actually accepts anything, even no passphrase and the system will attempt to boot. Just because it boots doesn't mean the passphrase is correct. I previously assumed this was the case but turns out it is not.
Using a simple passphrase - test123
The install completes and it reboots. On the GELI prompt, type test123 and the boot sequence completes without a hitch. I could setup interfaces, access it remotely and restore a backup. I could reboot, type the passphrase and it reboots fine. Tried this 4 times with no failures.However, when typing the wrong passphrase, it would hang, see attachment random-unblock jpeg. At this point, nothing works, no capslock, numlock, ctrl-alt-del. I had to pull the power.
On poweron, when entering the wrong passphrase again, and continually hitting enter or random keys, it would actually prompt for a new passphrase after attempting to mount zroot. This behavioris consistent with the reports JimP linked to (see comment 1). However, no passphrase is accepted, all come back wrong (contrary to that support thread). I tried typing slow, fast, really slow, nothing matters. All wrong.
During reinstallation
When doing a reinstallation on a system with an encrypted filesystem, the installer attempts to mount and unlock it. It will ask for the passphrase in the same manner (not the GELI prompt at boot, but when trying to mount zroot). Again, no passphrase is accepted. Remember this is still the simple test123 passphrase.Maybe you want to reconsider this behavior. Attempting to mount would only be required during an upgrade, not a reinstallation for which you're going to wipe the disk anyway. There are 4 or 5 mountpoints and each will ask for a passphrase 3 times with long delays in between, this is quite annoying.
More complex passphrase - Remote Paper Engine Capacitor
This is where I got weird and intermittend results.The problem is that I would randomly get the "random-unblock" issue (see attachment) as if I had mistyped my passphrase. Obviously I might have mistyped the passphrase, which is not unlikely given the fact that it's long with caps. But what should also be obvious is that I make a real good attempt at typing it correctly. Yet sometimes it would boot through, other times it wouldn't. On the GELI prompt you can see the rotating slash and I watched it on each keystroke. Still it would hang sometimes. Rattling on the keyboard would popup the passphrase prompt during zroot mount attempt but again, nothing is accepted there.
So not sure what's happening here, its weird. I tried 10 times and it failed 3 or 4 times. I was keeping count but that failed too haha ::). Now, anything is possible and I won't claim I couldn't have typed it wrong 4 times but there is no way to check, really. I'd be disappointed in myself though if I can't type a password correctly with more than 60% accuracy ;D
Final thoughts
During all the reinstallations, I restored my backup each time. After reinstallation, there is an option to restore an xml. This is useless at this point since the system is empty and the autobackuprestore requiring a subscription is unconfigured. What would be really nice is if this option from the console menu allowed me to type in my subscription details and, obviouisly if internet is active, is able to restore from an online backup.As a fallback restore option, it would be really nice to allow me to choose a file to restore from (show mounted usb devices and their base path). I had a usb thumb drive with an xml file I couldn't access that. So each install I had to go through setting up of the interfaces and ip config manually, to access the web interface and restore from a downloaded xml file.
Then, there is what looks like a cosmetic issue, I get an error during boot about kldload no such file or directory (see 2nd attachment). I saw that without encryption too. Not sure when running ufs, though.
Last but not least, system is finally running on a zfs encrypted filesystem 8). I can start screwing around with encrypted zfs send/receives to my zfs fileserver which is also running some encrypted datasets since this week ;D.
If there are any question, please fire away. If you need me to test something I can do that too.
-
I feel like a dumb ass :-[.
On my server:
[code]me@server ~ $ zpool get all | grep encr
backup feature@encryption active local
data feature@encryption enabled localme@server ~ $ zfs get all | grep encr|grep -v @ backup encryption off default backup/blackhole encryption aes-256-ccm - backup/blackhole encryptionroot backup/blackhole -On my router:
[2.4.0-RC][admin@router.local]/root: zpool get all|grep encr [2.4.0-RC][admin@router.local]/root: zfs get all | grep encr [2.4.0-RC][admin@router.local]/root:In other words, nothing is returned. ZFS is not using encryption. It doesn't even support encryption:
[2.4.0-RC][admin@router.local]/root: zfs create -o encryption=on -o keyformat=passphrase zroot/test cannot create 'zroot/test': invalid property 'encryption'Not blaming anyone, clearly my FreeBSD knowledge and the feature set ZFS supports on this platform is lacking. I totally assumed encryption was supported on FreeBSD.
My whole purpose was to use native ZFS encrypted send/receive :-.
-
On FreeBSD it uses geli(8).
-
Now you tellin' me haha ;D ;)
It's cool though, just need to figure out whether I want to keep it like this or not.
-
Yes, the native encryption is not in the OpenZFS inplementation (yet), at the moment it's only in Solaris and of course it's closed source because Oracle sits on top of it.
The main problem with GELI and the keyboard input during the password entry time is that it relies on the BIOS/UEFI PS/2 keyboard emulation when a USB keyboard is used, on many systems this emulation is completely broken for other operating systems than MS Windows and nobody has been able to come up with fixes for FreeBSD to make the emulation work on the problematic systems.
-
@kpa:
Yes, the native encryption is not in the OpenZFS inplementation (yet), at the moment it's only in Solaris and of course it's closed source because Oracle sits on top of it.
The main problem with GELI and the keyboard input during the password entry time is that it relies on the BIOS/UEFI PS/2 keyboard emulation when a USB keyboard is used, on many systems this emulation is completely broken for other operating systems than MS Windows and nobody has been able to come up with fixes for FreeBSD to make the emulation work on the problematic systems.
Thanks for the extra info.
I decided to remove encryption and go with just ZFS. I can still do zfs send/recv which is actually the more important feature of the two.
Offtopic, but with all the layoffs at Oracle completely cutting away the Solaris and ZFS departments, maybe (I know, wishful thinking) they will release the code to the community?
