Hyper-V
-
@johnkeates:
I do wonder if we could get around some of this by chainloading pfSense, I've seen people loading OpenFirmware, TianoCore, OVMF and with some hacks even SeaBIOS on top of existing firmwares.
Well, that would be too much diy toying for me. Not my style.
And since that forked sense already went half the way to a working Hyper-V Gen2 autoinstall (of cause without console keyboard yet) i am quite sure pfSense 2.4 will offer similar support in the future. :)
-
@johnkeates:
@johnkeates:
But why does it need a custom kbd or DVD driver in the first place? That stuff is so standard, it should just work out of the box, unless the UEFI implementation doesn't have normal I/O devices.
They ripped out a lot of legacy stuff in Hyper-V gen2.
There are some obvious devices we removed. For example, the legacy network adapter (which is an emulated device based on a DEC/Intel/Tulip 21140). Then we removed the IDE controller. And the floppy controller plus associated DMA controller. And the serial controller (COM ports). These are all things you can directly see in the VM settings.
Then we changed other devices such as removing the legacy i8042 keyboard controller (which has an interesting side effect I will talk about in a future part), PS/2 mouse, S3 Video, the Programmable Interrupt Controller (PIC), the Programmable Interrupt Timer (PIT), the Super I/O device on which floppy support relied. We actually went even further by removing the PCI bus as well. For good measure, we also removed the speaker and the numerical co-processor. We also revised ACPI.
Of course, when you rip this much out, you may initially think ‘could an operating system boot in this environment?’. With just the above changes, the answer would be no. Primarily because the most common ways of booting a generation 1 VM are a disk/VHD attached to an IDE controller, an ISO/DVD drive attached to an IDE controller, or PXE boot from the legacy network adapter. I’m ignoring boot from floppy (.VFD) – I’m sure not many people do this these days!
Given that both IDE and Legacy NIC are no longer present, we had to introduce alternate boot mechanisms, and a new DVD device. We extended the UEFI firmware to be aware of software based devices (ie those using VMBus), and support booting from our software-based SCSI and Network controllers, plus the new software-based DVD device.
https://blogs.technet.microsoft.com/jhoward/2013/10/24/hyper-v-generation-2-virtual-machines-part-1/
I know, but UEFI provides services for I/O, there is framebuffer/console, there is graphics, there is keyboard and mouse I/O, and those are what you would use on a modern UEFI OS. Now, it could be that FreeBSD hasn't fixed that yet and simply ignores those and uses the legacy devices, but if that were the case, it wouldn't work on OVMF UEFI either, yet it does. It seems to me that the real problem is Microsoft inventing VMBus instead of using what already exists and already has support. I get that it can be implemented on all and any operating systems where you have source code access, but it cripples the hypervisor in a way that it's only paravirtualisation from that point onwards.
In the referenced article, Microsoft says, "Our implementation is based on TianoCore, EDK II and is a UEFI 2.3.1 implementation with necessary modifications and bug-fixes", as of three years ago. That's the same code base that you're talking about. How different can it be? FreeBSD doesn't support Gen 2 VMs so it probably shouldn't be a surprise that it doesn't work properly.
-
@johnkeates:
In the referenced article, Microsoft says, "Our implementation is based on TianoCore, EDK II and is a UEFI 2.3.1 implementation with necessary modifications and bug-fixes", as of three years ago. That's the same code base that you're talking about. How different can it be? FreeBSD doesn't support Gen 2 VMs so it probably shouldn't be a surprise that it doesn't work properly.
Well, I don't know how different it can be. It's really strange that it would not work as it does work with OVMF which is also based on the EDK2 and TianoCore. Hell, it's practically part of TianoCore http://www.tianocore.org/ovmf/
Well, I guess we will have to wait for BIS to be updated. Hopefully it won't take long.
-
Hi, I'm Dexuan From the BIS team.
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue. 10.2 SMP VM can't work properly either (it seems UP VM can work, as I mentioned in Bug 211746, but the keyboard definitely can't work.)
Details are at:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195819
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211746About why the keyboard can't work in Hyper-V UEFI VM, my understanding is: Hyper-V UEFI implementation doesn't emulate the legacy i8042 keyboard controller (UEFI spec doesn't require this legacy thing), and it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
And, to enable Secure Boot with Hyper-V UEFI, I suppose FreeBSD and Microsoft need work together to sign the binary to the hypervisor.
So, to make Hyper-V UEFI "work" at a minimal level for FreeBSD VM, we at least need to fix the loader issue in 10.3/11/HEAD, and add a keyboard driver (UEFI keyboard driver or VMBus based Hyper-V keyboard driver).
These are in our long TODO list, and we definitely want to get them done ASAP, but unluckily we have to prioritize the tasks since actually we have very limited resources. :-(Please let us know if my above information is not accurate.
-
@Dexuan:
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue.
… it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
...
Please let us know if my above information is not accurate.
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
It's not completly accurate.
FreeBSD 11 / pfSense 2.4 boots fine in Hyper-V Gen2 (unsecured mode) if one converts an existing UEFI install p2v or v2v to a vhdx and uses this image as disk device. Boots fine in Server 2016 Hyper-V Gen2, network works in 10GBit mode (thats why i work on it), network performance is fair. Network pumps a little (using a x520-da2 for hn0/1 as lan/wan), but this should be enhenceable with some driver setting fine tuning.
OpnSense, a pfSense fork, contains a autoinstaller in their FreeBSD 11 based devel version that works in HV Gen2 , so there must be some kind of working gen2 dvd driver out there.
So what imho is needed most is a port of the LIS keyboard driver to BIS, would really be nice to have a working console. :)
-
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
@johnkeates:
Thanks for the update! It's great to see someone involved connect back to downstream projects and users.
Agreed, it's nice to see.
-
@Dexuan:
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue.
… it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
...
Please let us know if my above information is not accurate.
It's not completly accurate.
FreeBSD 11 / pfSense 2.4 boots fine in Hyper-V Gen2 (unsecured mode) if one converts an existing UEFI install p2v or v2v to a vhdx and uses this image as disk device. Boots fine in Server 2016 Hyper-V Gen2, network works in 10GBit mode (thats why i work on it), network performance is fair. Network pumps a little (using a x520-da2 for hn0/1 as lan/wan), but this should be enhenceable with some driver setting fine tuning.
OpnSense, a pfSense fork, contains a autoinstaller in their FreeBSD 11 based devel version that works in HV Gen2 , so there must be some kind of working gen2 dvd driver out there.
So what imho is needed most is a port of the LIS keyboard driver to BIS, would really be nice to have a working console. :)
Thanks for the info!
When I said FreeBSD-11 Generation-2 VM on Hyper-V didn't work, I meant I couldn't install the VM with the FreeBSD-11 DVD (I tried FreeBSD-11.0-BETA1-amd64-dvd1.iso, and I don't think the final offcial DVD could work) due to the loader issue mentioned in FreeBSD bug 211746.
It looks with the p2v/v2v method the loader issue can be worked around somehow, and it looks the loader in OpenSence is good.
Yeah, the keyboard driver is the most needed.
BTW, about "gen2 dvd driver": I think in a gen-2 FreeBSD VM the existing storvsc driver should be able to properly handle the DVD device, so we don't need to add a special "dvd driver".
-
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
@johnkeates:
Thanks for the update! It's great to see someone involved connect back to downstream projects and users.
Agreed, it's nice to see.
Surely we want to better support all users of Hyper-V. :-)
BTW, we have a wiki page at https://wiki.freebsd.org/HyperV and you're welcome to post any "BSD VM on Hyper-V" questions to the FreeBSD Integration Services Team email (the bsdic@ email) on that page. We monitor the email daily and we respond directly and timely.
-
@johnkeates:
@Dexuan:
About why the keyboard can't work in Hyper-V UEFI VM, my understanding is: Hyper-V UEFI implementation doesn't emulate the legacy i8042 keyboard controller (UEFI spec doesn't require this legacy thing), and it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
I do wonder what the Xen and KVM OVMF UEFI boot method is doing different from the Microsoft implementation. Maybe they still have that legacy controller? Or possibly a USB keyboard controller? I haven't checked for any of those, but it would make sense if that's why it does work there.
I'm also curious. It would be great if somebody could look into Xen/KVM on this.
-
Secure boot in hyper-v 2016 is now much more mainstream and is supported by the biggest Linux distributions and will surely arrive to freebsd as well. It is a feature I can live without for the moment and most likely forever in 2012r2.
I think secure boot is something that needs to be solved by freebsd and not pfsense.
Linux Secure Boot (new)
Linux operating systems running on generation 2 virtual machines can now boot with the Secure Boot option enabled. Ubuntu 14.04 and later, SUSE Linux Enterprise Server 12 and later, Red Hat Enterprise Linux 7.0 and later, and CentOS 7.0 and later are enabled for Secure Boot on hosts that run Windows Server 2016. Before you boot the virtual machine for the first time, you must configure the virtual machine to use the Microsoft UEFI Certificate Authority. You can do this from Hyper-V Manager, Virtual Machine Manager, or an elevated Windows Powershell session. For Windows PowerShell, run this command:https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/what-s-new-in-hyper-v-on-windows
I think the response here is great and my hope is that we will get gen 2 support soon even tough it might not be in 2.4.0.
-
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
-
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
I bet bsdic@microsoft.com knows.
-
@johnkeates:
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
I bet bsdic@microsoft.com knows.
Yes, they do know. I contacted them and heard back this morning. Generation 2 is in the development plan and is scheduled to be completed in the March 2017 time frame.
-
-
It's been a while since this topic had posts, but it seemed appropriate to follow-up here.
@johnkeates: Have you been in touch with your contacts since your reply back in November? Since we're now March, I'm curious how far out generation 2 VM support in Hyper-V might be for pfSense 2.4.
-
It's been a while since this topic had posts, but it seemed appropriate to follow-up here.
@johnkeates: Have you been in touch with your contacts since your reply back in November? Since we're now March, I'm curious how far out generation 2 VM support in Hyper-V might be for pfSense 2.4.
You'll have to ask bimmerdriver or bsdic@microsoft.com as I don't have anything new at the moment.
-
@johnkeates:
It's been a while since this topic had posts, but it seemed appropriate to follow-up here.
@johnkeates: Have you been in touch with your contacts since your reply back in November? Since we're now March, I'm curious how far out generation 2 VM support in Hyper-V might be for pfSense 2.4.
You'll have to ask bimmerdriver or bsdic@microsoft.com as I don't have anything new at the moment.
I haven't heard anything since my last post. If you contact bsdic@microsoft.com, please post an update here.
-
@bimmerdriver: I did reach out to bsdic@microsoft.com, and received a reply last night stating the following about FreeBSD generation 2 VM support:
1. The HEAD can boot on Hyper-V as Generation-2 VM now (patches for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211746 will be pushed to stable/10 and stable/11 soon); however, Secure Boot must be disabled since it isn't supported yet.
2. The virtual DVD drive in the VM doesn't work yet. They are trying to fix it. In the meantime VM images can be created by following the "Bootable UEFI memory stick or Hard Disk" section of the UEFI doc on wiki.freebsd.org.
3. The keyboard of the VM doesn't work yet, but they are adding a driver for it so it should work soon.
Other than that they're running more tests to see if there are other issues. Sounds like it's pretty close though.
-
Hi all!
Dev team just added synthetic keyboard driver a few days ago :)
https://svnweb.freebsd.org/base/stable/11/sys/dev/hyperv/input/?view=log&sortby=file&pathrev=317189
Tested on fresh install of FreeBSD 11-STABLE within Hyper-V (MS 2016 Std as the host) and the keyboard works 8)
It seems the FreeBSD Gen 2 VMs is closing up to the final approach…
dmesg output:
Copyright
1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-STABLE #0 r317291: Sat Apr 22 22:51:09 CEST 2017
root@abc.def:/usr/obj/usr/src/sys/CUSTOM amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
SRAT: Ignoring memory at addr 0x808200000
SRAT: Ignoring memory at addr 0x1000000000
SRAT: Ignoring memory at addr 0x10000200000
SRAT: Ignoring memory at addr 0x20000200000
SRAT: Ignoring memory at addr 0x40000200000
SRAT: Ignoring memory at addr 0x80000200000
VT(efifb): resolution 1024x768
Hyper-V Version: 10.0.14393 [SP1]
Features=0x2e7f <vpruntime,tmrefcnt,synic,syntm,apic,hypercall,vpindex,reftsc,idle,tmfreq>PM Features=0x0 [C2]
Features3=0xed7b2 <debug,xmmhc,idle,numa,tmfreq,syncmc,crash,npiep>Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (3192.61-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0x1f83fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt>Features2=0xfed83203 <sse3,pclmulqdq,ssse3,fma,cx16,sse4.1,sse4.2,movbe,popcnt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x121 <lahf,abm,prefetch>Structured Extended Features=0x1c2b38 <bmi1,hle,avx2,bmi2,erms,rtm,nfpusg,rdseed,adx,smap>XSAVE Features=0x1 <xsaveopt>Hypervisor: Origin = "Microsoft Hv"
real memory = 34359738368 (32768 MB)
avail memory = 33349726208 (31804 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <vrtual microsft="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ioapic0 <version 1.1="">irqs 0-23 on motherboard
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd0 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80765d10, 0) error 19
nexus0
cryptosoft0: <software crypto="">on motherboard
acpi0: <vrtual microsft="">on motherboard
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
vmbus0: <hyper-v vmbus="">irq 5 on acpi0
Timecounters tick every 1.000 msec
ipfw2 initialized, divert loadable, nat loadable, default to deny, logging disabled
usb_needs_explore_all: no devclass
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]…
mountroot: waiting for device /dev/gpt/rootfs...
vmbus0: version 3.0
hvet0: <hyper-v event="" timer="">on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
hvkbd0: <hyper-v kbd="">on vmbus0</hyper-v>
hvheartbeat0: <hyper-v heartbeat="">on vmbus0
hvkvp0: <hyper-v kvp="">on vmbus0
hvshutdown0: <hyper-v shutdown="">on vmbus0
hvtimesync0: <hyper-v timesync="">on vmbus0
hvtimesync0: RTT
hvvss0: <hyper-v vss="">on vmbus0
storvsc0: <hyper-v scsi="">on vmbus0
(probe0:storvsc0:0:0:0): storvsc scsi_status = 2
hn0: <hyper-v network="" interface="">on vmbus0
da0 at storvsc0 bus 0 scbus0 target 0 lun 0
da0: <msft virtual="" disk="" 1.0="">Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 81920MB (167772160 512 byte sectors)
hn0: got notify, nvs type 128
hn0: Ethernet address: 00:15:5d:08:1e:00
hn0: link state changed to UP
hn1: <hyper-v network="" interface="">on vmbus0
hn1: got notify, nvs type 128
hn1: Ethernet address: 00:15:5d:08:1e:01
hn1: link state changed to UP
pcib0: <hyper-v pci="" express="" pass="" through="">on vmbus0
pcib0: vmbus_pcib failed to enable D0
device_attach: pcib0 attach returned 92
pcib0: <hyper-v pci="" express="" pass="" through="">on vmbus0
pcib0: vmbus_pcib failed to enable D0
device_attach: pcib0 attach returned 92</hyper-v></hyper-v></hyper-v></msft></hyper-v></hyper-v></hyper-v></hyper-v></hyper-v></hyper-v></hyper-v></hyper-v></hyper-v></at></acpi></acpi></acpi></acpi></vrtual></software></version></vrtual></xsaveopt></bmi1,hle,avx2,bmi2,erms,rtm,nfpusg,rdseed,adx,smap></lahf,abm,prefetch></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,ssse3,fma,cx16,sse4.1,sse4.2,movbe,popcnt,aesni,xsave,osxsave,avx,f16c,rdrand,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2,ss,htt></debug,xmmhc,idle,numa,tmfreq,syncmc,crash,npiep></vpruntime,tmrefcnt,synic,syntm,apic,hypercall,vpindex,reftsc,idle,tmfreq>
-
Yah, really good. Looks like support for gen 2 VM is getting close.