Hyper-V
-
When I was looking at some links about hyper-v and freebsd, I noticed there was an email address for the program manager. I emailed him and got a reply right away. Apparently for freebsd 11, they are adding vRSS and other network improvements, TRIM support, and VHDX resize. Gen 2 is in the plan. They are asking if there are any specific features that are required in order to prioritize the development.
I can pass any provided feedback to them, but since I don't speak for the pfsense project, it might be better if someone from the project stepped in. If someone would like to do that, I will make an introduction. (Send me a message and we can take it offline.)
-
@johnkeates:
I'd say correct UEFI support would be key, otherwise there is no storage access for the DVD/CD, and no console input. Checking with the hacks that are requied to make it work now, I'd say those are the first things that need fixing. New features are nice, but making it work is more important. Gen2 has to work like a real PC besides viridian interfaces. In Xen lingo: PVHVM or PV is nice, but without having a working HVM, it's useless.
Since Gen 2 only uses UEFI, I think it goes without saying UEFI is a fundamental requirement.
-
It's more the issue that whatever EFI they have now doesn't work like OVMF works, and as long as it doesn't have that level of compatibility it's not going to work like it does on other platforms.
-
@johnkeates:
It's more the issue that whatever EFI they have now doesn't work like OVMF works, and as long as it doesn't have that level of compatibility it's not going to work like it does on other platforms.
Okay, I see your point. If you have any other comments WRT prioritization of LIS / BIS features, please post them.
-
I did some additional Hyper-V Gen2 tests with similar OS's.
- FreeBSD-11.0-RELEASE-amd64.vhd is not usable as a Gen2 disk
- FreeBSD-11.0-RELEASE-amd64-disc1.iso hangs on iso access
- "forked"sense-17.1.a-LibreSSL-cdrom-amd64.iso autoinstalls, but no kbd access on console
Upstream has planned, but is not yet working on a kbd driver.
–- Comment #1 from Sepherosa Ziehau <sepherosa at="" gmail.com="">---
The keyboard driver for gen2 hyper-v is planned, though it has not been started
yet. I'd suggest you use gen1 for now.</sepherosa>https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210175
-
@johnkeates:
But why does it need a custom kbd or DVD driver in the first place? That stuff is so standard, it should just work out of the box, unless the UEFI implementation doesn't have normal I/O devices.
They ripped out a lot of legacy stuff in Hyper-V gen2.
There are some obvious devices we removed. For example, the legacy network adapter (which is an emulated device based on a DEC/Intel/Tulip 21140). Then we removed the IDE controller. And the floppy controller plus associated DMA controller. And the serial controller (COM ports). These are all things you can directly see in the VM settings.
Then we changed other devices such as removing the legacy i8042 keyboard controller (which has an interesting side effect I will talk about in a future part), PS/2 mouse, S3 Video, the Programmable Interrupt Controller (PIC), the Programmable Interrupt Timer (PIT), the Super I/O device on which floppy support relied. We actually went even further by removing the PCI bus as well. For good measure, we also removed the speaker and the numerical co-processor. We also revised ACPI.
Of course, when you rip this much out, you may initially think ‘could an operating system boot in this environment?’. With just the above changes, the answer would be no. Primarily because the most common ways of booting a generation 1 VM are a disk/VHD attached to an IDE controller, an ISO/DVD drive attached to an IDE controller, or PXE boot from the legacy network adapter. I’m ignoring boot from floppy (.VFD) – I’m sure not many people do this these days!
Given that both IDE and Legacy NIC are no longer present, we had to introduce alternate boot mechanisms, and a new DVD device. We extended the UEFI firmware to be aware of software based devices (ie those using VMBus), and support booting from our software-based SCSI and Network controllers, plus the new software-based DVD device.
https://blogs.technet.microsoft.com/jhoward/2013/10/24/hyper-v-generation-2-virtual-machines-part-1/
-
@johnkeates:
It seems to me that the real problem is Microsoft inventing VMBus instead of using what already exists and already has support.
I understand what you mean, but it is as it is. Someone has to make a decision, either to support this or to let it be.
It doesn't really matter to me, i already did my job at work and found a way to run 2.4 under Hyper-V Gen2. Would be nice if it were easier to set up, but i can live with it as it is. As soon as i had a clonable image the job was done. Now i wait for 2.4-rel, repeat the setup and the rest is history for me.
BTW. it runs quite stable, i have seen two weeks stable 4.5 GBit routed http transport now in our test rack.
-
@johnkeates:
I do wonder if we could get around some of this by chainloading pfSense, I've seen people loading OpenFirmware, TianoCore, OVMF and with some hacks even SeaBIOS on top of existing firmwares.
Well, that would be too much diy toying for me. Not my style.
And since that forked sense already went half the way to a working Hyper-V Gen2 autoinstall (of cause without console keyboard yet) i am quite sure pfSense 2.4 will offer similar support in the future. :)
-
@johnkeates:
@johnkeates:
But why does it need a custom kbd or DVD driver in the first place? That stuff is so standard, it should just work out of the box, unless the UEFI implementation doesn't have normal I/O devices.
They ripped out a lot of legacy stuff in Hyper-V gen2.
There are some obvious devices we removed. For example, the legacy network adapter (which is an emulated device based on a DEC/Intel/Tulip 21140). Then we removed the IDE controller. And the floppy controller plus associated DMA controller. And the serial controller (COM ports). These are all things you can directly see in the VM settings.
Then we changed other devices such as removing the legacy i8042 keyboard controller (which has an interesting side effect I will talk about in a future part), PS/2 mouse, S3 Video, the Programmable Interrupt Controller (PIC), the Programmable Interrupt Timer (PIT), the Super I/O device on which floppy support relied. We actually went even further by removing the PCI bus as well. For good measure, we also removed the speaker and the numerical co-processor. We also revised ACPI.
Of course, when you rip this much out, you may initially think ‘could an operating system boot in this environment?’. With just the above changes, the answer would be no. Primarily because the most common ways of booting a generation 1 VM are a disk/VHD attached to an IDE controller, an ISO/DVD drive attached to an IDE controller, or PXE boot from the legacy network adapter. I’m ignoring boot from floppy (.VFD) – I’m sure not many people do this these days!
Given that both IDE and Legacy NIC are no longer present, we had to introduce alternate boot mechanisms, and a new DVD device. We extended the UEFI firmware to be aware of software based devices (ie those using VMBus), and support booting from our software-based SCSI and Network controllers, plus the new software-based DVD device.
https://blogs.technet.microsoft.com/jhoward/2013/10/24/hyper-v-generation-2-virtual-machines-part-1/
I know, but UEFI provides services for I/O, there is framebuffer/console, there is graphics, there is keyboard and mouse I/O, and those are what you would use on a modern UEFI OS. Now, it could be that FreeBSD hasn't fixed that yet and simply ignores those and uses the legacy devices, but if that were the case, it wouldn't work on OVMF UEFI either, yet it does. It seems to me that the real problem is Microsoft inventing VMBus instead of using what already exists and already has support. I get that it can be implemented on all and any operating systems where you have source code access, but it cripples the hypervisor in a way that it's only paravirtualisation from that point onwards.
In the referenced article, Microsoft says, "Our implementation is based on TianoCore, EDK II and is a UEFI 2.3.1 implementation with necessary modifications and bug-fixes", as of three years ago. That's the same code base that you're talking about. How different can it be? FreeBSD doesn't support Gen 2 VMs so it probably shouldn't be a surprise that it doesn't work properly.
-
@johnkeates:
In the referenced article, Microsoft says, "Our implementation is based on TianoCore, EDK II and is a UEFI 2.3.1 implementation with necessary modifications and bug-fixes", as of three years ago. That's the same code base that you're talking about. How different can it be? FreeBSD doesn't support Gen 2 VMs so it probably shouldn't be a surprise that it doesn't work properly.
Well, I don't know how different it can be. It's really strange that it would not work as it does work with OVMF which is also based on the EDK2 and TianoCore. Hell, it's practically part of TianoCore http://www.tianocore.org/ovmf/
Well, I guess we will have to wait for BIS to be updated. Hopefully it won't take long.
-
Hi, I'm Dexuan From the BIS team.
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue. 10.2 SMP VM can't work properly either (it seems UP VM can work, as I mentioned in Bug 211746, but the keyboard definitely can't work.)
Details are at:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195819
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211746About why the keyboard can't work in Hyper-V UEFI VM, my understanding is: Hyper-V UEFI implementation doesn't emulate the legacy i8042 keyboard controller (UEFI spec doesn't require this legacy thing), and it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
And, to enable Secure Boot with Hyper-V UEFI, I suppose FreeBSD and Microsoft need work together to sign the binary to the hypervisor.
So, to make Hyper-V UEFI "work" at a minimal level for FreeBSD VM, we at least need to fix the loader issue in 10.3/11/HEAD, and add a keyboard driver (UEFI keyboard driver or VMBus based Hyper-V keyboard driver).
These are in our long TODO list, and we definitely want to get them done ASAP, but unluckily we have to prioritize the tasks since actually we have very limited resources. :-(Please let us know if my above information is not accurate.
-
@Dexuan:
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue.
… it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
...
Please let us know if my above information is not accurate.
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
It's not completly accurate.
FreeBSD 11 / pfSense 2.4 boots fine in Hyper-V Gen2 (unsecured mode) if one converts an existing UEFI install p2v or v2v to a vhdx and uses this image as disk device. Boots fine in Server 2016 Hyper-V Gen2, network works in 10GBit mode (thats why i work on it), network performance is fair. Network pumps a little (using a x520-da2 for hn0/1 as lan/wan), but this should be enhenceable with some driver setting fine tuning.
OpnSense, a pfSense fork, contains a autoinstaller in their FreeBSD 11 based devel version that works in HV Gen2 , so there must be some kind of working gen2 dvd driver out there.
So what imho is needed most is a port of the LIS keyboard driver to BIS, would really be nice to have a working console. :)
-
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
@johnkeates:
Thanks for the update! It's great to see someone involved connect back to downstream projects and users.
Agreed, it's nice to see.
-
@Dexuan:
AFAIK, FreeBSD 10.3, 11 and the HEAD can't work with Hyper-V Generation-2 VM (i.e. UEFI VM) due to a loader issue.
… it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
...
Please let us know if my above information is not accurate.
It's not completly accurate.
FreeBSD 11 / pfSense 2.4 boots fine in Hyper-V Gen2 (unsecured mode) if one converts an existing UEFI install p2v or v2v to a vhdx and uses this image as disk device. Boots fine in Server 2016 Hyper-V Gen2, network works in 10GBit mode (thats why i work on it), network performance is fair. Network pumps a little (using a x520-da2 for hn0/1 as lan/wan), but this should be enhenceable with some driver setting fine tuning.
OpnSense, a pfSense fork, contains a autoinstaller in their FreeBSD 11 based devel version that works in HV Gen2 , so there must be some kind of working gen2 dvd driver out there.
So what imho is needed most is a port of the LIS keyboard driver to BIS, would really be nice to have a working console. :)
Thanks for the info!
When I said FreeBSD-11 Generation-2 VM on Hyper-V didn't work, I meant I couldn't install the VM with the FreeBSD-11 DVD (I tried FreeBSD-11.0-BETA1-amd64-dvd1.iso, and I don't think the final offcial DVD could work) due to the loader issue mentioned in FreeBSD bug 211746.
It looks with the p2v/v2v method the loader issue can be worked around somehow, and it looks the loader in OpenSence is good.
Yeah, the keyboard driver is the most needed.
BTW, about "gen2 dvd driver": I think in a gen-2 FreeBSD VM the existing storvsc driver should be able to properly handle the DVD device, so we don't need to add a special "dvd driver".
-
Hi Dexuan,
great to see your team cares, thx a lot and greets to Josh Poulson!
@johnkeates:
Thanks for the update! It's great to see someone involved connect back to downstream projects and users.
Agreed, it's nice to see.
Surely we want to better support all users of Hyper-V. :-)
BTW, we have a wiki page at https://wiki.freebsd.org/HyperV and you're welcome to post any "BSD VM on Hyper-V" questions to the FreeBSD Integration Services Team email (the bsdic@ email) on that page. We monitor the email daily and we respond directly and timely.
-
@johnkeates:
@Dexuan:
About why the keyboard can't work in Hyper-V UEFI VM, my understanding is: Hyper-V UEFI implementation doesn't emulate the legacy i8042 keyboard controller (UEFI spec doesn't require this legacy thing), and it seems FreeBSD kernel doesn't have the UEFI I/O service based keyboard driver(?).
I do wonder what the Xen and KVM OVMF UEFI boot method is doing different from the Microsoft implementation. Maybe they still have that legacy controller? Or possibly a USB keyboard controller? I haven't checked for any of those, but it would make sense if that's why it does work there.
I'm also curious. It would be great if somebody could look into Xen/KVM on this.
-
Secure boot in hyper-v 2016 is now much more mainstream and is supported by the biggest Linux distributions and will surely arrive to freebsd as well. It is a feature I can live without for the moment and most likely forever in 2012r2.
I think secure boot is something that needs to be solved by freebsd and not pfsense.
Linux Secure Boot (new)
Linux operating systems running on generation 2 virtual machines can now boot with the Secure Boot option enabled. Ubuntu 14.04 and later, SUSE Linux Enterprise Server 12 and later, Red Hat Enterprise Linux 7.0 and later, and CentOS 7.0 and later are enabled for Secure Boot on hosts that run Windows Server 2016. Before you boot the virtual machine for the first time, you must configure the virtual machine to use the Microsoft UEFI Certificate Authority. You can do this from Hyper-V Manager, Virtual Machine Manager, or an elevated Windows Powershell session. For Windows PowerShell, run this command:https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/what-s-new-in-hyper-v-on-windows
I think the response here is great and my hope is that we will get gen 2 support soon even tough it might not be in 2.4.0.
-
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
-
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
I bet bsdic@microsoft.com knows.
-
@johnkeates:
Does anyone know what the status of freebsd 11 support on generation 2 hyper-v is?
I bet bsdic@microsoft.com knows.
Yes, they do know. I contacted them and heard back this morning. Generation 2 is in the development plan and is scheduled to be completed in the March 2017 time frame.