Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1

For any more info and/or finding a way to get it published i assume direct mail will work faster then a off topic forum chat.

jim-at-netgate-dot-com
or
jim-at-pfsense-dot-org

zootie

I've been hammering on it, trying to get it to build a working hyper-v ISO using the tools. They're small changes, it just takes a long time to test them (and it'd help to have help from someone familiar with the build process, specifically on kernel building)

Results have been mixed. Part of the problem has been that the compile options used to build the kernel that is used in the ISO don't seem to work with the hyper-v modules (they compile, just don't work). We'll likely have to use the same AMD64 SMP kernel that gets installed on HD on the ISO (don't know how to specify this in the tools).

When forcing the SMP kernel and modules onto the ISO, I can get the utilities and network driver to load (so the live CD can run a firewall with the synthetic network driver as you'd expect), but I've been unable to get it to work when loading the hyper-v storage modules because it can't find the optical drive when it boots (this is what the hv_ata_pci_disengage.ko module is supposed to prevent, but it doesn't seem to be working), and since it doesn't load the storage driver, you'd have to create the geom labels as described in the first post (if you want improved disk performance).

key4ce

Hmm,

Well it been a while back since i tried your iso (when it was still publicly available).
I do remember there where issues with CARP + FreeBSD at the time (which would be one of the things we do need working).

Seems our R&D is still on initial steps to first get a FreeBSD 8.3 100% working with hyper-V.

Once thats completed they (hopefully) know what to do with PFSense to make it work :-) (or so i was told)

zootie

I think I figured it out. I had to go through some false starts; but, with minor changes to pfsense tools, I finally was able to build an ISO that runs the live CD and installs using the synthetic device drivers. I'll post more details once I run some more tests.

With any luck, this will help the community short term, maybe allow to have an official Hyper-V edition within the pfSense codebase, and give key4ce and team a head start on making a CARP enabled version.

peterclark4

@zootie:

I think I figured it out. I had to go through some false starts; but, with minor changes to pfsense tools, I finally was able to build an ISO that runs the live CD and installs using the synthetic device drivers. I'll post more details once I run some more tests.

With any luck, this will help the community short term, maybe allow to have an official Hyper-V edition within the pfSense codebase, and give key4ce and team a head start on making a CARP enabled version.

That is great news!

If you need any help testing just let me know.

Peter

falumas

Great news for somebody like me who loves Hyper-V and Pfsense.

I can also help with the testing if needed.

tester_02

Now that pfsense is moving onto a newer freebsd, would that mean that the latest hyper-v code is in the new freebsd?

rainnyday

Hi Zootie,

Thanks for this great post. Can you provide the download link to me, the link on your original post did not work for me.

Rainny

zootie

Good news that pfSense 2.1.1 is now released. I'll try and test the modules with it and report back (but they should work using Option B work w/o needing to recompile).

Back to building an ISO with integrated drivers. The key element to be able to get the live CD to work was to set hw.ata.disk_enable so it wouldn't disconnect the optical drive from the live CD during boot (for more info, see "Both FastIDE and CD-Rom work in FreeBSD 10 on Hyper-V (…)" and "svn commit: r252645 (…) stordisengage storvsc vmbus").

As of my last testing (a few weeks ago), I had 2 variations of the ISO (both requiring minor changes to the pfsense builder scripts):

ISO-A - Changing build_iso.sh so it would set hw.ata.disk_enable and loading the ko drivers in the ISO's loader.conf.local
ISO-B - Changing rc.cdrom and detecting if it is running under Hyper-V and then loading the modules on demand and then apply the loader.conf.local changes to the installation

ISO-A is the simplest and most direct, but when I tested it on real HW, it would cause issues (the boot process would get disconnected from the CD), and it wouldn't necessarily setup pfSense on the direct access disk driver (it uses ad0 rather than da0 storage device). ISO-A would likely require having a "Hyper-V only edition ISO" (it doesn't require forking the source, just adding an optional option to make the Hyper-V changes).

ISO-B is more dynamic, and it only loads the drivers when it is running under Hyper-V (which I imagine is that FreeBSD 10's installer does) and then fixes up the new installation by loading the modules, so it wouldn't require a Hyper-V only ISO edition. However, it would allow both the ATA and direct access storage drivers to be visible from pfSense installer (you'd see both ad0 and da0 in the installer - see attached screenshot), so it could be a little confusing (and it might require some editing of loader.conf and/or creating labels afterwards if you want to switch drivers afterwards).

I'm thinking that as part of the fixup in ISO-B, the script could check what disk device driver is being used in the new install's fstab, and then only set hw.ata.disk_enable when the ad0 drivers is being used (or not even load the storage driver), but it seems non-optimal (the installer would still see the 2 disk devices during setup, maybe confusing users). I'll have to try it, and lacking a better alternative, it would work ok (IMO, better than having a Hyper-V only edition)

What would we prefer? A separate ISO only Hyper-V? Maybe there is a way to hide a disk device in the installer? Unsure if there is a way to selectively unload the ATA driver (so it is still working for the virtual CD, but no longer visible for the disk - I imagine this logic is in the FreeBSD 10 as well). Maybe there is another setting similar to hw.ata.disk_enable that only applies to optical drives?

Jim, maybe someone more familiar with the pfSense installer can help?

BTW, I emailed a month ago to try and get access to the tools repo, but I haven't got a reply yet.

pfSense-Installer-da0VSad0.png_thumb

taliesins

Thanks for you hard work Zootie. This will setup kvp so that the integration services will report the ip address. It was essential for me as I was spinning up pfsense in hyper-v with vagrant. Its brilliant for devs to host their own load balancers. Hopefully vagrant will take my patch to handle multiple network card and the ability to specify the switch that they are in, so we can all use it.

I would really love it if someone could create a package for hyper-v so we can simply install it on a base image.

So the approach I took was to try 2.1, 2.1.1 and 2.2.

For 2.1 the network card drivers work but the kvp daemon did not
For 2.1.1 the network card drivers didnt work but the kvp daemon did (hopefully zooties iso will fix the network card problem)
For 2.2 the network card drivers are built in. So I just setup the kvp daemon

https://github.com/FreeBSDonHyper-V/Hyperv-Ports/wiki/_pages

Add required packages

cd /tmp
pkg install curl
exit

Add hv-kvp


cd /tmp
curl -L https://github.com/FreeBSDonHyper-V/Hyperv-Ports/raw/hyperv-ic-master/BIS-1.0/FreeBSD-10.0/bin/hv-kvp-x64.txz -o hv-kvp-x64.txz
pkg add hv-kvp-x64.txz
exit

Start hv_kvpd on boot


cp /boot/kernel/hv_kvp.ko /usr/local/hyperv/
cp /etc/rc.d/hv_kvpd /usr/local/etc/rc.d/hv_kvpd.sh

Need to copy ko module back after an upgrade
/usr/local/etc/rc.d/hv_kvpd.sh


!test -f /boot/kernel/hv_kvp.ko || cp /usr/local/hyperv/hv_kvp.ko  /boot/kernel

Remove stuff from loader.conf as it should live in loader.conf.local
remove from /boot/loader.conf


# Loader labels for Hyper-V drivers -do not modify
hv_kvp_load="YES"

add to /boot/loader.conf.local


# Loader labels for Hyper-V drivers -do not modify
hv_kvp_load="YES"

Move stuff from rc.conf to rc.conf.local
/etc/rc.conf
Get nuked on reboot so we just need to reboot.

add to /etc/rc.conf.local


# Labels for KVP daemon -do not modify
hv_kvp_daemon_enable="YES"

Fix dhcp script
/usr/local/hyperv/scripts/hv_get_dhcp_info
Change:


if_file="/etc/rc.conf"

To:


if_file="/etc/rc.conf.local"

Fix timecounter:


 sysctl kern.timecounter.hardware=TSC

Fix QoS:
/etc/inc/interfaces.inc

search for altq
add "hn" to array of interfaces

duntuk

Thanks @zootie! looking forward to your release.

eri--

I would suggest to use 2.2 for this effort.
Its aim is just to move to FreeBSD 10 with small effort.

That means try to release it ASAP.
Probably your work there is simpler since most of the thing is there or tell us if anything missing, for now.

bryan.paradis

@zootie:

Good news that pfSense 2.1.1 is now released. I'll try and test the modules with it and report back (but they should work using Option B work w/o needing to recompile).

Back to building an ISO with integrated drivers. The key element to be able to get the live CD to work was to set hw.ata.disk_enable so it wouldn't disconnect the optical drive from the live CD during boot (for more info, see "Both FastIDE and CD-Rom work in FreeBSD 10 on Hyper-V (…)" and "svn commit: r252645 (…) stordisengage storvsc vmbus").

As of my last testing (a few weeks ago), I had 2 variations of the ISO (both requiring minor changes to the pfsense builder scripts):

ISO-A - Changing build_iso.sh so it would set hw.ata.disk_enable and loading the ko drivers in the ISO's loader.conf.local

ISO-B - Changing rc.cdrom and detecting if it is running under Hyper-V and then loading the modules on demand and then apply the loader.conf.local changes to the installation

ISO-A is the simplest and most direct, but when I tested it on real HW, it would cause issues (the boot process would get disconnected from the CD), and it wouldn't necessarily setup pfSense on the direct access disk driver (it uses ad0 rather than da0 storage device). ISO-A would likely require having a "Hyper-V only edition ISO" (it doesn't require forking the source, just adding an optional option to make the Hyper-V changes).

ISO-B is more dynamic, and it only loads the drivers when it is running under Hyper-V (which I imagine is that FreeBSD 10's installer does) and then fixes up the new installation by loading the modules, so it wouldn't require a Hyper-V only ISO edition. However, it would allow both the ATA and direct access storage drivers to be visible from pfSense installer (you'd see both ad0 and da0 in the installer - see attached screenshot), so it could be a little confusing (and it might require some editing of loader.conf and/or creating labels afterwards if you want to switch drivers afterwards).

I'm thinking that as part of the fixup in ISO-B, the script could check what disk device driver is being used in the new install's fstab, and then only set hw.ata.disk_enable when the ad0 drivers is being used (or not even load the storage driver), but it seems non-optimal (the installer would still see the 2 disk devices during setup, maybe confusing users). I'll have to try it, and lacking a better alternative, it would work ok (IMO, better than having a Hyper-V only edition)

What would we prefer? A separate ISO only Hyper-V? Maybe there is a way to hide a disk device in the installer? Unsure if there is a way to selectively unload the ATA driver (so it is still working for the virtual CD, but no longer visible for the disk - I imagine this logic is in the FreeBSD 10 as well). Maybe there is another setting similar to hw.ata.disk_enable that only applies to optical drives?

Jim, maybe someone more familiar with the pfSense installer can help?

BTW, I emailed a month ago to try and get access to the tools repo, but I haven't got a reply yet.

Have you been granted access to the repo yet? I would hate to see you become discouraged due to this.

stiggle

What are the options for pfSense on Hyper-V now?

I've got access to an Intel Xeon based failover cluster and a standalone AMD based box to test on.

sirrus

Hi - how about 2.1.2 - OpenSSL HearBleed bug is really serious!

Any news? New Image / Release time?

zootie

I'm still working on the ISO. I kind of went back to the drawing board a bit to try and use taliesins changes to get the hv_kvp service working (thank you, taliesins), and to try and figure out why it didn't work with the initial patch. However, given the urgency to get the OpenSSL Heartbleed bug fix in our pfSense on Hyper-V installations, and pfSense 2.1.2 Release yesterday to address it, I decided to test the modules with current options available to get the new Hyper-V drivers working on an updated pfSense installation (so we wouldn't have to wait for an ISO with the hyperv drivers).

I'm working on the ISO, so it is easier to create a new VM install, but even after I finish my changes and test them, I don't know how long it will take to incorporate these changes into the pfSense build process so there is an official release (the changes are small, so it shouldn't take too long). In the meantime, while the links to the preconfigured VMs are no longer available; you can still download the kernel modules using the zip file attached to the first post and follow the Option B instructions.

I tested the modules with 2.1.1 REL and 2.1.2 REL and, as expected, they work fine. I tested using an official pfSense 2.1.2 ISO to install a new VM (initially using Legacy Network Adapters) and then install the drivers (Option B); and I also tested upgrading a VM I had running 2.1.1 Prerelease (already with these drivers) using the snapshot server. Both options worked ok.

Below are slightly improved Option B instructions. No major changes (just using loader.conf.local rather than loader.conf, but it doesn't seem to make a difference right now); I'm just trying to clarify them a bit. If an admin sees this, you might want to update the first post with these updated instructions.

Early on, I took some screenshots, and I was thinking to create a page with more detailed instructions for Option B, but then we got into the issue with the VM distribution, and I figured I should better concentrate on getting the ISO working and getting the changes into the official release. If anyone is inclined, you could write up friendlier instructions (albeit, it might be a matter of days before an ISO release makes these instructions obsolete).

Updated Option B. Add Precompiled Kernel Modules to an Existing pfSense 2.1, 2.1.1, or 2.1.2 VM Installation

Use this option if you want to specify your own setting when configuring the initial pfSense VM (disk size and partitions, memory, etc.)

Download pfSensewHyperv-ics_1.0_KernelModules.zip.txt (attached to the first post) and rename it to remove the .txt extension so you're left with a .zip extension
Extract the files
Create a new VM with 2 Legacy Network Adapters using a pfSense 2.1, 2.1.1, or 2.1.2 ISO downloaded from one of pfSense.org download mirrors - obviously, you want to use a 2.1.2 ISO in order to get the Heartbleed fix. Do not use an ISO with the Summer 2012 drivers - don't use older ISOs created by me or PollyPy or older alexappleton kernels from the older thread
It is recommended you create and configure GEOM labels, as described in Labeling Disk Devices. To do this, first boot in single user mode (option 5 in the boot menu), and if using the default partitioning scheme, use these commands

cat /etc/fstab
/sbin/glabel label rootfs /dev/ad0s1a
/sbin/glabel label swap /dev/ad0s1b
exit

After you type exit (to continue to multi-user mode), don't forget to modify your /etc/fstab to use the labels you created in single user mode above (you can edit /etc/fstab using vi or the WebConfigurator)


/dev/label/rootfs	/		ufs	rw		1	1
/dev/label/swap		none		swap	sw		0	0

Alternatively, if you don't want to create GEOM labels, you can change fstab so it uses the da device rather than ad - but if the storage driver doesn't load, you will have to mount the root filesystem manually on the next reboot


/dev/da0s1a	/		ufs	rw		1	1
/dev/da0s1b	none		swap	sw		0	0

If you're going to use SSH to copy files, you will need to reset the legacy interfaces. If you're using DHCP on the WAN interface connected to de0:

ifconfig de0 down
ifconfig de0 up
dhclient de0
ifconfig de1 down
ifconfig de1 up

Copy the kernel modules into this new VM into /boot/modules. You can use a FAT or FAT32 formatted VHD or (easier) enable SSH on pfSense and use WinSCP to copy the files
Set the file permisions for the modules to executable

chmod +x /boot/modules/hv_*.ko

Edit /boot/loader.conf.local (better than editing /boot/loader.conf) so it loads the modules on startup

hv_vmbus_load="YES"
hv_utils_load="YES"
hv_netvsc_load="YES"
hv_storvsc_load="YES"
hv_ata_pci_disengage_load="YES"

Shutdwon the VM and remove Legacy Network Adapters and add the (normal) non-Legacy Network Adapters and configure them in Hyper-V Manager
Start the VM, and assign interfaces when prompted

stiggle

On the AMD based Hyper-V box (AMD FX 8 core, 16Gb RAM, 5 NIC)

Installed 2.1.2 last night - added the kernel modules. Stable for the last 24 hours.
All the integrated services are enabled, not changed anything in the system tunables.
Not getting any calcru messages (unlike last time).

Won't be getting around to installing on the Xeon box for a week or so though.

But so far - definately working a lot better than the previous build I was using.

tspeirs

Thanks a lot for this. It is appreciated.

2.1.2 Option B.
Server 2012 R2 With Update 1
2 X Intel(R) Xeon(R) CPU E5-2680
WAN - VDSL Modem - PPPOE
LAN - Intel i350 in LACP Dynamic team

Seems to be working well so far. Although the old ISO worked for me too.

The important thing for me was to enable MAC spoofing also on the LAN otherwise I would not have network access on boot sometimes. I had that with the previous release too although I was not affected on another machine using Cable modem with DHCP.

EDIT: UPDATE: I do still seem to have the issue where the network is not available on some boots. Its not the MAC spoofing. I don't know if this is a pfsense bug with PPPOE, it occurred to me. I do have an identical server at another location using the older iso which does not have this issue. Differences is there is no team and WAN is DHCP and not PPPOE. The adapters show up fine and look normal but the networking is broken.

FinDev

Thank you very much!

This version works much better and what's most important it has been stable so far, previous 2.1 with hyper-v (which I download several month ago) was crashing in openvpn (under load) with some mbuf related exception. Now it is stable! The only thing I've noticed is "kernel: hv_kvp_callback: Transaction already active" in system logs, but seems to be not so important, everything still works, also no tunnables anymore :)

Thanks again for your efforts and thanks pfSense team for the great product!

P.S.: looking forward official support too :)

segobi

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!