Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1

stiggle

What are the options for pfSense on Hyper-V now?

I've got access to an Intel Xeon based failover cluster and a standalone AMD based box to test on.

sirrus

Hi - how about 2.1.2 - OpenSSL HearBleed bug is really serious!

Any news? New Image / Release time?

zootie

I'm still working on the ISO. I kind of went back to the drawing board a bit to try and use taliesins changes to get the hv_kvp service working (thank you, taliesins), and to try and figure out why it didn't work with the initial patch. However, given the urgency to get the OpenSSL Heartbleed bug fix in our pfSense on Hyper-V installations, and pfSense 2.1.2 Release yesterday to address it, I decided to test the modules with current options available to get the new Hyper-V drivers working on an updated pfSense installation (so we wouldn't have to wait for an ISO with the hyperv drivers).

I'm working on the ISO, so it is easier to create a new VM install, but even after I finish my changes and test them, I don't know how long it will take to incorporate these changes into the pfSense build process so there is an official release (the changes are small, so it shouldn't take too long). In the meantime, while the links to the preconfigured VMs are no longer available; you can still download the kernel modules using the zip file attached to the first post and follow the Option B instructions.

I tested the modules with 2.1.1 REL and 2.1.2 REL and, as expected, they work fine. I tested using an official pfSense 2.1.2 ISO to install a new VM (initially using Legacy Network Adapters) and then install the drivers (Option B); and I also tested upgrading a VM I had running 2.1.1 Prerelease (already with these drivers) using the snapshot server. Both options worked ok.

Below are slightly improved Option B instructions. No major changes (just using loader.conf.local rather than loader.conf, but it doesn't seem to make a difference right now); I'm just trying to clarify them a bit. If an admin sees this, you might want to update the first post with these updated instructions.

Early on, I took some screenshots, and I was thinking to create a page with more detailed instructions for Option B, but then we got into the issue with the VM distribution, and I figured I should better concentrate on getting the ISO working and getting the changes into the official release. If anyone is inclined, you could write up friendlier instructions (albeit, it might be a matter of days before an ISO release makes these instructions obsolete).

Updated Option B. Add Precompiled Kernel Modules to an Existing pfSense 2.1, 2.1.1, or 2.1.2 VM Installation

Use this option if you want to specify your own setting when configuring the initial pfSense VM (disk size and partitions, memory, etc.)

Download pfSensewHyperv-ics_1.0_KernelModules.zip.txt (attached to the first post) and rename it to remove the .txt extension so you're left with a .zip extension
Extract the files
Create a new VM with 2 Legacy Network Adapters using a pfSense 2.1, 2.1.1, or 2.1.2 ISO downloaded from one of pfSense.org download mirrors - obviously, you want to use a 2.1.2 ISO in order to get the Heartbleed fix. Do not use an ISO with the Summer 2012 drivers - don't use older ISOs created by me or PollyPy or older alexappleton kernels from the older thread
It is recommended you create and configure GEOM labels, as described in Labeling Disk Devices. To do this, first boot in single user mode (option 5 in the boot menu), and if using the default partitioning scheme, use these commands

cat /etc/fstab
/sbin/glabel label rootfs /dev/ad0s1a
/sbin/glabel label swap /dev/ad0s1b
exit

After you type exit (to continue to multi-user mode), don't forget to modify your /etc/fstab to use the labels you created in single user mode above (you can edit /etc/fstab using vi or the WebConfigurator)


/dev/label/rootfs	/		ufs	rw		1	1
/dev/label/swap		none		swap	sw		0	0

Alternatively, if you don't want to create GEOM labels, you can change fstab so it uses the da device rather than ad - but if the storage driver doesn't load, you will have to mount the root filesystem manually on the next reboot


/dev/da0s1a	/		ufs	rw		1	1
/dev/da0s1b	none		swap	sw		0	0

If you're going to use SSH to copy files, you will need to reset the legacy interfaces. If you're using DHCP on the WAN interface connected to de0:

ifconfig de0 down
ifconfig de0 up
dhclient de0
ifconfig de1 down
ifconfig de1 up

Copy the kernel modules into this new VM into /boot/modules. You can use a FAT or FAT32 formatted VHD or (easier) enable SSH on pfSense and use WinSCP to copy the files
Set the file permisions for the modules to executable

chmod +x /boot/modules/hv_*.ko

Edit /boot/loader.conf.local (better than editing /boot/loader.conf) so it loads the modules on startup

hv_vmbus_load="YES"
hv_utils_load="YES"
hv_netvsc_load="YES"
hv_storvsc_load="YES"
hv_ata_pci_disengage_load="YES"

Shutdwon the VM and remove Legacy Network Adapters and add the (normal) non-Legacy Network Adapters and configure them in Hyper-V Manager
Start the VM, and assign interfaces when prompted

stiggle

On the AMD based Hyper-V box (AMD FX 8 core, 16Gb RAM, 5 NIC)

Installed 2.1.2 last night - added the kernel modules. Stable for the last 24 hours.
All the integrated services are enabled, not changed anything in the system tunables.
Not getting any calcru messages (unlike last time).

Won't be getting around to installing on the Xeon box for a week or so though.

But so far - definately working a lot better than the previous build I was using.

tspeirs

Thanks a lot for this. It is appreciated.

2.1.2 Option B.
Server 2012 R2 With Update 1
2 X Intel(R) Xeon(R) CPU E5-2680
WAN - VDSL Modem - PPPOE
LAN - Intel i350 in LACP Dynamic team

Seems to be working well so far. Although the old ISO worked for me too.

The important thing for me was to enable MAC spoofing also on the LAN otherwise I would not have network access on boot sometimes. I had that with the previous release too although I was not affected on another machine using Cable modem with DHCP.

EDIT: UPDATE: I do still seem to have the issue where the network is not available on some boots. Its not the MAC spoofing. I don't know if this is a pfsense bug with PPPOE, it occurred to me. I do have an identical server at another location using the older iso which does not have this issue. Differences is there is no team and WAN is DHCP and not PPPOE. The adapters show up fine and look normal but the networking is broken.

FinDev

Thank you very much!

This version works much better and what's most important it has been stable so far, previous 2.1 with hyper-v (which I download several month ago) was crashing in openvpn (under load) with some mbuf related exception. Now it is stable! The only thing I've noticed is "kernel: hv_kvp_callback: Transaction already active" in system logs, but seems to be not so important, everything still works, also no tunnables anymore :)

Thanks again for your efforts and thanks pfSense team for the great product!

P.S.: looking forward official support too :)

segobi

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!

cpresco

@segobi:

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!

I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.

tspeirs

Just an update on my post above. No stability or performance issues to report although again other than the boot issue with network access on occasion I was also fine with the earlier ISO. That issue does seem much better though. It boots up fine more often than not wheras the old version was just hit or miss and easy to reproduce.

Couple of questions to ask:
Is there an easy way to test performance other than the usual speedtest.net on the WAN which is not ideal test as that can vary a lot.
Also, is there any recommended configuration in regards to hardware acceleration, such as VMQs, offloading etc, both on the WAN (which I my case is connected to a modem) and LAN?

Thanks again for this. I've messed with both Hyper-V and VMWARE in both free and enterprise versions and for me Hyper-V is much better for my use which is only a small software house setup but I do use remoteFX and there is no VMWARE offering that competes identically with that. So I hope you guys continue to work on this.

Thanks.
Tom.

segobi

@cpresco:

@segobi:

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!

I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.

it seems to load the modules now - however non legacy nics are still not found :(

cpresco

@segobi:

@cpresco:

@segobi:

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!

I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.

it seems to load the modules now - however non legacy nics are still not found :(

Were you able to ping the LAN interface with the legacy network adapters? and/or were you able to access the pfsense webConfigurator with the legacy network adapters before doing zootie's Updated Option B steps for the (non-legacy) network adapters?

zootie - I'm just about ready to commit my pfsense 2.1.2 VM in place of my physical pfsense 2.1.2 system. Thanks for the kernel modules and steps in this thread, your work on this is appreciated.

segobi

@cpresco:

@segobi:

@cpresco:

@segobi:

Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.

I added the correct entries into /boot/loader.conf.local

Any help would be appreciated! Thx!

I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.

it seems to load the modules now - however non legacy nics are still not found :(

Were you able to ping the LAN interface with the legacy network adapters? and/or were you able to access the pfsense webConfigurator with the legacy network adapters before doing zootie's Updated Option B steps for the (non-legacy) network adapters?

zootie - I'm just about ready to commit my pfsense 2.1.2 VM in place of my physical pfsense 2.1.2 system. Thanks for the kernel modules and steps in this thread, your work on this is appreciated.

Yes - the legacy adapters were working - I was able to access the system using ssh and webconfigurator.
Seems really strange. It loads the modules now when the entries are in loader.conf - but nothing else chances. Im also getting those calru kernel messages all the time.

I set up pfsense using the latest 2.1.2 64 bit iso.

update: its working when i use the 2.1.1 iso to install pfsense
received some calcru messages but now after everything is set up its working nicely.
maybe the guys who updated 2.1.1 to 2.1.2 had it working ? It seems like fresh installs of 2.1.2 do not work - at least not for me.

cpresco

Since you were able to get 2.1.1 working I'd recommend taking a snapshot then trying to upgrade to 2.1.2 via the webconfigurator.

Spaghetti

Thanks for these HyperV drivers, very useful. However, my hn0 and hn1 NIC's are negotiating at 10Mbps according to SNMP monitoring app.
How can I confirm this within pfSense and ensure that its negotiating at 1000Mbps speeds?

I'm going to re-emphasize that if the community wants an official build of pfSense for Hyper-V, all that is required is that someone who knows what they are doing get in-touch.

Because I don't really have time to learn Hyper-V, but I am interested in making this happen.

doktornotor

@gonzopancho:

I'm going to re-emphasize that if the community wants an official build of pfSense for Hyper-V, all that is required is that someone who knows what they are doing get in-touch.

You mean, like this?

@zootie:

BTW, I emailed a month ago to try and get access to the tools repo, but I haven't got a reply yet.

::) ::) ::) ::) ::)

zootie

I don't claim to be an expert, I just have enough persistence and knowledge of the different techs involved to try and dig for a way to get it working. It just takes longer (and even more patience) to figure it out.

Last week, I got the instructions on how to get into the pfsense-tools repo, but I had some issues (browser compatibility?) and I couldn't get it to work until just now. I can now get the latest version of the tools, I'll just have to reconfigure/recreate my build environment, become familiar with git and figure out how to do pull requests. Or I'll try and summarize the changes here later (it might be easier for someone with more knowledge of git to get the changes committed, they're fairly small).

In the meantime…

I've been working on the ISO-B variation (dynamic load of the modules, detect if using da or ad disk driver and set options on the new installation as appropriate). I've also been trying to get the hv_kvp service working, but there seems to be something amiss with the code itself, since I can't get it to report the IP address back to Hyper-V Manager even after I followed taliesins instructions to try and fixup a running installation or on a clean FreeBSD 8.3 installation (using the pre-compiled modules). There are some recent code changes to this functionality that are specific to FreeBSD 10 (that use APIs introduced in FreeBSD 9.x) and that make the FreeBSD 10 compiled module incompatible with the FreeBSD 8 module (in FreeBSD 10, developers seem to either be renaming the utilities module into hv_kvp or splitting out the kvp functionality), so this might take some more time to figure out (have to do some actual c programming) so we might need to leave it out of an initial Hyper-V edition (or it might have to wait for 2.2 on FreeBSD 10).

@tspeirs:

Is there an easy way to test performance other than the usual speedtest.net on the WAN which is not ideal test as that can vary a lot.
Also, is there any recommended configuration in regards to hardware acceleration, such as VMQs, offloading etc, both on the WAN (which I my case is connected to a modem) and LAN?

(…) I've messed with both Hyper-V and VMWARE in both free and enterprise versions and for me Hyper-V is much better for my use which is only a small software house setup but I do use remoteFX and there is no VMWARE offering that competes identically with that. (...)

@Spaghetti:

How can I confirm this within pfSense and ensure that its negotiating at 1000Mbps speeds?

To confirm stability, rather than a web benchmark, I used SCP to copy files in and out of pfSense. I attached to the VM a preallocated FAT32 disk with a couple gigs of large files (bunch of pfSense ISOs) and then used WinSCP to download the files to a server with a fast disk array. It's an easy benchmark setup to test for 1 Gbe performance (enable SSH on the console menu, use WInSCP to connect using root as the user and your web configurator password). To further stress test it, I also set it up to use an internal switch for the LAN interface to transfer files in and out on different interfaces at the same time (to get 2+ Gbe throughput, copy files into pfSense using SCP on the command line connecting to an SSH server on the "wan" interface, which was connected to my LAN, and use WinSCP from the host to copy files out of pfSense on the LAN interface connected to an internal switch).

On optimizations, in my setups, I've been enabling SR-IOV when creating the virtual switch (using Intel ET adapters teamed using Intel teaming SW). I haven't used this new version on servers running Hyper-V on 2008 R2 (enabling VMQ in the adapter settings) - better to push to upgrade the server to 2012 R2.

While VMWare is more common (it was there first) on established data centers, Hyper-V is a low cost entry point for small departmental and small branch data centers, specially when you only have Windows admins. VMWare's free ESX edition is a bit too limited, many might outgrow it quickly (so it gets expensive sooner), and Hyper-V enlightened drivers help performance (allowing for lighter VMs). It is a shame that RemoteFX HW acceleration doesn't work with Terminal Server (only with VDI Windows desktops).

@segobi:

update: its working when i use the 2.1.1 iso to install pfsense
received some calcru messages but now after everything is set up its working nicely.
maybe the guys who updated 2.1.1 to 2.1.2 had it working ? It seems like fresh installs of 2.1.2 do not work - at least not for me.

Good to hear you got it working (hopefull we'll get an ISO out soon, so it is easier). I don't know why you had so much trouble. I did try both, adding the ko modules to a fresh 2.1.2 install and to upgrade 2.1.1 using the daily build server URL. I see some calcru messages shortly after boot, but few or none afterwards. Setting sysctl kern.timecounter.hardware=TSC doesn't seem to have an effect on my setup (other users have reported that that fixes it, it might be hardware dependent?). For my setup, Using nearby NTP servers seemed to work best to lower calcru messages.

LoZio

@zootie:

I don't claim to be an expert, I just have enough persistence and knowledge of the different techs involved to try and dig for a way to get it working. It just takes longer (and even more patience) to figure it out.

First of all thanks to zootie for his work (and patience). Since there are lots of italian users of both pfSense and Hyper-V, I put together a step by step tutorial in italian to have zootie drivers loaded and running. The tutorial contains pics and descriptions. It can be useful for english speakers too, it is very "visual" and all the products snapshots are from English versions.
Just some questions, mainly for zootie:

looking at this thread, I'm not sure I can redistribute your drivers adding a mirror for them. The first place to search will always be the first post of this thread.
are you going to publish updates to the drivers? Using 2.1.2 on Hyper-V 3 results in "drivers must be updated" (see the tutorial ending) and no IP address indication.

The tutorial is here (il tutorial in italiano è qui) http://goo.gl/oUYtN4

Thanks to all for helping pfSense to run on Hyper-V

kanylbullen

@LoZio:

First of all thanks to zootie for his work (and patience). Since there are lots of italian users of both pfSense and Hyper-V, I put together a step by step tutorial in italian to have zootie drivers loaded and running. The tutorial contains pics and descriptions. It can be useful for english speakers too, it is very "visual" and all the products snapshots are from English versions.
Just some questions, mainly for zootie:

looking at this thread, I'm not sure I can redistribute your drivers adding a mirror for them. The first place to search will always be the first post of this thread.

are you going to publish updates to the drivers? Using 2.1.2 on Hyper-V 3 results in "drivers must be updated" (see the tutorial ending) and no IP address indication.

The tutorial is here (il tutorial in italiano è qui) http://goo.gl/oUYtN4

Thanks to all for helping pfSense to run on Hyper-V

Thanks for the guide, it was very helpful (with google translate)!

I am running 2.1.2 on Hyper-V in production as a router with a gigabit fiber connection, 8 synthetic interfaces and two IPSEC tunnels.

kanylbullen

@gonzopancho:

I'm going to re-emphasize that if the community wants an official build of pfSense for Hyper-V, all that is required is that someone who knows what they are doing get in-touch.

Because I don't really have time to learn Hyper-V, but I am interested in making this happen.

I tried the 2.2-DEV-snapshots for a couple of days on Hyper-V, since BSD 10 has native Hyper-V support. Everything installs without hassle and the basic firewall functions perfectly out of the box. IPSEC doesn't work and there is zero logging done, but I suppose that's because those aren't ready in the snapshots yet?