Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1
-
Thanks @zootie! looking forward to your release.
-
I would suggest to use 2.2 for this effort.
Its aim is just to move to FreeBSD 10 with small effort.That means try to release it ASAP.
Probably your work there is simpler since most of the thing is there or tell us if anything missing, for now. -
Good news that pfSense 2.1.1 is now released. I'll try and test the modules with it and report back (but they should work using Option B work w/o needing to recompile).
Back to building an ISO with integrated drivers. The key element to be able to get the live CD to work was to set hw.ata.disk_enable so it wouldn't disconnect the optical drive from the live CD during boot (for more info, see "Both FastIDE and CD-Rom work in FreeBSD 10 on Hyper-V (…)" and "svn commit: r252645 (…) stordisengage storvsc vmbus").
As of my last testing (a few weeks ago), I had 2 variations of the ISO (both requiring minor changes to the pfsense builder scripts):
-
ISO-A - Changing build_iso.sh so it would set hw.ata.disk_enable and loading the ko drivers in the ISO's loader.conf.local
-
ISO-B - Changing rc.cdrom and detecting if it is running under Hyper-V and then loading the modules on demand and then apply the loader.conf.local changes to the installation
ISO-A is the simplest and most direct, but when I tested it on real HW, it would cause issues (the boot process would get disconnected from the CD), and it wouldn't necessarily setup pfSense on the direct access disk driver (it uses ad0 rather than da0 storage device). ISO-A would likely require having a "Hyper-V only edition ISO" (it doesn't require forking the source, just adding an optional option to make the Hyper-V changes).
ISO-B is more dynamic, and it only loads the drivers when it is running under Hyper-V (which I imagine is that FreeBSD 10's installer does) and then fixes up the new installation by loading the modules, so it wouldn't require a Hyper-V only ISO edition. However, it would allow both the ATA and direct access storage drivers to be visible from pfSense installer (you'd see both ad0 and da0 in the installer - see attached screenshot), so it could be a little confusing (and it might require some editing of loader.conf and/or creating labels afterwards if you want to switch drivers afterwards).
I'm thinking that as part of the fixup in ISO-B, the script could check what disk device driver is being used in the new install's fstab, and then only set hw.ata.disk_enable when the ad0 drivers is being used (or not even load the storage driver), but it seems non-optimal (the installer would still see the 2 disk devices during setup, maybe confusing users). I'll have to try it, and lacking a better alternative, it would work ok (IMO, better than having a Hyper-V only edition)
What would we prefer? A separate ISO only Hyper-V? Maybe there is a way to hide a disk device in the installer? Unsure if there is a way to selectively unload the ATA driver (so it is still working for the virtual CD, but no longer visible for the disk - I imagine this logic is in the FreeBSD 10 as well). Maybe there is another setting similar to hw.ata.disk_enable that only applies to optical drives?
Jim, maybe someone more familiar with the pfSense installer can help?
BTW, I emailed a month ago to try and get access to the tools repo, but I haven't got a reply yet.
Have you been granted access to the repo yet? I would hate to see you become discouraged due to this.
-
-
What are the options for pfSense on Hyper-V now?
I've got access to an Intel Xeon based failover cluster and a standalone AMD based box to test on.
-
Hi - how about 2.1.2 - OpenSSL HearBleed bug is really serious!
Any news? New Image / Release time?
-
I'm still working on the ISO. I kind of went back to the drawing board a bit to try and use taliesins changes to get the hv_kvp service working (thank you, taliesins), and to try and figure out why it didn't work with the initial patch. However, given the urgency to get the OpenSSL Heartbleed bug fix in our pfSense on Hyper-V installations, and pfSense 2.1.2 Release yesterday to address it, I decided to test the modules with current options available to get the new Hyper-V drivers working on an updated pfSense installation (so we wouldn't have to wait for an ISO with the hyperv drivers).
I'm working on the ISO, so it is easier to create a new VM install, but even after I finish my changes and test them, I don't know how long it will take to incorporate these changes into the pfSense build process so there is an official release (the changes are small, so it shouldn't take too long). In the meantime, while the links to the preconfigured VMs are no longer available; you can still download the kernel modules using the zip file attached to the first post and follow the Option B instructions.
I tested the modules with 2.1.1 REL and 2.1.2 REL and, as expected, they work fine. I tested using an official pfSense 2.1.2 ISO to install a new VM (initially using Legacy Network Adapters) and then install the drivers (Option B); and I also tested upgrading a VM I had running 2.1.1 Prerelease (already with these drivers) using the snapshot server. Both options worked ok.
Below are slightly improved Option B instructions. No major changes (just using loader.conf.local rather than loader.conf, but it doesn't seem to make a difference right now); I'm just trying to clarify them a bit. If an admin sees this, you might want to update the first post with these updated instructions.
Early on, I took some screenshots, and I was thinking to create a page with more detailed instructions for Option B, but then we got into the issue with the VM distribution, and I figured I should better concentrate on getting the ISO working and getting the changes into the official release. If anyone is inclined, you could write up friendlier instructions (albeit, it might be a matter of days before an ISO release makes these instructions obsolete).
Updated Option B. Add Precompiled Kernel Modules to an Existing pfSense 2.1, 2.1.1, or 2.1.2 VM Installation
Use this option if you want to specify your own setting when configuring the initial pfSense VM (disk size and partitions, memory, etc.)
-
Download pfSensewHyperv-ics_1.0_KernelModules.zip.txt (attached to the first post) and rename it to remove the .txt extension so you're left with a .zip extension
-
Extract the files
-
Create a new VM with 2 Legacy Network Adapters using a pfSense 2.1, 2.1.1, or 2.1.2 ISO downloaded from one of pfSense.org download mirrors - obviously, you want to use a 2.1.2 ISO in order to get the Heartbleed fix. Do not use an ISO with the Summer 2012 drivers - don't use older ISOs created by me or PollyPy or older alexappleton kernels from the older thread
-
It is recommended you create and configure GEOM labels, as described in Labeling Disk Devices. To do this, first boot in single user mode (option 5 in the boot menu), and if using the default partitioning scheme, use these commands
cat /etc/fstab /sbin/glabel label rootfs /dev/ad0s1a /sbin/glabel label swap /dev/ad0s1b exit- After you type exit (to continue to multi-user mode), don't forget to modify your /etc/fstab to use the labels you created in single user mode above (you can edit /etc/fstab using vi or the WebConfigurator)
/dev/label/rootfs / ufs rw 1 1 /dev/label/swap none swap sw 0 0- Alternatively, if you don't want to create GEOM labels, you can change fstab so it uses the da device rather than ad - but if the storage driver doesn't load, you will have to mount the root filesystem manually on the next reboot
/dev/da0s1a / ufs rw 1 1 /dev/da0s1b none swap sw 0 0- If you're going to use SSH to copy files, you will need to reset the legacy interfaces. If you're using DHCP on the WAN interface connected to de0:
ifconfig de0 down ifconfig de0 up dhclient de0 ifconfig de1 down ifconfig de1 up-
Copy the kernel modules into this new VM into /boot/modules. You can use a FAT or FAT32 formatted VHD or (easier) enable SSH on pfSense and use WinSCP to copy the files
-
Set the file permisions for the modules to executable
chmod +x /boot/modules/hv_*.ko- Edit /boot/loader.conf.local (better than editing /boot/loader.conf) so it loads the modules on startup
hv_vmbus_load="YES" hv_utils_load="YES" hv_netvsc_load="YES" hv_storvsc_load="YES" hv_ata_pci_disengage_load="YES"-
Shutdwon the VM and remove Legacy Network Adapters and add the (normal) non-Legacy Network Adapters and configure them in Hyper-V Manager
-
Start the VM, and assign interfaces when prompted
-
-
On the AMD based Hyper-V box (AMD FX 8 core, 16Gb RAM, 5 NIC)
Installed 2.1.2 last night - added the kernel modules. Stable for the last 24 hours.
All the integrated services are enabled, not changed anything in the system tunables.
Not getting any calcru messages (unlike last time).Won't be getting around to installing on the Xeon box for a week or so though.
But so far - definately working a lot better than the previous build I was using.
-
Thanks a lot for this. It is appreciated.
2.1.2 Option B.
Server 2012 R2 With Update 1
2 X Intel(R) Xeon(R) CPU E5-2680
WAN - VDSL Modem - PPPOE
LAN - Intel i350 in LACP Dynamic teamSeems to be working well so far. Although the old ISO worked for me too.
The important thing for me was to enable MAC spoofing also on the LAN otherwise I would not have network access on boot sometimes. I had that with the previous release too although I was not affected on another machine using Cable modem with DHCP.
EDIT: UPDATE: I do still seem to have the issue where the network is not available on some boots. Its not the MAC spoofing. I don't know if this is a pfsense bug with PPPOE, it occurred to me. I do have an identical server at another location using the older iso which does not have this issue. Differences is there is no team and WAN is DHCP and not PPPOE. The adapters show up fine and look normal but the networking is broken.
-
Thank you very much!
This version works much better and what's most important it has been stable so far, previous 2.1 with hyper-v (which I download several month ago) was crashing in openvpn (under load) with some mbuf related exception. Now it is stable! The only thing I've noticed is "kernel: hv_kvp_callback: Transaction already active" in system logs, but seems to be not so important, everything still works, also no tunnables anymore :)
Thanks again for your efforts and thanks pfSense team for the great product!
P.S.: looking forward official support too :)
-
Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.I added the correct entries into /boot/loader.conf.local
Any help would be appreciated! Thx!
-
Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.I added the correct entries into /boot/loader.conf.local
Any help would be appreciated! Thx!
I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.
-
Just an update on my post above. No stability or performance issues to report although again other than the boot issue with network access on occasion I was also fine with the earlier ISO. That issue does seem much better though. It boots up fine more often than not wheras the old version was just hit or miss and easy to reproduce.
Couple of questions to ask:
Is there an easy way to test performance other than the usual speedtest.net on the WAN which is not ideal test as that can vary a lot.
Also, is there any recommended configuration in regards to hardware acceleration, such as VMQs, offloading etc, both on the WAN (which I my case is connected to a modem) and LAN?Thanks again for this. I've messed with both Hyper-V and VMWARE in both free and enterprise versions and for me Hyper-V is much better for my use which is only a small software house setup but I do use remoteFX and there is no VMWARE offering that competes identically with that. So I hope you guys continue to work on this.
Thanks.
Tom. -
Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.I added the correct entries into /boot/loader.conf.local
Any help would be appreciated! Thx!
I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.
it seems to load the modules now - however non legacy nics are still not found :(
-
Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.I added the correct entries into /boot/loader.conf.local
Any help would be appreciated! Thx!
I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.
it seems to load the modules now - however non legacy nics are still not found :(
Were you able to ping the LAN interface with the legacy network adapters? and/or were you able to access the pfsense webConfigurator with the legacy network adapters before doing zootie's Updated Option B steps for the (non-legacy) network adapters?
zootie - I'm just about ready to commit my pfsense 2.1.2 VM in place of my physical pfsense 2.1.2 system. Thanks for the kernel modules and steps in this thread, your work on this is appreciated.
-
Hmm..getting "unsupported file layout" when trying to manually load the Modules using kldload.
They also do not load on boot - at least thats what kldstat says and it still does not find non legacy NICs.I added the correct entries into /boot/loader.conf.local
Any help would be appreciated! Thx!
I had also created /boot/loader.conf.local and the drivers didn't appear to load so I added the entries to /boot/loader.conf and the drivers loaded without an issue.
it seems to load the modules now - however non legacy nics are still not found :(
Were you able to ping the LAN interface with the legacy network adapters? and/or were you able to access the pfsense webConfigurator with the legacy network adapters before doing zootie's Updated Option B steps for the (non-legacy) network adapters?
zootie - I'm just about ready to commit my pfsense 2.1.2 VM in place of my physical pfsense 2.1.2 system. Thanks for the kernel modules and steps in this thread, your work on this is appreciated.
Yes - the legacy adapters were working - I was able to access the system using ssh and webconfigurator.
Seems really strange. It loads the modules now when the entries are in loader.conf - but nothing else chances. Im also getting those calru kernel messages all the time.I set up pfsense using the latest 2.1.2 64 bit iso.
update: its working when i use the 2.1.1 iso to install pfsense
received some calcru messages but now after everything is set up its working nicely.
maybe the guys who updated 2.1.1 to 2.1.2 had it working ? It seems like fresh installs of 2.1.2 do not work - at least not for me. -
Since you were able to get 2.1.1 working I'd recommend taking a snapshot then trying to upgrade to 2.1.2 via the webconfigurator.
-
Thanks for these HyperV drivers, very useful. However, my hn0 and hn1 NIC's are negotiating at 10Mbps according to SNMP monitoring app.
How can I confirm this within pfSense and ensure that its negotiating at 1000Mbps speeds? -
I'm going to re-emphasize that if the community wants an official build of pfSense for Hyper-V, all that is required is that someone who knows what they are doing get in-touch.
Because I don't really have time to learn Hyper-V, but I am interested in making this happen.
-
@gonzopancho:
I'm going to re-emphasize that if the community wants an official build of pfSense for Hyper-V, all that is required is that someone who knows what they are doing get in-touch.
You mean, like this?
BTW, I emailed a month ago to try and get access to the tools repo, but I haven't got a reply yet.
::) ::) ::) ::) ::)
-
I don't claim to be an expert, I just have enough persistence and knowledge of the different techs involved to try and dig for a way to get it working. It just takes longer (and even more patience) to figure it out.
Last week, I got the instructions on how to get into the pfsense-tools repo, but I had some issues (browser compatibility?) and I couldn't get it to work until just now. I can now get the latest version of the tools, I'll just have to reconfigure/recreate my build environment, become familiar with git and figure out how to do pull requests. Or I'll try and summarize the changes here later (it might be easier for someone with more knowledge of git to get the changes committed, they're fairly small).
In the meantime…
I've been working on the ISO-B variation (dynamic load of the modules, detect if using da or ad disk driver and set options on the new installation as appropriate). I've also been trying to get the hv_kvp service working, but there seems to be something amiss with the code itself, since I can't get it to report the IP address back to Hyper-V Manager even after I followed taliesins instructions to try and fixup a running installation or on a clean FreeBSD 8.3 installation (using the pre-compiled modules). There are some recent code changes to this functionality that are specific to FreeBSD 10 (that use APIs introduced in FreeBSD 9.x) and that make the FreeBSD 10 compiled module incompatible with the FreeBSD 8 module (in FreeBSD 10, developers seem to either be renaming the utilities module into hv_kvp or splitting out the kvp functionality), so this might take some more time to figure out (have to do some actual c programming) so we might need to leave it out of an initial Hyper-V edition (or it might have to wait for 2.2 on FreeBSD 10).
Is there an easy way to test performance other than the usual speedtest.net on the WAN which is not ideal test as that can vary a lot.
Also, is there any recommended configuration in regards to hardware acceleration, such as VMQs, offloading etc, both on the WAN (which I my case is connected to a modem) and LAN?(…) I've messed with both Hyper-V and VMWARE in both free and enterprise versions and for me Hyper-V is much better for my use which is only a small software house setup but I do use remoteFX and there is no VMWARE offering that competes identically with that. (...)
How can I confirm this within pfSense and ensure that its negotiating at 1000Mbps speeds?
To confirm stability, rather than a web benchmark, I used SCP to copy files in and out of pfSense. I attached to the VM a preallocated FAT32 disk with a couple gigs of large files (bunch of pfSense ISOs) and then used WinSCP to download the files to a server with a fast disk array. It's an easy benchmark setup to test for 1 Gbe performance (enable SSH on the console menu, use WInSCP to connect using root as the user and your web configurator password). To further stress test it, I also set it up to use an internal switch for the LAN interface to transfer files in and out on different interfaces at the same time (to get 2+ Gbe throughput, copy files into pfSense using SCP on the command line connecting to an SSH server on the "wan" interface, which was connected to my LAN, and use WinSCP from the host to copy files out of pfSense on the LAN interface connected to an internal switch).
On optimizations, in my setups, I've been enabling SR-IOV when creating the virtual switch (using Intel ET adapters teamed using Intel teaming SW). I haven't used this new version on servers running Hyper-V on 2008 R2 (enabling VMQ in the adapter settings) - better to push to upgrade the server to 2012 R2.
While VMWare is more common (it was there first) on established data centers, Hyper-V is a low cost entry point for small departmental and small branch data centers, specially when you only have Windows admins. VMWare's free ESX edition is a bit too limited, many might outgrow it quickly (so it gets expensive sooner), and Hyper-V enlightened drivers help performance (allowing for lighter VMs). It is a shame that RemoteFX HW acceleration doesn't work with Terminal Server (only with VDI Windows desktops).
update: its working when i use the 2.1.1 iso to install pfsense
received some calcru messages but now after everything is set up its working nicely.
maybe the guys who updated 2.1.1 to 2.1.2 had it working ? It seems like fresh installs of 2.1.2 do not work - at least not for me.Good to hear you got it working (hopefull we'll get an ISO out soon, so it is easier). I don't know why you had so much trouble. I did try both, adding the ko modules to a fresh 2.1.2 install and to upgrade 2.1.1 using the daily build server URL. I see some calcru messages shortly after boot, but few or none afterwards. Setting sysctl kern.timecounter.hardware=TSC doesn't seem to have an effect on my setup (other users have reported that that fixes it, it might be hardware dependent?). For my setup, Using nearby NTP servers seemed to work best to lower calcru messages.