Hyper-V integration installed with pfSense 2.0.1
-
I've been trying alexappleton method with pfSense 2.0.1, 2.0.2, and 2.1 Beta1 (thank you alexappleton for showing this is possible) and I can see the synthetic drivers working (no need to reset interfaces on startup, LAN interface responds to pings, can shutdown from Hyper-V). However, with all of these versions, I get errors during boot up (sysctl; unknown oid ‘net.enc.out.ipsec_bpf_mask’, ‘net.inet.ipcomp.ipcomp_enable’, etc.) and the firewall rules aren't being loaded (I get "file doesn't exist '/dev/pf'" alert in the WebConfigurator). So far, I haven't been able to create a working router.
Moreover, the github repository being used (https://github.com/FreeBSDonHyper-V/freebsd/wiki/Build-the-kernel-with-the-HyperV-drivers) seems to be specific to FreeBSD 8.2 (and doesn't include fixes post August), so regardless of the version of the local FreeBSD installation being used, you end up creating an 8.2 kernel. Additional work is necessary in order to match the FreeBSD version expected by pfsense (http://doc.pfsense.org/index.php/PfSense_and_FreeBSD_Versions). This likely contributes to the errors I'm seeing, and it would require more research to figure out how to match the correct version. Given that pfSense 2.0.x is based on FreeBSD 8.1 and that 2.1 is based on 8.3, these github instructions might not work very well in most cases.
However, I think I found another way. To be continued.
-
http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso
These are the instructions on how to configure a FreeBSD system to build a custom pfSense ISO image. As far as I can tell, the instructions are current and should work with 2.0.x and 2.1.http://blog.chrisara.com.au/2012/08/hyper-v-integration-components-for_13.html
And here you can find Chris Knight source patch files to add Hyper-V support to FreeBSD 8.2, 8.3, 9.0 and 9.1 Beta. In addition of the patch files, there is a link in the comments (https://github.com/FreeBSDonHyper-V/freebsd/issues/65) that points to a couple minor source code changes that seem to greatly improve TCP performance. I don't know if there have been other patches since August.Considering that the integration services code is not officially compatible with FreeBSD 8.1, I'm first focusing on adding Hyper-V support to the 8.3 kernel to get a working pfSense 2.1 beta. Later we can try and figure out a way to get a Hyper-V 8.2 kernel that works with pfSense 2.0.x (or figure out a way to add Hyper-V support to 8.1).
My current strategy:
- Install a new VM with FreeBSD 8.3 (calling this VM pfBuilder83)
- Get the source files, Chris Knight patch (plus "issue 65" fixes), and compile an 8.3-RELEASE-p5 kernel with Hyper-V support for this FreeBSD VM
- Use this VM to build a custom pfSense 2.1 Beta ISO and test it. The key thing to figure out is how to get the pfSense build process to include the patched sources when it builds the kernel that will go in the ISO.
I'm currently building a custom pfSense 2.1 Beta ISO, but the process takes a while. I'll report back when it's done.
-
@zootie
Please let me know when you're finished, I would like obtain that ISO very much :)
If you agree I will even host it to the public. ;) -
I've been trying to create a pfBuilder VM for 2.1 (using the DevelopersBootStrapAndDevIso procedure), but I haven't been successful. Tried both on FreeBSD 8.3 x86 and x64. It might be 2.1's beta code, or some other issue I haven't found.
Even when I try and build the pfsense ISO w/o making changes, it attempts to build a Hyper-V kernel, but it fails because it is missing the kernel configuration file (and it doesn't work if I put the file manually), so there might be some Hyper-V related code in the source base (albeit not all necessary code is present).
I followed Chris Knight's instructions, and was able to build a Hyper-V enabled kernel for FreeBSD, which seemed to work (and it seemed faster in general), but this kernel doesn't seem to work with pfSense - pretty much the same as with the 8.2 kernel from before, it boots and it recognizes synthetic adapters, but still won't work. Likely there are patches in the pfSense source tree that need to be incorporated into this kernel (maybe the code that supports /dev/pf, given the most glaring error so far?)
I'll keep digging. Next thing I might try is to try and see if I can include pfsense kernel patches into the FreeBSD source, to try and just build a compatible kernel (and avoid the remaining complexity of the ISO build).
It will take a while. If anybody has insight on FreeBSD and/or the DevelopersBootStrapAndDevIso it would be appreciated.
-
Please let us know if you have succeeded :)
I wish you the best of luck 8) -
We just got some hardware in today that's going to be dedicated to a Hyper-V server or two, though we're about to move to a new office and won't be bringing it up until then. We'll be dedicating some time to getting the Hyper-V patches integrated with ours in the next month or two.
-
@cmb:
We'll be dedicating some time to getting the Hyper-V patches integrated with ours in the next month or two.
That's great news, we have a number of installs running on Hyper-V and would benefit from being able to use the proper NICs and being able to shut down an install from the Hyper-V console.
-
Hi All,
I've got my pfSense running the kernel successfully, and also got a FreeBSD box building the pfSense kernel :).Now I'm trying to get an ISO built with the added kernel but hit a bit of a stop on where the kernel should be added. When i look through it seems to compile the kernel as part of the build process, but i guess i need to strip that out and copy the prebuilt kernel in.
Any ideas?
-
@cmb:
We just got some hardware in today that's going to be dedicated to a Hyper-V server or two, though we're about to move to a new office and won't be bringing it up until then. We'll be dedicating some time to getting the Hyper-V patches integrated with ours in the next month or two.
WOOHOOOOO!!!
-
It's great that someone more qualified will be looking into this and into the future. Thank you for letting us know, cmb.
In the meantime, pending more testing, it seems I finally got the pfBuilder to generate a working pfSense 2.1 Beta ISO. It boots w/o major errors on the console, and seems to be working properly (can see hn0, hn1, and can shutdown from Hyper-v). I still have more testing to do, and there a few things still to check:
-
So far, I've only been able to get it working with amd64. Earlier, I got other errors when I tried building an i386 ISO and I haven't tried again. The hyper-v patch might be missing some files for i386.
-
During the build, it couldn't get the sources for syslog-ng. It could be a transient error on a source server or some problem in the beta code. I'll retry to get this package later in the week.
-
On the webConfigurator, it fails to open the System Logs page (Call to undefined function isallowedpage() in /usr/local/www/fbegin.inc), might be a consequence of missing syslog-ng
-
Sometimes on the console, there are warnings that seem to be coming from the webConfigurator PHP ("Warning: sessions_start(): …") - again, this is likely due to beta code or the missing syslog-ng
These are the key steps I followed to get the pfBuilder to build a Hyper-V aware kernel:
1 - Installed FreeBSD 8.3 amd64 release (standard minimal setup, didn't change its kernel and didn't run update-freebsd). If doing this on a Hyper-V VM, don't forget to reset the nic each time you reboot to get connectivity (ifconfig de0 down/up, dhclient de0 - or add it to a startup script, as we have to do with pfSense itself)
2 - Followed the DevelopersBootStrapAndDevIso instructions, including the commands to "Ensure BSDInstaller is sound" (made a copy of the VM before building, so I'd have a backup to revert to).
3 - Did a test build run w/o making changes, and tested that the ISO boots, to make sure that pfBuilder is working properly. After the first build, I started getting some errors on the console just before the login ("libczmq.so.1 not found", "failed to start syslogd", "can't open %%RC_SUBR%%"), this seems to be a side effect of the build process and read somewhere that it's to be expected and can be ignored.
4 - Incorporated the changes to fix the TCP speed issue into Christ Knight's 8.3 patch, and made a patch for ip_output.c to initialize mtu = 0 in function ip_output () - to avoid a compiler warning that would stop the process. Copied both patch files to /home/pfsense/tools/patches/RELENG_8_3 and appended both patches to /home/pfsense/tools/builder_scripts/patches.RELENG_8_3 ("-p1
fbsd83b-hyperv.patch~" and "-p1ip_output.c_mtu.patch~"). Rename attached file extension to zip to get these patches.5 - Modified /home/pfsense/tools/builder_scripts/conf/pfSense_SMP.8 (changed "include GENERIC" to "include HYPERV_VM"). I don't know if there is a better way to incorporate the new kernel options (add another kernel choice to the build and setup?)
6 - Do a cleanup in the builder menu, build ISO.
I'll keep testing it, maybe attempt to transplant this kernel to a 2.0.x install to see how it behaves, or figure out if there is a way to make pfBuilder use a patched 8.2 source to build a 2.0.x ISO (or apply the patch to 8.1 sources).
Depending on testing, I'll maybe post the ISOs or kernel somewhere for a temporary quick fix for anyone looking to early test it (mid term, you might want to have your own pfBuilder so you can incorporate changes to the 2.1 beta, and/or wait for cmb's team to better incorporate the changes into the pfSense source).
-
-
@zootie
Could you share your hyper-v pfSense ISO with us ? -
Success!!!!! :D
I was able to create ISOs with a Hyper-V kernel for both 2.0.x and 2.1 Beta. Both install and show no major errors and seem functional. More testing is needed, but it is a good starting point for all of us needing to have better Hyper-V support in pfSense.
I'll post more details once I get some sleep and go deal with life. In the meantime, I posted the ISOs on RapidShare:
~~http://rapidshare.com/files/1592931654/pfSense-LiveCD-2.0.3-PRERELEASE-amd64-hyperv-kernel-20130119-0048.zipTo get a recent version please see here: http://forum.pfsense.org/index.php/topic,56565.msg362435.html#msg362435
-
I cant download the file, ii get the following error:
Download not available
Download permission denied by uploader. (0b67c2f5) -
Please try again, the download link should be working now.
-
Yes its working thank you very much ;D
-
This is super exciting!!!
-
The RapidShare download links are limited to about 10 downloads per day (if you get an error about over quota, just give it a day to retry). The zip files include the patch files used (just the original and modified Chris Knight patches).
Some details.
On the 2.1 Beta build, no changes in the build process since my prior post. However, it seems there were some fixes checked in the beta code, and it is reporting less errors from PHP and the WebConfigurator. It should be a good testing platform for anyone trying 2.1 under Hyper-V.
On the 2.0.x build. It is using FreeBSD 8.2 (p10) and it reports version 2.0.3 PRERELEASE. The code should be fairly close to 2.0.2 RELEASE. You'd have to try the features you need to make sure it works for you.
After installing 2.0.3, you probably want to fix the missing beep (http://redmine.pfsense.org/issues/2738), if only to avoid the on screen error.
These are the changes I did to get 2.0.x built:
-
Changed the version on the menu and did a build w/o any changes
-
In /home/pfsense/tools/builder_scripts/pfsense-build.conf change FreeBSD 8.1 references to 8.2
-
As with 2.1 Beta, add the TCP fixes to the 8.2 patch file and put it in pfBuilder 8.2 patch directory (/home/pfsense/tools/patches/RELENG_8_2) and add it to pfBuilder 8.2 patch list (/home/pfsense/tools/builder_scripts/patches.RELENG_8_2). Also apply the same ip_output.c mtu intialization patch.
-
As with 2.1 Beta, same changes to /home/pfsense/tools/builder_scripts/conf/pfSense_SMP.8
-
Change /home/pfsense/tools/builder_scripts/pfsense_local.sh so it doesn't build the sfxge module - change
if [ ${FREEBSD_VERSION} -gt 7 -a "$FREEBSD_BRANCH" != "RELENG_8_1" ]; thento```
if [ ${FREEBSD_VERSION} -gt 7 -a "$FREEBSD_BRANCH" != "RELENG_8_1" -a "$FREEBSD_BRANCH" != "RELENG_8_2" ]; then
-
-
I've tested the PRERELEASE ISO and found some things which are not working:
- Traffic Sharping (no interfaces shown)
- not VLan captable (für 2008+2008R2 correct, for 2012 maybe not)
- RDD Graphs empty (both on Dashboard and in Status->RDD Graph)
The following error code is shown in the System Log Tab:
php: : RRD create failed exited with 127, the error is: nice: /usr/local/bin/rrdtool: No such file or directory php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:U:U:U:U' returned exit code '127', the output was 'nice: /usr/local/bin/rrdtool: No such file or directory' apinger: Error while feeding rrdtool: Broken pipe@zootie: please don't take this as a criticism for your work. This is GREAT. This is more some kind of bug/missing feature list for final integration :).
-
I know, we're just trying to get a working set and help the community. Maybe get enough testing done so the drivers make it into 2.1 moving forward. Or someone more familiar with the development process might have more insight what are the barriers to get 2.0.x working on an 8.2 kernel.
As for missing features
-
Don't know why the interfaces don't show up under traffic shaping. Maybe the driver doesn't support needed features (alt-q enabled), as described in http://redmine.pfsense.com/issues/1802
-
I haven't tried using VLANs inside a VM (usually do it on the host). It's something that supposedly can work in WSrv 2012, but I haven't had a chance to try it.
-
Looking for the rrd error in the log, it seems that you only need to reinstall rrdtool (pkg_add -r rrdtool, per http://forum.pfsense.org/index.php?topic=37489.0). It takes a while (it downloads several X related packages) and displays some warnings along the way, but rrd graphs are back and I haven't noticed any ill effects.
Other things I've noticed on 2.0.3
-
Early in the boot process, it displays errors related to ipw_monitor_fw, about it missing legal.intel_ipw.license_ack=1 in /boot/loader.conf to accept the license agreement for this module.
-
Still see "calcru: runtime went backwards" warning, usually shortly after a reboot (but no different from 2.0.2 w/o Hyper-V).
-
It complains about not finding the IPSec PID during boot, and it can take some time to re-establish the first IPSec network connection (if using IPSec Site to Site), but it does establish the connection after a while (might need to reconnect it manually), and otherwise site to site IPSec seems to be working properly.
-
-
- Looking for the rrd error in the log, it seems that you only need to reinstall rrdtool (pkg_add -r rrdtool, per http://forum.pfsense.org/index.php?topic=37489.0). It takes a while (it downloads several X related packages) and displays some warnings along the way, but rrd graphs are back and I haven't noticed any ill effects.
works :).
- Still see "calcru: runtime went backwards" warning, usually shortly after a reboot (but no different from 2.0.2 w/o Hyper-V).
I'm not sure if that is some kind of wanted behaviour. Because this behaviour occours an an XP VM, too. But right after this message the web interface stops respondung and I have to restart it manually.
Edit: if you have an idea how vlans could work on an 2012 machine, let me know :).