Hyper-V integration installed with pfSense 2.0.1
-
Hi All,
I've got my pfSense running the kernel successfully, and also got a FreeBSD box building the pfSense kernel :).Now I'm trying to get an ISO built with the added kernel but hit a bit of a stop on where the kernel should be added. When i look through it seems to compile the kernel as part of the build process, but i guess i need to strip that out and copy the prebuilt kernel in.
Any ideas?
-
@cmb:
We just got some hardware in today that's going to be dedicated to a Hyper-V server or two, though we're about to move to a new office and won't be bringing it up until then. We'll be dedicating some time to getting the Hyper-V patches integrated with ours in the next month or two.
WOOHOOOOO!!!
-
It's great that someone more qualified will be looking into this and into the future. Thank you for letting us know, cmb.
In the meantime, pending more testing, it seems I finally got the pfBuilder to generate a working pfSense 2.1 Beta ISO. It boots w/o major errors on the console, and seems to be working properly (can see hn0, hn1, and can shutdown from Hyper-v). I still have more testing to do, and there a few things still to check:
-
So far, I've only been able to get it working with amd64. Earlier, I got other errors when I tried building an i386 ISO and I haven't tried again. The hyper-v patch might be missing some files for i386.
-
During the build, it couldn't get the sources for syslog-ng. It could be a transient error on a source server or some problem in the beta code. I'll retry to get this package later in the week.
-
On the webConfigurator, it fails to open the System Logs page (Call to undefined function isallowedpage() in /usr/local/www/fbegin.inc), might be a consequence of missing syslog-ng
-
Sometimes on the console, there are warnings that seem to be coming from the webConfigurator PHP ("Warning: sessions_start(): …") - again, this is likely due to beta code or the missing syslog-ng
These are the key steps I followed to get the pfBuilder to build a Hyper-V aware kernel:
1 - Installed FreeBSD 8.3 amd64 release (standard minimal setup, didn't change its kernel and didn't run update-freebsd). If doing this on a Hyper-V VM, don't forget to reset the nic each time you reboot to get connectivity (ifconfig de0 down/up, dhclient de0 - or add it to a startup script, as we have to do with pfSense itself)
2 - Followed the DevelopersBootStrapAndDevIso instructions, including the commands to "Ensure BSDInstaller is sound" (made a copy of the VM before building, so I'd have a backup to revert to).
3 - Did a test build run w/o making changes, and tested that the ISO boots, to make sure that pfBuilder is working properly. After the first build, I started getting some errors on the console just before the login ("libczmq.so.1 not found", "failed to start syslogd", "can't open %%RC_SUBR%%"), this seems to be a side effect of the build process and read somewhere that it's to be expected and can be ignored.
4 - Incorporated the changes to fix the TCP speed issue into Christ Knight's 8.3 patch, and made a patch for ip_output.c to initialize mtu = 0 in function ip_output () - to avoid a compiler warning that would stop the process. Copied both patch files to /home/pfsense/tools/patches/RELENG_8_3 and appended both patches to /home/pfsense/tools/builder_scripts/patches.RELENG_8_3 ("-p1
fbsd83b-hyperv.patch~" and "-p1ip_output.c_mtu.patch~"). Rename attached file extension to zip to get these patches.5 - Modified /home/pfsense/tools/builder_scripts/conf/pfSense_SMP.8 (changed "include GENERIC" to "include HYPERV_VM"). I don't know if there is a better way to incorporate the new kernel options (add another kernel choice to the build and setup?)
6 - Do a cleanup in the builder menu, build ISO.
I'll keep testing it, maybe attempt to transplant this kernel to a 2.0.x install to see how it behaves, or figure out if there is a way to make pfBuilder use a patched 8.2 source to build a 2.0.x ISO (or apply the patch to 8.1 sources).
Depending on testing, I'll maybe post the ISOs or kernel somewhere for a temporary quick fix for anyone looking to early test it (mid term, you might want to have your own pfBuilder so you can incorporate changes to the 2.1 beta, and/or wait for cmb's team to better incorporate the changes into the pfSense source).
-
-
@zootie
Could you share your hyper-v pfSense ISO with us ? -
Success!!!!! :D
I was able to create ISOs with a Hyper-V kernel for both 2.0.x and 2.1 Beta. Both install and show no major errors and seem functional. More testing is needed, but it is a good starting point for all of us needing to have better Hyper-V support in pfSense.
I'll post more details once I get some sleep and go deal with life. In the meantime, I posted the ISOs on RapidShare:
~~http://rapidshare.com/files/1592931654/pfSense-LiveCD-2.0.3-PRERELEASE-amd64-hyperv-kernel-20130119-0048.zipTo get a recent version please see here: http://forum.pfsense.org/index.php/topic,56565.msg362435.html#msg362435
-
I cant download the file, ii get the following error:
Download not available
Download permission denied by uploader. (0b67c2f5) -
Please try again, the download link should be working now.
-
Yes its working thank you very much ;D
-
This is super exciting!!!
-
The RapidShare download links are limited to about 10 downloads per day (if you get an error about over quota, just give it a day to retry). The zip files include the patch files used (just the original and modified Chris Knight patches).
Some details.
On the 2.1 Beta build, no changes in the build process since my prior post. However, it seems there were some fixes checked in the beta code, and it is reporting less errors from PHP and the WebConfigurator. It should be a good testing platform for anyone trying 2.1 under Hyper-V.
On the 2.0.x build. It is using FreeBSD 8.2 (p10) and it reports version 2.0.3 PRERELEASE. The code should be fairly close to 2.0.2 RELEASE. You'd have to try the features you need to make sure it works for you.
After installing 2.0.3, you probably want to fix the missing beep (http://redmine.pfsense.org/issues/2738), if only to avoid the on screen error.
These are the changes I did to get 2.0.x built:
-
Changed the version on the menu and did a build w/o any changes
-
In /home/pfsense/tools/builder_scripts/pfsense-build.conf change FreeBSD 8.1 references to 8.2
-
As with 2.1 Beta, add the TCP fixes to the 8.2 patch file and put it in pfBuilder 8.2 patch directory (/home/pfsense/tools/patches/RELENG_8_2) and add it to pfBuilder 8.2 patch list (/home/pfsense/tools/builder_scripts/patches.RELENG_8_2). Also apply the same ip_output.c mtu intialization patch.
-
As with 2.1 Beta, same changes to /home/pfsense/tools/builder_scripts/conf/pfSense_SMP.8
-
Change /home/pfsense/tools/builder_scripts/pfsense_local.sh so it doesn't build the sfxge module - change
if [ ${FREEBSD_VERSION} -gt 7 -a "$FREEBSD_BRANCH" != "RELENG_8_1" ]; thento```
if [ ${FREEBSD_VERSION} -gt 7 -a "$FREEBSD_BRANCH" != "RELENG_8_1" -a "$FREEBSD_BRANCH" != "RELENG_8_2" ]; then
-
-
I've tested the PRERELEASE ISO and found some things which are not working:
- Traffic Sharping (no interfaces shown)
- not VLan captable (für 2008+2008R2 correct, for 2012 maybe not)
- RDD Graphs empty (both on Dashboard and in Status->RDD Graph)
The following error code is shown in the System Log Tab:
php: : RRD create failed exited with 127, the error is: nice: /usr/local/bin/rrdtool: No such file or directory php: : The command '/usr/bin/nice -n20 /usr/local/bin/rrdtool update /var/db/rrd/wan-traffic.rrd N:U:U:U:U' returned exit code '127', the output was 'nice: /usr/local/bin/rrdtool: No such file or directory' apinger: Error while feeding rrdtool: Broken pipe@zootie: please don't take this as a criticism for your work. This is GREAT. This is more some kind of bug/missing feature list for final integration :).
-
I know, we're just trying to get a working set and help the community. Maybe get enough testing done so the drivers make it into 2.1 moving forward. Or someone more familiar with the development process might have more insight what are the barriers to get 2.0.x working on an 8.2 kernel.
As for missing features
-
Don't know why the interfaces don't show up under traffic shaping. Maybe the driver doesn't support needed features (alt-q enabled), as described in http://redmine.pfsense.com/issues/1802
-
I haven't tried using VLANs inside a VM (usually do it on the host). It's something that supposedly can work in WSrv 2012, but I haven't had a chance to try it.
-
Looking for the rrd error in the log, it seems that you only need to reinstall rrdtool (pkg_add -r rrdtool, per http://forum.pfsense.org/index.php?topic=37489.0). It takes a while (it downloads several X related packages) and displays some warnings along the way, but rrd graphs are back and I haven't noticed any ill effects.
Other things I've noticed on 2.0.3
-
Early in the boot process, it displays errors related to ipw_monitor_fw, about it missing legal.intel_ipw.license_ack=1 in /boot/loader.conf to accept the license agreement for this module.
-
Still see "calcru: runtime went backwards" warning, usually shortly after a reboot (but no different from 2.0.2 w/o Hyper-V).
-
It complains about not finding the IPSec PID during boot, and it can take some time to re-establish the first IPSec network connection (if using IPSec Site to Site), but it does establish the connection after a while (might need to reconnect it manually), and otherwise site to site IPSec seems to be working properly.
-
-
- Looking for the rrd error in the log, it seems that you only need to reinstall rrdtool (pkg_add -r rrdtool, per http://forum.pfsense.org/index.php?topic=37489.0). It takes a while (it downloads several X related packages) and displays some warnings along the way, but rrd graphs are back and I haven't noticed any ill effects.
works :).
- Still see "calcru: runtime went backwards" warning, usually shortly after a reboot (but no different from 2.0.2 w/o Hyper-V).
I'm not sure if that is some kind of wanted behaviour. Because this behaviour occours an an XP VM, too. But right after this message the web interface stops respondung and I have to restart it manually.
Edit: if you have an idea how vlans could work on an 2012 machine, let me know :).
-
Hyper-V as a host will allow vlans connected to different virtual adapters on 2008 version
It however requires that the physical nic that is connected to the virtual switch is Vlan enabled.Hyper-V as a host will allow Vlan trunking on one virtual adapter in 2012 Version.
Edit: Don't do advanced networking before 10 a clock :)
I have a testbox running now and it seems to work. Now it's just a matter of getting this into the stable release :) -
I have a testbox running now and it seems to work. Now it's just a matter of getting this into the stable release Smiley
With a pfSense box? In my case the console configuration script says me that there are no vlan captable interfaces.
Same config is running with a Windows Server 2012 so the Hyper-V config seems to be correct. -
Thank you for this :)
2.1 on Hyper-V 2012, I have 3 vNics (vlans set on host), PPPoE and DHCP WAN's load balanced plus IPSec VPN and all seems much like my physical 2.0.1 box. I can live migrate it between Hyper-V hosts with only one dropped packet at the clients.
I used "sysctl kern.timecounter.hardware=TSC" to fix the calc runtime error.
Only potential issue I have is spikes in CPU usage despite no activity, plus I'm only 60/20 VDSL and its on a Xeon E5620 host so plenty of power.
-
I have a testbox running now and it seems to work. Now it's just a matter of getting this into the stable release Smiley
With a pfSense box? In my case the console configuration script says me that there are no vlan captable interfaces.
Same config is running with a Windows Server 2012 so the Hyper-V config seems to be correct.Yes and no :)
2008R2SP1 as base OS running a Hyper-V server.
I do feed that server with Vlans so my virtual Switch called trunk has a number of VLans on it.When i Created my virtual machine fpr PFSense i gave it to Network cards attached to the Trunk Vswitch.
On for Vlan 1 and one for Vlan 666 (guess wich is the Evil internet)In Pfsense I get two non-vlan capable Nics witch i Assaign to Internal and Wan.
If i try to use Vlan here I get the same message as you did.Im not sure if the nic driver for Hyper-v is Capable of Vlans. The BSD driver has the function if it's of the right version but it didn't have it from the beggining.
-
Thak you for your explanation. That's how I use pfSene at the moment.
-
I am running this is On a Windows 2012 Hyper-V host and are seeing some errors when I start the pfSense VM
A storage device in 'pfSense' loaded but has a different version from the server. Server version 5.1 Client version 2.0
Networking driver in pfSense loaded but has a different version from the server. Server version 4.0 Client version 3.2
Also there is an error later after about an 10-28 hour period of running perfectly that the adapter connected to the Hyper-V switch disconnects (mostly the WAN side but the LAN side also) The only way to correct the issue is to restart the Hyper-V Host. I have tried disconnecting the cables and replacing the cables, disabling and enabling the network cards associated to those virtual switches, upgrading the Intel 82574L drivers to Intel’s latest, and disabled most settings in different combinations to see if a setting was to blame. Still the network adapter seems to disappear randomly. Also tried a different network card fearing the integrated NIC was faulty. Still no joy
Is there any plans from Microsoft to release the updated drivers for FreeBSD on Hyper-V and if so zootie do you think you can update the ISOs that you created because this is the best integration I have seen with Hyper-V
Shout out to zootie for the time he took to make the ISO
-
There are maybe plans because the driver is in beta state. For that time you maybe should enable MAC Adress spoofing at the HyperV Network configuration interface.
With Server 2008 R2 there are no Problems.I used "sysctl kern.timecounter.hardware=TSC" to fix the calc runtime error.
This resolves the error outputs but makes the pfSense clock run. I get +8 hours a day.