Fix a Hyper-V Synthetic Driver Issue US$500
-
I'm trying to get Hyper-V synthetic drivers working to get a better throughput on my 25 pfSense instances that are all running under Hyper-V.
I have followed instructions here:
http://alexappleton.net/post/37410981279/pfsense-in-hyper-v-with-integration-componentsBut it doesn't work - probably because i'm running 2.1-RELEASE. It all looks like it's worked, and the machine boots and gets to the menu, but it won't pass traffic.
Problems:-
1. This pops up a couple of times during boot: unknown oid ‘net.inet.ipcomp.ipcomp_enable’
2. Notices appear saying: [There were error(s) loading the rules: pfctl: /dev/pf No such file or directory - The line in question reads [0]: ]For the bounty i need to have a 2.1-RELEASE version of pfSense running the HyperV drivers and able to push a throughput that exceeds 100mb (the current limit of the Legacy network adapter).
-
I'm trying to get Hyper-V synthetic drivers working to get a better throughput on my 25 pfSense instances that are all running under Hyper-V.
I have followed instructions here:
http://alexappleton.net/post/37410981279/pfsense-in-hyper-v-with-integration-componentsBut it doesn't work - probably because i'm running 2.1-RELEASE. It all looks like it's worked, and the machine boots and gets to the menu, but it won't pass traffic.
Problems:-
1. This pops up a couple of times during boot: unknown oid ‘net.inet.ipcomp.ipcomp_enable’
2. Notices appear saying: [There were error(s) loading the rules: pfctl: /dev/pf No such file or directory - The line in question reads [0]: ]For the bounty i need to have a 2.1-RELEASE version of pfSense running the HyperV drivers and able to push a throughput that exceeds 100mb (the current limit of the Legacy network adapter).
1. What release source did you compile with 8.2 like the guide? or 8.3? pFsense 2.1 runs on 8.3
2. Were there any compiler warnings
3. Did your original post get deleted about this subject?
4. Have you tried the driver I compiled here after seeing your original posthttps://forum.pfsense.org/index.php/topic,72081.0.html -
I apologise for removing the first post - i thought i had solved the problem myself… but no.
1. I compiled 8.3 - matching the Kernel version of 2-1-Release.
2. I did not notice any compiler warnings, but i'm not 100% there weren't any now.
3. Yes, I removed it - I thought I had done so before anyone saw it, Views was still '1' - sorry about that.
4. I haven't tried your driver (hadn't seen that post) - but i will attempt to do so now.I'm not hugely experienced in the workings of BSD, apologies for that.
-
I've given it a try. No luck just yet.
This is what I did:
1. Reverted out previous kernel changes to take me back to a standard 2-1-Release.
2. Downloaded your file into /boot/modules
3. chmod +x
4. Added to loader.conf.local
5. Rebooted
–Nothing at this point, no new network cards--
6. Tried kldload hv_netvsc.ko and got this:KLD hv_netvsc.ko: depends on vmbus - not available or version mismatch
linker_load_file: Unsupported file type
kldload: can't load hv_netvsc.ko: Exec format errorRegards,
-Carl. -
I've given it a try. No luck just yet.
This is what I did:
1. Reverted out previous kernel changes to take me back to a standard 2-1-Release.
2. Downloaded your file into /boot/modules
3. chmod +x
4. Added to loader.conf.local
5. Rebooted
–Nothing at this point, no new network cards--
6. Tried kldload hv_netvsc.ko and got this:KLD hv_netvsc.ko: depends on vmbus - not available or version mismatch
linker_load_file: Unsupported file type
kldload: can't load hv_netvsc.ko: Exec format errorRegards,
-Carl.move this one to /boot/modules and load it first kldload hv_vmbus.ko then kldload hv_netvsc.ko
-
Getting Close! After the two KLDLoad statements, i can see HyperV network interfaces in ifconfig. Not had a chance to test yet, i'll do that tonight.
However - it's not working on boot. I added these into loader.conf.local
hv_vmbus_load="YES"
hv_netvsc_load="YES"First couple of boots is fine - we have 4 network cards, 2 legacy (mapped to lan/wan) and 2 HV cards.
Then i remove the legacy cards and reboot….And now it crashes on boot and reboots itself.
It reboots just shortly after the question about wanting to remap the network cards pops up. The crash is very quick, can't catch it's text.So, i tried booting with 4 cards connected, and then manually reassigning the adapters using option (1). I did this, and as soon as i confirmed my changes are OK, same crash. And now it crashes every boot.
One interesting thing to note is that during the assigning of the adapters, the two HyperV adapters reported that they are 'down', even though they are not.
Finally i remove the hyperV adapters from the machine, and it boots happily and i can reassign back to the 'de' legacy adapters.
-
I just tried another test.
I booted up with 4x network cards as before.
I then manually loaded the kernel modules.
i then used ifconfig to assign an IP address…. and it crashed. same crash.kldload hv_vmbus.ko
kldload hv_netvsc.ko
ifconfig hn0 1.2.3.4 255.255.255.0
**crash **it only appears for a sec so i can't capture it...but i got it from the logs:
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
vmbus0: [FILTER]
padlock0: No ACE support.
Netvsc probe… DONE
hn0: <synthetic network="" interface="">on vmbus0
Netvsc initializing... done!
Netvsc probe... DONE
hn1: <synthetic network="" interface="">on vmbus0
Netvsc initializing... Already initialized!
Netvsc initializing... Already initialized!Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x800
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8194d472
stack pointer = 0x28:0xffffff8027955970
frame pointer = 0x28:0xffffff80279559f0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: hv_event)
version.txt06000024112272726133 7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic> -
I just tried another test.
I booted up with 4x network cards as before.
I then manually loaded the kernel modules.
i then used ifconfig to assign an IP address…. and it crashed. same crash.kldload hv_vmbus.ko
kldload hv_netvsc.ko
ifconfig hn0 1.2.3.4 255.255.255.0
**crash **it only appears for a sec so i can't capture it...but i got it from the logs:
Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
vmbus0: [FILTER]
padlock0: No ACE support.
Netvsc probe… DONE
hn0: <synthetic network="" interface="">on vmbus0
Netvsc initializing... done!
Netvsc probe... DONE
hn1: <synthetic network="" interface="">on vmbus0
Netvsc initializing... Already initialized!
Netvsc initializing... Already initialized!Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x800
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff8194d472
stack pointer = 0x28:0xffffff8027955970
frame pointer = 0x28:0xffffff80279559f0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: hv_event)
version.txt06000024112272726133 7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic>Is there anymore log information avaliable? Can you run "bt" or "backtrace" after it crashes.
-
BT and Backtrace are command not found. But here is the entire log from the web front end:
Crash report is over max size for the forum, so it's here.
https://drive.google.com/file/d/0BwOWfIoqpmmfZ25LQmlTemVpVU0/edit?usp=sharing -
Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.
I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver. Something connected to this? I'm just speculating.
-C
-
Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.
I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver. Something connected to this? I'm just speculating.
-C
I don't think so as the whole package does a bunch of stuff. This is just the vmbus and the netsvc driver. I can try with the older driver source next. This was from current source. I would have to do a lot of file comparing to see where abouts the issue is with the vmbus driver. Does it survive a ifconfig interface up? Or is it just crashing on assign
-
I just tried 'ifconfig hn0 up'…. instant death.
-
I just tried 'ifconfig hn0 up'…. instant death.
You may just want to wait until the next pfsense release as the hyperv drivers may potentially be built in.
Well here is compiled from 8.2
-
https://forum.pfsense.org/index.php/topic,56565.msg393651.html#new
Bill seems to be up and running
-
I'm also up and running, with that same image…. BUT ....
Every time i reboot my box, the interfaces swap! hv0 and hv1 literally swap.
Any clues on how they are mapped to try and stop this?
-
I'm also up and running, with that same image…. BUT ....
Every time i reboot my box, the interfaces swap! hv0 and hv1 literally swap.
Any clues on how they are mapped to try and stop this?
I think they were talking about it in that thread a bit. Please do an ifconfig of the swap
-
As soon as heavy traffic goes on the box - it crashes:
Sleeping thread (tid 100036, pid 12) owns a non-sleepable lock
sched_switch() at sched_switch+0x102
mi_switch() at mi_switch+0x176
sleepq_wait() at sleepq_wait+0x42
_sx_xlock_hard() at _sx_xlock_hard+0x305
hn_start() at hn_start+0x57d
if_transmit() at if_transmit+0xea
ether_output_frame() at ether_output_frame+0x33
ether_output() at ether_output+0x50d
ip_output() at ip_output+0xda9
tcp_output() at tcp_output+0xa7d
tcp_do_segment() at tcp_do_segment+0xbbc
tcp_input() at tcp_input+0xcb8
ip_input() at ip_input+0x162
netisr_dispatch_src() at netisr_dispatch_src+0x7b
ether_demux() at ether_demux+0x169
ether_input() at ether_input+0x191
netvsc_recv() at netvsc_recv+0x202
hv_rf_on_receive() at hv_rf_on_receive+0x147
hv_nv_on_channel_callback() at hv_nv_on_channel_callback+0x38e
hv_vmbus_on_events() at hv_vmbus_on_events+0xa8
intr_event_execute_handlers() at intr_event_execute_handlers+0x104
ithread_loop() at ithread_loop+0x95
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
–- trap 0, rip = 0, rsp = 0xffffff8000125d00, rbp = 0 ---
panic: sleeping thread
cpuid = 1
KDB: enter: panic
panic.txt0600001712274404005 7134 ustarrootwheelsleeping threadversion.txt06000022412274404005 7607 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #0: Sun Sep 15 12:27:10 PDT 2013
root@fbsd83minx64.corp.itbxb.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 -
Bounty increased to $1000 for a fix to the instability.
-
I would install a FreeBSD 10 VM and load up the same network configuration and see if the problem still exists. You could also try making sure pfsense only has 1 CPU available.
-
Posted up a SkyDrive link to a Hyper-V VM export of the 2.1 Release here:
Also sent PM. I've deployed this in many areas, works really well for hosted solutions and have pushed well over 100mb through this without issues for over a year now. Also had much feedback on success stories with this.
However…..
I am still having issues with CARP, and VLANs directly within pfSense. VLANs can be set on the settings of the VM in Hyper-V manager virtual NIC settings. CARP I believe I have a workaround for which is in testing right now.
If this works and you want to reward the bounty I want half to go to a charity of my choice. Other half will go towards getting CARP working as well as any other bugs that may be around.
-
The previous version of the Hyper-V drivers (older than those included with FreeBSD 10.0 Release) contain a bug in which the management of a critical thread lock is not handled properly. It does not seem to cause an issue for systems allocated two NICs at low transfer speeds (below 100Mbits). I have been able to consistently cause a kernel panic in pfsense 2.1 with the original Hyper-V drivers when the VM has three or more NICs and is moving data at rates above 100Mbits.
I would propose a back port of the drivers included with FreeBSD 10.0 Release as a resolution to your problem.
-
The previous version of the Hyper-V drivers (older than those included with FreeBSD 10.0 Release) contain a bug in which the management of a critical thread lock is not handled properly. It does not seem to cause an issue for systems allocated two NICs at low transfer speeds (below 100Mbits). I have been able to consistently cause a kernel panic in pfsense 2.1 with the original Hyper-V drivers when the VM has three or more NICs and is moving data at rates above 100Mbits.
I would propose a back port of the drivers included with FreeBSD 10.0 Release as a resolution to your problem.
I had a look at the code and the changes and it looks like it goes rather deep. Would need to grab all the different source versions and put it through a differencing then apply all the base file changes across to support the newer driver source. Best bet would be the newer release of pfsense? Anyone try one of the alpha builds based on FreeBSD 10?
-
I have tried to compile the development branch against FreeBSD 10.0 Release. I did not get very far with the various build errors. It would appear the development team is still working on getting the environment copasetic with the new OS.
-
I have tried to compile the development branch against FreeBSD 10.0 Release. I did not get very far with the various build errors. It would appear the development team is still working on getting the environment copasetic with the new OS.
-
I just did it.
I was able to re-compile the hyperv-ic 1.0 drivers (the updated drivers as of 2 months ago, included in FreeBSD 10 and available for FreeBSD 8.3) so they would be compatible with pfSense. I loaded them in a running 2.1 VM, and was able to assign interfaces and remove the legacy adapters.
To stress the network driver, I copied files over Gbe using scp at full speed while running a 10 Mbps internet speed test (that's all my ISP offers in this area). The VM hasn't failed - I hadn't experienced the crashes other users have seen with the August 2012 "beta" drivers, so I can't fully confirm it fixes those issues, but it seems encouraging that it handles this level of stress on both LAN and WAN interfaces w/o any problems.
To test it further, I configured the VM as my primary personal router (restored my existing pfSense 2.1 XML config) and everything seems to be working (internet access, IPSec and OpenVPN VPNs, etc).
I'd like to do some more testing (check the disk driver, test/compile for 2.1.1, maybe try and make an ISO so it is easier for novice users) and go over my notes and document the process - all in case I should do some more tweaks before distributing these binaries. In the next few days, I'll report back, post the binaries and instructions, and start a thread in the virtualization forum, to get more people involved in testing.
-
I started a new thread in the Virtualization board and uploaded the new drivers in https://forum.pfsense.org/index.php/topic,73258.0.html
If you had issues with the old drivers, please give the new drivers a try and let us know how they work in your environment.
-
After months of testing and overwhelming positive feedback, it's clear the drivers I compiled work more than ok for most environments.
Early on, I contacted carlmeek to see if he needed help testing the new drivers, and to see if they met the bounty requirements. Having just gone through considerable upheaval in their environment (which the other versions helped somewhat), he was reticent to introduce these new drivers before they got more mileage from other users and there was more feedback: he had a solution that seemed stable enough. Since then, I've sent other PMs after it became even more clear that the new drivers work much better than all prior solutions, and that Heartbleed and other security patches were integrated into pfSense (requiring updating all pfSense installations), but I've gotten no replies.
I devoted considerable time and resources chasing this bounty (the "fix" seems simple once you figure it out and look backwards, but it is very time and resource intensive to try different things to figure it out). And I've kept trying to integrate the drivers into the 2.1.x ISO build process, with moderate success, so they can be distributed as part of an official release. However, w/o the hope of ever winning the bounty, I can't afford to devote enough time to integrating the drivers into the 2.1.3+ build process in a timely manner (specially given that at this rate, pfSense 2.2 with native hyperv support might be released by then, rendering all my efforts moot).
I haven't had a chance to test it first hand, but it seems that 2.2 is coming along nicely, and that we might see a release sooner than later. Then again, it might take months, and we don't know yet what features and configuration changes might need tweaking between 2.1.x and 2.2.
So, in the mean time, I'll try and keep working on the ISO integration for 2.1.x, if only to give the community options, but if carlmeek comes back, or if there are other commercial users willing to replenish the bounty, it would enable me to devote more time and resources to the 2.1.x driver integration ISO build.
-
After months of testing and overwhelming positive feedback, it's clear the drivers I compiled work more than ok for most environments.
Early on, I contacted carlmeek to see if he needed help testing the new drivers, and to see if they met the bounty requirements. Having just gone through considerable upheaval in their environment (which the other versions helped somewhat), he was reticent to introduce these new drivers before they got more mileage from other users and there was more feedback: he had a solution that seemed stable enough. Since then, I've sent other PMs after it became even more clear that the new drivers work much better than all prior solutions, and that Heartbleed and other security patches were integrated into pfSense (requiring updating all pfSense installations), but I've gotten no replies.
I devoted considerable time and resources chasing this bounty (the "fix" seems simple once you figure it out and look backwards, but it is very time and resource intensive to try different things to figure it out). And I've kept trying to integrate the drivers into the 2.1.x ISO build process, with moderate success, so they can be distributed as part of an official release. However, w/o the hope of ever winning the bounty, I can't afford to devote enough time to integrating the drivers into the 2.1.3+ build process in a timely manner (specially given that at this rate, pfSense 2.2 with native hyperv support might be released by then, rendering all my efforts moot).
I haven't had a chance to test it first hand, but it seems that 2.2 is coming along nicely, and that we might see a release sooner than later. Then again, it might take months, and we don't know yet what features and configuration changes might need tweaking between 2.1.x and 2.2.
So, in the mean time, I'll try and keep working on the ISO integration for 2.1.x, if only to give the community options, but if carlmeek comes back, or if there are other commercial users willing to replenish the bounty, it would enable me to devote more time and resources to the 2.1.x driver integration ISO build.
Sounds like a lot of work. I would wait for FREEBSD 10 based release really. Have you been poking around the 2.2 alpha?