Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Fix a Hyper-V Synthetic Driver Issue US$500

    Scheduled Pinned Locked Moved Bounties
    28 Posts 5 Posters 11.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      carlmeek
      last edited by

      I just tried another test.

      I booted up with 4x network cards as before.
      I then manually loaded the kernel modules.
      i then used ifconfig to assign an IP address…. and it crashed. same crash.

      kldload hv_vmbus.ko
      kldload hv_netvsc.ko
      ifconfig hn0 1.2.3.4 255.255.255.0
      **crash **

      it only appears for a sec so i can't capture it...but i got it from the logs:

      Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
      vmbus0: [FILTER]
      padlock0: No ACE support.
      Netvsc probe… DONE
      hn0: <synthetic network="" interface="">on vmbus0
      Netvsc initializing... done!
      Netvsc probe... DONE
      hn1: <synthetic network="" interface="">on vmbus0
      Netvsc initializing... Already initialized!
      Netvsc initializing... Already initialized!

      Fatal trap 12: page fault while in kernel mode
      cpuid = 0; apic id = 00
      fault virtual address = 0x800
      fault code = supervisor read data, page not present
      instruction pointer = 0x20:0xffffffff8194d472
      stack pointer         = 0x28:0xffffff8027955970
      frame pointer         = 0x28:0xffffff80279559f0
      code segment = base 0x0, limit 0xfffff, type 0x1b
      = DPL 0, pres 1, long 1, def32 0, gran 1
      processor eflags = interrupt enabled, resume, IOPL = 0
      current process = 12 (swi4: hv_event)
      version.txt06000024112272726133  7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
          root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic>

      1 Reply Last reply Reply Quote 0
      • B
        bryan.paradis
        last edited by

        @carlmeek:

        I just tried another test.

        I booted up with 4x network cards as before.
        I then manually loaded the kernel modules.
        i then used ifconfig to assign an IP address…. and it crashed. same crash.

        kldload hv_vmbus.ko
        kldload hv_netvsc.ko
        ifconfig hn0 1.2.3.4 255.255.255.0
        **crash **

        it only appears for a sec so i can't capture it...but i got it from the logs:

        Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
        vmbus0: [FILTER]
        padlock0: No ACE support.
        Netvsc probe… DONE
        hn0: <synthetic network="" interface="">on vmbus0
        Netvsc initializing... done!
        Netvsc probe... DONE
        hn1: <synthetic network="" interface="">on vmbus0
        Netvsc initializing... Already initialized!
        Netvsc initializing... Already initialized!

        Fatal trap 12: page fault while in kernel mode
        cpuid = 0; apic id = 00
        fault virtual address = 0x800
        fault code = supervisor read data, page not present
        instruction pointer = 0x20:0xffffffff8194d472
        stack pointer         = 0x28:0xffffff8027955970
        frame pointer         = 0x28:0xffffff80279559f0
        code segment = base 0x0, limit 0xfffff, type 0x1b
        = DPL 0, pres 1, long 1, def32 0, gran 1
        processor eflags = interrupt enabled, resume, IOPL = 0
        current process = 12 (swi4: hv_event)
        version.txt06000024112272726133  7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
            root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic>

        Is there anymore log information avaliable? Can you run "bt" or "backtrace" after it crashes.

        1 Reply Last reply Reply Quote 0
        • C
          carlmeek
          last edited by

          BT and Backtrace are command not found.  But here is the entire log from the web front end:

          Crash report is over max size for the forum, so it's here.
          https://drive.google.com/file/d/0BwOWfIoqpmmfZ25LQmlTemVpVU0/edit?usp=sharing

          1 Reply Last reply Reply Quote 0
          • C
            carlmeek
            last edited by

            Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.

            I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver.  Something connected to this?  I'm just speculating.

            -C

            1 Reply Last reply Reply Quote 0
            • B
              bryan.paradis
              last edited by

              @carlmeek:

              Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.

              I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver.  Something connected to this?  I'm just speculating.

              -C

              I don't think so as the whole package does a bunch of stuff. This is just the vmbus and the netsvc driver. I can try with the older driver source next. This was from current source. I would have to do a lot of file comparing to see where abouts the issue is with the vmbus driver. Does it survive a ifconfig interface up? Or is it just crashing on assign

              1 Reply Last reply Reply Quote 0
              • C
                carlmeek
                last edited by

                I just tried 'ifconfig hn0 up'…. instant death.

                1 Reply Last reply Reply Quote 0
                • B
                  bryan.paradis
                  last edited by

                  @carlmeek:

                  I just tried 'ifconfig hn0 up'…. instant death.

                  You may just want to wait until the next pfsense release as the hyperv drivers may potentially be built in.

                  Well here is compiled from 8.2

                  hv_vmbus.ko.amd64.txt
                  hv_netvsc.ko.amd64.txt

                  1 Reply Last reply Reply Quote 0
                  • B
                    bryan.paradis
                    last edited by

                    https://forum.pfsense.org/index.php/topic,56565.msg393651.html#new

                    Bill seems to be up and running

                    1 Reply Last reply Reply Quote 0
                    • C
                      carlmeek
                      last edited by

                      I'm also up and running, with that same image…. BUT ....

                      Every time i reboot my box, the interfaces swap!  hv0 and hv1 literally swap.

                      Any clues on how they are mapped to try and stop this?

                      1 Reply Last reply Reply Quote 0
                      • B
                        bryan.paradis
                        last edited by

                        @carlmeek:

                        I'm also up and running, with that same image…. BUT ....

                        Every time i reboot my box, the interfaces swap!  hv0 and hv1 literally swap.

                        Any clues on how they are mapped to try and stop this?

                        I think they were talking about it in that thread a bit. Please do an ifconfig of the swap

                        1 Reply Last reply Reply Quote 0
                        • C
                          carlmeek
                          last edited by

                          As soon as heavy traffic goes on the box - it crashes:

                          Sleeping thread (tid 100036, pid 12) owns a non-sleepable lock
                          sched_switch() at sched_switch+0x102
                          mi_switch() at mi_switch+0x176
                          sleepq_wait() at sleepq_wait+0x42
                          _sx_xlock_hard() at _sx_xlock_hard+0x305
                          hn_start() at hn_start+0x57d
                          if_transmit() at if_transmit+0xea
                          ether_output_frame() at ether_output_frame+0x33
                          ether_output() at ether_output+0x50d
                          ip_output() at ip_output+0xda9
                          tcp_output() at tcp_output+0xa7d
                          tcp_do_segment() at tcp_do_segment+0xbbc
                          tcp_input() at tcp_input+0xcb8
                          ip_input() at ip_input+0x162
                          netisr_dispatch_src() at netisr_dispatch_src+0x7b
                          ether_demux() at ether_demux+0x169
                          ether_input() at ether_input+0x191
                          netvsc_recv() at netvsc_recv+0x202
                          hv_rf_on_receive() at hv_rf_on_receive+0x147
                          hv_nv_on_channel_callback() at hv_nv_on_channel_callback+0x38e
                          hv_vmbus_on_events() at hv_vmbus_on_events+0xa8
                          intr_event_execute_handlers() at intr_event_execute_handlers+0x104
                          ithread_loop() at ithread_loop+0x95
                          fork_exit() at fork_exit+0x11f
                          fork_trampoline() at fork_trampoline+0xe
                          –- trap 0, rip = 0, rsp = 0xffffff8000125d00, rbp = 0 ---
                          panic: sleeping thread
                          cpuid = 1
                          KDB: enter: panic
                          panic.txt0600001712274404005  7134 ustarrootwheelsleeping threadversion.txt06000022412274404005  7607 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #0: Sun Sep 15 12:27:10 PDT 2013
                              root@fbsd83minx64.corp.itbxb.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8

                          1 Reply Last reply Reply Quote 0
                          • C
                            carlmeek
                            last edited by

                            Bounty increased to $1000 for a fix to the instability.

                            1 Reply Last reply Reply Quote 0
                            • B
                              bryan.paradis
                              last edited by

                              I would install a FreeBSD 10 VM and load up the same network configuration and see if the problem still exists. You could also try making sure pfsense only has 1 CPU available.

                              1 Reply Last reply Reply Quote 0
                              • A
                                alexappleton
                                last edited by

                                Posted up a SkyDrive link to a Hyper-V VM export of the 2.1 Release here:

                                http://sdrv.ms/1565IAQ-

                                Also sent PM.  I've deployed this in many areas, works really well for hosted solutions and have pushed well over 100mb through this without issues for over a year now.  Also had much feedback on success stories with this.

                                However…..

                                I am still having issues with CARP, and VLANs directly within pfSense.  VLANs can be set on the settings of the VM in Hyper-V manager virtual NIC settings.  CARP I believe I have a workaround for which is in testing right now.

                                If this works and you want to reward the bounty I want half to go to a charity of my choice.  Other half will go towards getting CARP working as well as any other bugs that may be around.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  pingpong
                                  last edited by

                                  The previous version of the Hyper-V drivers (older than those included with FreeBSD 10.0 Release) contain a bug in which the management of a critical thread lock is not handled properly.  It does not seem to cause an issue for systems allocated two NICs at low transfer speeds (below 100Mbits).  I have been able to consistently cause a kernel panic in pfsense 2.1 with the original Hyper-V drivers when the VM has three or more NICs  and is moving data at rates above 100Mbits.

                                  I would propose a back port of the drivers included with FreeBSD 10.0 Release as a resolution to your problem.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bryan.paradis
                                    last edited by

                                    @pingpong:

                                    The previous version of the Hyper-V drivers (older than those included with FreeBSD 10.0 Release) contain a bug in which the management of a critical thread lock is not handled properly.  It does not seem to cause an issue for systems allocated two NICs at low transfer speeds (below 100Mbits).  I have been able to consistently cause a kernel panic in pfsense 2.1 with the original Hyper-V drivers when the VM has three or more NICs  and is moving data at rates above 100Mbits.

                                    I would propose a back port of the drivers included with FreeBSD 10.0 Release as a resolution to your problem.

                                    I had a look at the code and the changes and it looks like it goes rather deep. Would need to grab all the different source versions and put it through a differencing then apply all the base file changes across to support the newer driver source. Best bet would be the newer release of pfsense? Anyone try one of the alpha builds based on FreeBSD 10?

                                    1 Reply Last reply Reply Quote 0
                                    • P
                                      pingpong
                                      last edited by

                                      I have tried to compile the development branch against FreeBSD 10.0 Release.  I did not get very far with the various build errors.  It would appear the development team is still working on getting the environment copasetic with the new OS.

                                      1 Reply Last reply Reply Quote 0
                                      • B
                                        bryan.paradis
                                        last edited by

                                        @pingpong:

                                        I have tried to compile the development branch against FreeBSD 10.0 Release.  I did not get very far with the various build errors.  It would appear the development team is still working on getting the environment copasetic with the new OS.

                                        https://forum.pfsense.org/index.php/topic,72887.0.html

                                        1 Reply Last reply Reply Quote 0
                                        • Z
                                          zootie
                                          last edited by

                                          I just did it.

                                          I was able to re-compile the hyperv-ic 1.0 drivers (the updated drivers as of 2 months ago, included in FreeBSD 10 and available for FreeBSD 8.3) so they would be compatible with pfSense. I loaded them in a running 2.1 VM, and was able to assign interfaces and remove the legacy adapters.

                                          To stress the network driver, I copied files over Gbe using scp at full speed while running a 10 Mbps internet speed test (that's all my ISP offers in this area). The VM hasn't  failed - I hadn't experienced the crashes other users have seen with the August 2012 "beta" drivers, so I can't fully confirm it fixes those issues, but it seems encouraging that it handles this level of stress on both LAN and WAN interfaces w/o any problems.

                                          To test it further, I configured the VM as my primary personal router (restored my existing pfSense 2.1 XML config) and everything seems to be working (internet access, IPSec and OpenVPN VPNs, etc).

                                          I'd like to do some more testing (check the disk driver, test/compile for 2.1.1, maybe try and make an ISO so it is easier for novice users) and go over my notes and document the process - all in case I should do some more tweaks before distributing these binaries. In the next few days, I'll report back, post the binaries and instructions, and start a thread in the virtualization forum, to get more people involved in testing.

                                          1 Reply Last reply Reply Quote 0
                                          • Z
                                            zootie
                                            last edited by

                                            I started a new thread in the Virtualization board and uploaded the new drivers in https://forum.pfsense.org/index.php/topic,73258.0.html

                                            If you had issues with the old drivers, please give the new drivers a try and let us know how they work in your environment.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.