Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Fix a Hyper-V Synthetic Driver Issue US$500

    Scheduled Pinned Locked Moved Bounties
    28 Posts 5 Posters 11.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      carlmeek
      last edited by

      I'm trying to get Hyper-V synthetic drivers working to get a better throughput on my 25 pfSense instances that are all running under Hyper-V.

      I have followed instructions here:
      http://alexappleton.net/post/37410981279/pfsense-in-hyper-v-with-integration-components

      But it doesn't work - probably because i'm running 2.1-RELEASE.  It all looks like it's worked, and the machine boots and gets to the menu, but it won't pass traffic.

      Problems:-
      1. This pops up a couple of times during boot: unknown oid ‘net.inet.ipcomp.ipcomp_enable’
      2. Notices appear saying: [There were error(s) loading the rules: pfctl: /dev/pf No such file or directory - The line in question reads [0]: ]

      For the bounty i need to have a 2.1-RELEASE version of pfSense running the HyperV drivers and able to push a throughput that exceeds 100mb (the current limit of the Legacy network adapter).

      1 Reply Last reply Reply Quote 0
      • B
        bryan.paradis
        last edited by

        @carlmeek:

        I'm trying to get Hyper-V synthetic drivers working to get a better throughput on my 25 pfSense instances that are all running under Hyper-V.

        I have followed instructions here:
        http://alexappleton.net/post/37410981279/pfsense-in-hyper-v-with-integration-components

        But it doesn't work - probably because i'm running 2.1-RELEASE.  It all looks like it's worked, and the machine boots and gets to the menu, but it won't pass traffic.

        Problems:-
        1. This pops up a couple of times during boot: unknown oid ‘net.inet.ipcomp.ipcomp_enable’
        2. Notices appear saying: [There were error(s) loading the rules: pfctl: /dev/pf No such file or directory - The line in question reads [0]: ]

        For the bounty i need to have a 2.1-RELEASE version of pfSense running the HyperV drivers and able to push a throughput that exceeds 100mb (the current limit of the Legacy network adapter).

        1. What release source did you compile with 8.2 like the guide? or 8.3? pFsense 2.1 runs on 8.3
        2. Were there any compiler warnings
        3. Did your original post get deleted about this subject?
        4. Have you tried the driver I compiled here after seeing your original posthttps://forum.pfsense.org/index.php/topic,72081.0.html

        1 Reply Last reply Reply Quote 0
        • C
          carlmeek
          last edited by

          I apologise for removing the first post - i thought i had solved the problem myself… but no.

          1. I compiled 8.3 - matching the Kernel version of 2-1-Release.
          2. I did not notice any compiler warnings, but i'm not 100% there weren't any now.
          3. Yes, I removed it - I thought I had done so before anyone saw it, Views was still '1' - sorry about that.
          4. I haven't tried your driver (hadn't seen that post) - but i will attempt to do so now.

          I'm not hugely experienced in the workings of BSD, apologies for that.

          1 Reply Last reply Reply Quote 0
          • C
            carlmeek
            last edited by

            I've given it a try.  No luck just yet.

            This is what I did:

            1. Reverted out previous kernel changes to take me back to a standard 2-1-Release.
            2. Downloaded your file into /boot/modules
            3. chmod +x
            4. Added to loader.conf.local
            5. Rebooted
            –Nothing at this point, no new network cards--
            6. Tried kldload hv_netvsc.ko and got this:

            KLD hv_netvsc.ko: depends on vmbus - not available or version mismatch
            linker_load_file: Unsupported file type
            kldload: can't load hv_netvsc.ko: Exec format error

            Regards,
            -Carl.

            1 Reply Last reply Reply Quote 0
            • B
              bryan.paradis
              last edited by

              @carlmeek:

              I've given it a try.  No luck just yet.

              This is what I did:

              1. Reverted out previous kernel changes to take me back to a standard 2-1-Release.
              2. Downloaded your file into /boot/modules
              3. chmod +x
              4. Added to loader.conf.local
              5. Rebooted
              –Nothing at this point, no new network cards--
              6. Tried kldload hv_netvsc.ko and got this:

              KLD hv_netvsc.ko: depends on vmbus - not available or version mismatch
              linker_load_file: Unsupported file type
              kldload: can't load hv_netvsc.ko: Exec format error

              Regards,
              -Carl.

              move this one to /boot/modules and load it first kldload hv_vmbus.ko then kldload hv_netvsc.ko

              hv_vmbus.ko.amd64.txt

              1 Reply Last reply Reply Quote 0
              • C
                carlmeek
                last edited by

                Getting Close!  After the two KLDLoad statements, i can see HyperV network interfaces in ifconfig.  Not had a chance to test yet, i'll do that tonight.

                However - it's not working on boot.  I added these into loader.conf.local

                hv_vmbus_load="YES"
                hv_netvsc_load="YES"

                First couple of boots is fine - we have 4 network cards, 2 legacy (mapped to lan/wan) and 2 HV cards.
                Then i remove the legacy cards and reboot….And now it crashes on boot and reboots itself.
                It reboots just shortly after the question about wanting to remap the network cards pops up.  The crash is very quick, can't catch it's text.

                So, i tried booting with 4 cards connected, and then manually reassigning the adapters using option (1).  I did this, and as soon as i confirmed my changes are OK, same crash.  And now it crashes every boot.

                One interesting thing to note is that during the assigning of the adapters, the two HyperV adapters reported that they  are 'down', even though they are not.

                Finally i remove the hyperV adapters from the machine, and it boots happily and i can reassign back to the 'de' legacy adapters.

                1 Reply Last reply Reply Quote 0
                • C
                  carlmeek
                  last edited by

                  I just tried another test.

                  I booted up with 4x network cards as before.
                  I then manually loaded the kernel modules.
                  i then used ifconfig to assign an IP address…. and it crashed. same crash.

                  kldload hv_vmbus.ko
                  kldload hv_netvsc.ko
                  ifconfig hn0 1.2.3.4 255.255.255.0
                  **crash **

                  it only appears for a sec so i can't capture it...but i got it from the logs:

                  Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
                  vmbus0: [FILTER]
                  padlock0: No ACE support.
                  Netvsc probe… DONE
                  hn0: <synthetic network="" interface="">on vmbus0
                  Netvsc initializing... done!
                  Netvsc probe... DONE
                  hn1: <synthetic network="" interface="">on vmbus0
                  Netvsc initializing... Already initialized!
                  Netvsc initializing... Already initialized!

                  Fatal trap 12: page fault while in kernel mode
                  cpuid = 0; apic id = 00
                  fault virtual address = 0x800
                  fault code = supervisor read data, page not present
                  instruction pointer = 0x20:0xffffffff8194d472
                  stack pointer         = 0x28:0xffffff8027955970
                  frame pointer         = 0x28:0xffffff80279559f0
                  code segment = base 0x0, limit 0xfffff, type 0x1b
                  = DPL 0, pres 1, long 1, def32 0, gran 1
                  processor eflags = interrupt enabled, resume, IOPL = 0
                  current process = 12 (swi4: hv_event)
                  version.txt06000024112272726133  7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
                      root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic>

                  1 Reply Last reply Reply Quote 0
                  • B
                    bryan.paradis
                    last edited by

                    @carlmeek:

                    I just tried another test.

                    I booted up with 4x network cards as before.
                    I then manually loaded the kernel modules.
                    i then used ifconfig to assign an IP address…. and it crashed. same crash.

                    kldload hv_vmbus.ko
                    kldload hv_netvsc.ko
                    ifconfig hn0 1.2.3.4 255.255.255.0
                    **crash **

                    it only appears for a sec so i can't capture it...but i got it from the logs:

                    Timecounter "Hyper-V" frequency 10000000 Hz quality 10000000
                    vmbus0: [FILTER]
                    padlock0: No ACE support.
                    Netvsc probe… DONE
                    hn0: <synthetic network="" interface="">on vmbus0
                    Netvsc initializing... done!
                    Netvsc probe... DONE
                    hn1: <synthetic network="" interface="">on vmbus0
                    Netvsc initializing... Already initialized!
                    Netvsc initializing... Already initialized!

                    Fatal trap 12: page fault while in kernel mode
                    cpuid = 0; apic id = 00
                    fault virtual address = 0x800
                    fault code = supervisor read data, page not present
                    instruction pointer = 0x20:0xffffffff8194d472
                    stack pointer         = 0x28:0xffffff8027955970
                    frame pointer         = 0x28:0xffffff80279559f0
                    code segment = base 0x0, limit 0xfffff, type 0x1b
                    = DPL 0, pres 1, long 1, def32 0, gran 1
                    processor eflags = interrupt enabled, resume, IOPL = 0
                    current process = 12 (swi4: hv_event)
                    version.txt06000024112272726133  7615 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #1: Wed Sep 11 18:59:48 EDT 2013
                        root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8</synthetic></synthetic>

                    Is there anymore log information avaliable? Can you run "bt" or "backtrace" after it crashes.

                    1 Reply Last reply Reply Quote 0
                    • C
                      carlmeek
                      last edited by

                      BT and Backtrace are command not found.  But here is the entire log from the web front end:

                      Crash report is over max size for the forum, so it's here.
                      https://drive.google.com/file/d/0BwOWfIoqpmmfZ25LQmlTemVpVU0/edit?usp=sharing

                      1 Reply Last reply Reply Quote 0
                      • C
                        carlmeek
                        last edited by

                        Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.

                        I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver.  Something connected to this?  I'm just speculating.

                        -C

                        1 Reply Last reply Reply Quote 0
                        • B
                          bryan.paradis
                          last edited by

                          @carlmeek:

                          Just tested again on a second, new build, virtual machine - same result. Kernel crash whenever you assign an IP to bring an interface up.

                          I wonder - from the previous instructions where the kernel had to be recompiled - the fact they had to disable the fast IDE driver.  Something connected to this?  I'm just speculating.

                          -C

                          I don't think so as the whole package does a bunch of stuff. This is just the vmbus and the netsvc driver. I can try with the older driver source next. This was from current source. I would have to do a lot of file comparing to see where abouts the issue is with the vmbus driver. Does it survive a ifconfig interface up? Or is it just crashing on assign

                          1 Reply Last reply Reply Quote 0
                          • C
                            carlmeek
                            last edited by

                            I just tried 'ifconfig hn0 up'…. instant death.

                            1 Reply Last reply Reply Quote 0
                            • B
                              bryan.paradis
                              last edited by

                              @carlmeek:

                              I just tried 'ifconfig hn0 up'…. instant death.

                              You may just want to wait until the next pfsense release as the hyperv drivers may potentially be built in.

                              Well here is compiled from 8.2

                              hv_vmbus.ko.amd64.txt
                              hv_netvsc.ko.amd64.txt

                              1 Reply Last reply Reply Quote 0
                              • B
                                bryan.paradis
                                last edited by

                                https://forum.pfsense.org/index.php/topic,56565.msg393651.html#new

                                Bill seems to be up and running

                                1 Reply Last reply Reply Quote 0
                                • C
                                  carlmeek
                                  last edited by

                                  I'm also up and running, with that same image…. BUT ....

                                  Every time i reboot my box, the interfaces swap!  hv0 and hv1 literally swap.

                                  Any clues on how they are mapped to try and stop this?

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bryan.paradis
                                    last edited by

                                    @carlmeek:

                                    I'm also up and running, with that same image…. BUT ....

                                    Every time i reboot my box, the interfaces swap!  hv0 and hv1 literally swap.

                                    Any clues on how they are mapped to try and stop this?

                                    I think they were talking about it in that thread a bit. Please do an ifconfig of the swap

                                    1 Reply Last reply Reply Quote 0
                                    • C
                                      carlmeek
                                      last edited by

                                      As soon as heavy traffic goes on the box - it crashes:

                                      Sleeping thread (tid 100036, pid 12) owns a non-sleepable lock
                                      sched_switch() at sched_switch+0x102
                                      mi_switch() at mi_switch+0x176
                                      sleepq_wait() at sleepq_wait+0x42
                                      _sx_xlock_hard() at _sx_xlock_hard+0x305
                                      hn_start() at hn_start+0x57d
                                      if_transmit() at if_transmit+0xea
                                      ether_output_frame() at ether_output_frame+0x33
                                      ether_output() at ether_output+0x50d
                                      ip_output() at ip_output+0xda9
                                      tcp_output() at tcp_output+0xa7d
                                      tcp_do_segment() at tcp_do_segment+0xbbc
                                      tcp_input() at tcp_input+0xcb8
                                      ip_input() at ip_input+0x162
                                      netisr_dispatch_src() at netisr_dispatch_src+0x7b
                                      ether_demux() at ether_demux+0x169
                                      ether_input() at ether_input+0x191
                                      netvsc_recv() at netvsc_recv+0x202
                                      hv_rf_on_receive() at hv_rf_on_receive+0x147
                                      hv_nv_on_channel_callback() at hv_nv_on_channel_callback+0x38e
                                      hv_vmbus_on_events() at hv_vmbus_on_events+0xa8
                                      intr_event_execute_handlers() at intr_event_execute_handlers+0x104
                                      ithread_loop() at ithread_loop+0x95
                                      fork_exit() at fork_exit+0x11f
                                      fork_trampoline() at fork_trampoline+0xe
                                      –- trap 0, rip = 0, rsp = 0xffffff8000125d00, rbp = 0 ---
                                      panic: sleeping thread
                                      cpuid = 1
                                      KDB: enter: panic
                                      panic.txt0600001712274404005  7134 ustarrootwheelsleeping threadversion.txt06000022412274404005  7607 ustarrootwheelFreeBSD 8.3-RELEASE-p11 #0: Sun Sep 15 12:27:10 PDT 2013
                                          root@fbsd83minx64.corp.itbxb.com:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8

                                      1 Reply Last reply Reply Quote 0
                                      • C
                                        carlmeek
                                        last edited by

                                        Bounty increased to $1000 for a fix to the instability.

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          bryan.paradis
                                          last edited by

                                          I would install a FreeBSD 10 VM and load up the same network configuration and see if the problem still exists. You could also try making sure pfsense only has 1 CPU available.

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            alexappleton
                                            last edited by

                                            Posted up a SkyDrive link to a Hyper-V VM export of the 2.1 Release here:

                                            http://sdrv.ms/1565IAQ-

                                            Also sent PM.  I've deployed this in many areas, works really well for hosted solutions and have pushed well over 100mb through this without issues for over a year now.  Also had much feedback on success stories with this.

                                            However…..

                                            I am still having issues with CARP, and VLANs directly within pfSense.  VLANs can be set on the settings of the VM in Hyper-V manager virtual NIC settings.  CARP I believe I have a workaround for which is in testing right now.

                                            If this works and you want to reward the bounty I want half to go to a charity of my choice.  Other half will go towards getting CARP working as well as any other bugs that may be around.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.