Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1

lburton

FWIW throughput looks fine.. most of the time (>30MB/s to google hosts and others) I do have some odd exceptions which I will look into (valve used to provide at least 40MB/s for static content but now I see only 2MB/s).

Inside the lan looks good - iperf sees about 730Mb/s bi-directional with very little config - from a Linux box in hyper-v on a different machine:


lburton@downquark ~ # iperf -c lburton-router -i 10 -t 60 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to lburton-router, TCP port 5001
TCP window size:  183 KByte (default)
------------------------------------------------------------
[  3] local 44.44.92.2 port 48434 connected with 44.44.92.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   824 MBytes   691 Mbits/sec
[  3] 10.0-20.0 sec   905 MBytes   759 Mbits/sec
[  3] 20.0-30.0 sec   882 MBytes   740 Mbits/sec
[  3] 30.0-40.0 sec   878 MBytes   737 Mbits/sec
[  3] 40.0-50.0 sec   875 MBytes   734 Mbits/sec
[  3] 50.0-60.0 sec   866 MBytes   727 Mbits/sec
[  3]  0.0-60.0 sec  5.11 GBytes   731 Mbits/sec

Also while I kinda hate it.. I figure I might as well: ;) :P

One thing I see in the hyper-v drivers that I don't see in any of the other drivers (looking at the intel NIC drivers for example) is that during init they manually set if_flags (not that it probably matters.. I'm guessing the flags are changed by the OS code when the interface is configured but I haven't looked)


ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;

https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c#L265

Edit:
external iperf is having similar issues to valve/others – I see good outbound throughput and poor inbound..


------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to iperf.scottlinux.com, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 44.44.92.5 port 42344 connected with 173.230.156.66 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-30.0 sec  2.76 GBytes   790 Mbits/sec
[  5] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[  4] local 44.44.92.5 port 5001 connected with 173.230.156.66 port 35736
[  4]  0.0-30.1 sec   176 MBytes  49.1 Mbits/sec
[  4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

cpresco

Can you elaborate on "That path leads to madness."?

I've been running a standard/basic pfsense 2.1.2 -> 2.1.3 -> 2.1.5 (2.1.3 and 2.1.5 were upgrades from 2.1.2) without any significant issues on Hyper-V Server 2012 R2.

I've considered trying the 2.2 betas but 2.1.x with these drivers has been stable enough for me that I figured that I'd wait for a 2.2 stable release

@gonzopancho:

I don't recommend even trying 2.1 on Hyper-V. That path leads to madness.

2.2 is in pretty good shape. There are 10 (count them, 10) outstanding bugs in the way of cutting a release candidate.

one of them concerns CARP.

try it out! (It's a VM environment, what do you have to lose?)

Jim

cmb

@cpresco:

Can you elaborate on "That path leads to madness."?

The most stable option available for Hyper-V is 2.2-RC (or newer if available and you're reading this in the future). By far. You're effectively hacking together something on a base OS that isn't officially supported by Microsoft. I'm sure it works fine in some specific circumstances. It is, without question, a riskier proposition than running 2.2 and has been for some time.

Guest

Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.

I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.

Email me if you want to test this.

If it works well, we'll get it into pfSense 2.2

Jim

Guest

@lburton:

@gonzopancho:

In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.

I can definitely confirm it's a Hyper-V NIC driver issue for carp but I don't believe it to multicast related in carp's case – it looks to be related to the NIC state information as a quick super-hacky-terrible patch I did to the carp kernel code has resulted in functional carp on a test setup for me in Hyper-V. (I haven't tried to get pfsync going in the same setup -- it doesn't seem to be syncing state properly according to pftop.) I added it to the existing ip_carp.c.diff in the patch list:
root@freebsd:~ # cat /home/pfsense/tools/patches/releng/10.1/ip_carp.c.diff
diff --git a/sys/netinet/ip_carp.c b/sys/netinet/ip_carp.c
index a170e34..0a3607e 100644
--- a/sys/netinet/ip_carp.c
+++ b/sys/netinet/ip_carp.c
@@ -532,8 +532,8 @@ carp6_input(struct mbuf **mp, int *offp, int proto)
        /* check if received on a valid carp interface */
        if (m->m_pkthdr.rcvif->if_carp == NULL) {
                CARPSTATS_INC(carps_badif);
-               CARP_DEBUG("%s: packet received on non-carp interface: %s\n",
-                   __func__, m->m_pkthdr.rcvif->if_xname);
+               //CARP_DEBUG("%s: packet received on non-carp interface: %s\n",
+               //    __func__, m->m_pkthdr.rcvif->if_xname);
                m_freem(m);
                return (IPPROTO_DONE);
        }
@@ -1195,8 +1195,7 @@

        CARP_LOCK_ASSERT(sc);

-       if ((sc->sc_carpdev->if_flags & IFF_UP) == 0 ||
-           sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
+       if (
            (sc->sc_naddrs == 0 && sc->sc_naddrs6 == 0))
                return;

@@ -2001,27 +2000,11 @@

        CARP_LOCK_ASSERT(sc);

-       if (sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
-           !(sc->sc_carpdev->if_flags & IFF_UP)) {
-               callout_stop(&sc->sc_ad_tmo);
-#ifdef INET
-               callout_stop(&sc->sc_md_tmo);
-#endif
-#ifdef INET6
-               callout_stop(&sc->sc_md6_tmo);
-#endif
-               carp_set_state(sc, INIT);
-               carp_setrun(sc, 0);
-               if (!sc->sc_suppress)
-                       carp_demote_adj(V_carp_ifdown_adj, "interface down");
-               sc->sc_suppress = 1;
-       } else {
                carp_set_state(sc, INIT);
                carp_setrun(sc, 0);
                if (sc->sc_suppress)
                        carp_demote_adj(-V_carp_ifdown_adj, "interface up");
                sc->sc_suppress = 0;
-       }
 }

 static void
I'm not experienced enough with FreeBSD kernel debugging / drivers to really take this all that much further without it being fairly efforty – but it looks like:
-       if (sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
-           !(sc->sc_carpdev->if_flags & IFF_UP)) {
is not behaving correctly under hyper-v's network drivers. As for the cause for pfsync's woes I may try to take a look later.

The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.

lburton

@gonzopancho:

The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.

Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.

Guest

@lburton:

@gonzopancho:

The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.

Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.

patch is now in the next snapshot. (So much for all the key4ce bullshit.)

cmb

I can confirm that CARP is fully functional in Hyper-V with the most recent 2.2 snapshot available! Thanks to the folks at Microsoft for fixing it and getting us the patch. Those who'd like to use CARP in Hyper-V, check out the most recent 2.2 from snapshots.pfsense.org and let us know your experiences.

cmb

@cmb:

@key4ce:

That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.

It's pretty clear from your code changes you have no idea what you're doing. Outside of the Hyper-V NIC driver bug, I'm going to call you out on this one - open just ONE actual bug with CARP at redmine.pfsense.org. Just one would suffice. You're spreading FUD and won't be able to do so.

I'll re-issue my call out now that CARP in 2.2 is working in Hyper-V as well, just in case anyone thinks they have any degree of credibility. We haven't gotten a single bug report from them to prove their point (because they're lying). Ours works with IPv6 too! Which key4ce's hacked up mess does not.

nomaded

@gonzopancho:

Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.

I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.

Email me if you want to test this.

If it works well, we'll get it into pfSense 2.2

Jim

I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?

nomaded

@nomaded:

I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?

Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.

Guest

@nomaded:

@nomaded:

I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?

Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.

You can look in the pfsense-tools repo for the same patch.

duntuk

Is this patch necessary for the pfsense 2.2 release ? If so, how do you apply it?

cmb

@duntuk:

Is this patch necessary for the pfsense 2.2 release ?

No, it's included.