Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1
-
I recalled the vtnet driver of virtio originally had the same problem as here. Though looking through the change log, I don't see any commit logs that mention fixing that issue. It was fixed and works fine now, was hoping to be able to easily pinpoint the commit that fixed that but it's not obvious at a glance.
http://svnweb.freebsd.org/base/releng/10.1/sys/dev/virtio/network/if_vtnet.c?view=logI agree that multicast doesn't seem to be completely broken, some things work. Hopefully I'll get some time in the next week to get back to a bit more testing here, but it'll likely be this weekend at best as I'm focused on getting 2.2 release out at the moment.
-
@cmb:
Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.That's something we'll get into with the folks at Microsoft,
That's awesome… everything what can't run in virtualized environments anymore is a lack of support and not in "our" (my collegues for example. ;) ) focus anymore.
So I was really excited to see the 2.2 snapshots running like a charm at the moment.
Thanks to you guys. thumps up.… but generally speaking for this type of use case, you don't do full VM backups at all.
I don't understand. Why?
Have HA systems in place, and config backups ready to restore, and you're good. Generally just a waste of disk/tape/cloud/whatever-you-backup-to space to get full backups of a system like pfSense where it's really easy and fast to rebuild from scratch and restore (probably faster than restoring a full disk VM backup actually).
In my case it doesn't. It would need a much more automatic install process I can't afford or setting up at the moment.
HA is a bit to much and a recovery run doesn't take as long as I would need to reinstall and setup all the small needed things. It integrates at the moment much better in this kind of environment I'm running. So there's no need for a full Hyper-V support at the moment for me but it will be… sooner or later. :)
I'm on DPM and use pfsense at the moment more for testing purposes.waste of drive space sounds a bit nineties. fortunately dedup and reliable backup solutions kills that argument at all. ;)
Btw.
I would like to spend some money but I couldn't get a way to support pfsense except buying the book.
-
FWIW throughput looks fine.. most of the time (>30MB/s to google hosts and others) I do have some odd exceptions which I will look into (valve used to provide at least 40MB/s for static content but now I see only 2MB/s).
Inside the lan looks good - iperf sees about 730Mb/s bi-directional with very little config - from a Linux box in hyper-v on a different machine:
lburton@downquark ~ # iperf -c lburton-router -i 10 -t 60 -d ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to lburton-router, TCP port 5001 TCP window size: 183 KByte (default) ------------------------------------------------------------ [ 3] local 44.44.92.2 port 48434 connected with 44.44.92.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 824 MBytes 691 Mbits/sec [ 3] 10.0-20.0 sec 905 MBytes 759 Mbits/sec [ 3] 20.0-30.0 sec 882 MBytes 740 Mbits/sec [ 3] 30.0-40.0 sec 878 MBytes 737 Mbits/sec [ 3] 40.0-50.0 sec 875 MBytes 734 Mbits/sec [ 3] 50.0-60.0 sec 866 MBytes 727 Mbits/sec [ 3] 0.0-60.0 sec 5.11 GBytes 731 Mbits/secAlso while I kinda hate it.. I figure I might as well: ;) :P
One thing I see in the hyper-v drivers that I don't see in any of the other drivers (looking at the intel NIC drivers for example) is that during init they manually set if_flags (not that it probably matters.. I'm guessing the flags are changed by the OS code when the interface is configured but I haven't looked)
ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c#L265
Edit:
external iperf is having similar issues to valve/others – I see good outbound throughput and poor inbound..------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to iperf.scottlinux.com, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 5] local 44.44.92.5 port 42344 connected with 173.230.156.66 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-30.0 sec 2.76 GBytes 790 Mbits/sec [ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [ 4] local 44.44.92.5 port 5001 connected with 173.230.156.66 port 35736 [ 4] 0.0-30.1 sec 176 MBytes 49.1 Mbits/sec [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet) -
Can you elaborate on "That path leads to madness."?
I've been running a standard/basic pfsense 2.1.2 -> 2.1.3 -> 2.1.5 (2.1.3 and 2.1.5 were upgrades from 2.1.2) without any significant issues on Hyper-V Server 2012 R2.
I've considered trying the 2.2 betas but 2.1.x with these drivers has been stable enough for me that I figured that I'd wait for a 2.2 stable release
@gonzopancho:
I don't recommend even trying 2.1 on Hyper-V. That path leads to madness.
2.2 is in pretty good shape. There are 10 (count them, 10) outstanding bugs in the way of cutting a release candidate.
one of them concerns CARP.
try it out! (It's a VM environment, what do you have to lose?)
Jim
-
Can you elaborate on "That path leads to madness."?
The most stable option available for Hyper-V is 2.2-RC (or newer if available and you're reading this in the future). By far. You're effectively hacking together something on a base OS that isn't officially supported by Microsoft. I'm sure it works fine in some specific circumstances. It is, without question, a riskier proposition than running 2.2 and has been for some time.
-
Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.
I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.
Email me if you want to test this.
If it works well, we'll get it into pfSense 2.2
Jim
-
@gonzopancho:
In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.I can definitely confirm it's a Hyper-V NIC driver issue for carp but I don't believe it to multicast related in carp's case – it looks to be related to the NIC state information as a quick super-hacky-terrible patch I did to the carp kernel code has resulted in functional carp on a test setup for me in Hyper-V. (I haven't tried to get pfsync going in the same setup -- it doesn't seem to be syncing state properly according to pftop.) I added it to the existing ip_carp.c.diff in the patch list:
root@freebsd:~ # cat /home/pfsense/tools/patches/releng/10.1/ip_carp.c.diff diff --git a/sys/netinet/ip_carp.c b/sys/netinet/ip_carp.c index a170e34..0a3607e 100644 --- a/sys/netinet/ip_carp.c +++ b/sys/netinet/ip_carp.c @@ -532,8 +532,8 @@ carp6_input(struct mbuf **mp, int *offp, int proto) /* check if received on a valid carp interface */ if (m->m_pkthdr.rcvif->if_carp == NULL) { CARPSTATS_INC(carps_badif); - CARP_DEBUG("%s: packet received on non-carp interface: %s\n", - __func__, m->m_pkthdr.rcvif->if_xname); + //CARP_DEBUG("%s: packet received on non-carp interface: %s\n", + // __func__, m->m_pkthdr.rcvif->if_xname); m_freem(m); return (IPPROTO_DONE); } @@ -1195,8 +1195,7 @@ CARP_LOCK_ASSERT(sc); - if ((sc->sc_carpdev->if_flags & IFF_UP) == 0 || - sc->sc_carpdev->if_link_state != LINK_STATE_UP || + if ( (sc->sc_naddrs == 0 && sc->sc_naddrs6 == 0)) return; @@ -2001,27 +2000,11 @@ CARP_LOCK_ASSERT(sc); - if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) { - callout_stop(&sc->sc_ad_tmo); -#ifdef INET - callout_stop(&sc->sc_md_tmo); -#endif -#ifdef INET6 - callout_stop(&sc->sc_md6_tmo); -#endif - carp_set_state(sc, INIT); - carp_setrun(sc, 0); - if (!sc->sc_suppress) - carp_demote_adj(V_carp_ifdown_adj, "interface down"); - sc->sc_suppress = 1; - } else { carp_set_state(sc, INIT); carp_setrun(sc, 0); if (sc->sc_suppress) carp_demote_adj(-V_carp_ifdown_adj, "interface up"); sc->sc_suppress = 0; - } } static voidI'm not experienced enough with FreeBSD kernel debugging / drivers to really take this all that much further without it being fairly efforty – but it looks like:
- if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) {is not behaving correctly under hyper-v's network drivers. As for the cause for pfsync's woes I may try to take a look later.
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
-
@gonzopancho:
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.
-
@gonzopancho:
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.
patch is now in the next snapshot. (So much for all the key4ce bullshit.)
-
I can confirm that CARP is fully functional in Hyper-V with the most recent 2.2 snapshot available! Thanks to the folks at Microsoft for fixing it and getting us the patch. Those who'd like to use CARP in Hyper-V, check out the most recent 2.2 from snapshots.pfsense.org and let us know your experiences.
-
@cmb:
That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.
It's pretty clear from your code changes you have no idea what you're doing. Outside of the Hyper-V NIC driver bug, I'm going to call you out on this one - open just ONE actual bug with CARP at redmine.pfsense.org. Just one would suffice. You're spreading FUD and won't be able to do so.
I'll re-issue my call out now that CARP in 2.2 is working in Hyper-V as well, just in case anyone thinks they have any degree of credibility. We haven't gotten a single bug report from them to prove their point (because they're lying). Ours works with IPv6 too! Which key4ce's hacked up mess does not.
-
@gonzopancho:
Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.
I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.
Email me if you want to test this.
If it works well, we'll get it into pfSense 2.2
Jim
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
-
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.
-
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.
You can look in the pfsense-tools repo for the same patch.
-
Is this patch necessary for the pfsense 2.2 release ? If so, how do you apply it?
-