Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1
-
That's awesome. Full hyper-v support…yeah.
At the moment i have several installations of the snapshots series running smooth like a charm.Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service. -
Just in case anyone actually thinks these guys know what they're doing, and to debunk their FUD flinging. I installed their iso and spent a few minutes messing around.
Ours already works and tested with over 40 Gbit's speeds.
There aren't frames big enough to get > 40 Gb through pf at this point. We've done a great deal of analysis internally here, and continue to do so. There is no way you've made any significant performance improvements beyond what we've already done (having multiple FreeBSD committers on the payroll, one on contract who's a FreeBSD core team member and co-author of The Design and Implementation of FreeBSD book).
That our release is effectively 2.2 is total BS and you know it.
It's not BS at all. It looks like you changed a tiny fraction of 1 percent of the LOC excluding the web interface design changes, and what you did change there is broken. Out of curiosity I looked at what you're actually doing to "make CARP work", which was pretty entertaining (and scary for anyone who actually thought using your mess was a good idea).
It's complete with source code comments on their own changes including:
needs rework!That's the only really accurate code change I saw.
Then there's this gem:
Very ugly bitch failsafe...We're clearly dealing with professionals here, folks!
They replaced CARP with ucarp, which isn't a good idea for this type of use case, and did a poor implementation of it.
That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.
It's pretty clear from your code changes you have no idea what you're doing. Outside of the Hyper-V NIC driver bug, I'm going to call you out on this one - open just ONE actual bug with CARP at redmine.pfsense.org. Just one would suffice. You're spreading FUD and won't be able to do so.
I personally think our testing grounds are a bit more advanced then yours
That's hilarious given how much basic stuff you broke, like you trashed something in the interfaces code that creates an absurd slew of bugs. If you did any worthwhile degree of testing you wouldn't have thrown out such a mess to the world.
We've been doing this project years before your company even existed. Then for years while your company was fixing people's home computers. It's pretty clear who's competent at what they're doing here, the folks who are behind one of the most widely used network firewall distributions in the world and have over a decade of results to show for it (read: us), not the one that has some weird twisted definition of open source that matches nothing that exists but yet won't release the source to their "open source project."
@gonzopancho:
In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.This would fix the only real outstanding issues with Hyper-V, and would put both a pfSense and Microsoft "seal of approval" on the result.
And…game over for key4ce. ;D I'm sure there are plenty of home computers they can go fix.
-
Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.That's something we'll get into with the folks at Microsoft, but generally speaking for this type of use case, you don't do full VM backups at all. Have HA systems in place, and config backups ready to restore, and you're good. Generally just a waste of disk/tape/cloud/whatever-you-backup-to space to get full backups of a system like pfSense where it's really easy and fast to rebuild from scratch and restore (probably faster than restoring a full disk VM backup actually).
-
@gonzopancho:
In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.I can definitely confirm it's a Hyper-V NIC driver issue for carp but I don't believe it to multicast related in carp's case – it looks to be related to the NIC state information as a quick super-hacky-terrible patch I did to the carp kernel code has resulted in functional carp on a test setup for me in Hyper-V. (I haven't tried to get pfsync going in the same setup -- it doesn't seem to be syncing state properly according to pftop.) I added it to the existing ip_carp.c.diff in the patch list:
root@freebsd:~ # cat /home/pfsense/tools/patches/releng/10.1/ip_carp.c.diff diff --git a/sys/netinet/ip_carp.c b/sys/netinet/ip_carp.c index a170e34..0a3607e 100644 --- a/sys/netinet/ip_carp.c +++ b/sys/netinet/ip_carp.c @@ -532,8 +532,8 @@ carp6_input(struct mbuf **mp, int *offp, int proto) /* check if received on a valid carp interface */ if (m->m_pkthdr.rcvif->if_carp == NULL) { CARPSTATS_INC(carps_badif); - CARP_DEBUG("%s: packet received on non-carp interface: %s\n", - __func__, m->m_pkthdr.rcvif->if_xname); + //CARP_DEBUG("%s: packet received on non-carp interface: %s\n", + // __func__, m->m_pkthdr.rcvif->if_xname); m_freem(m); return (IPPROTO_DONE); } @@ -1195,8 +1195,7 @@ CARP_LOCK_ASSERT(sc); - if ((sc->sc_carpdev->if_flags & IFF_UP) == 0 || - sc->sc_carpdev->if_link_state != LINK_STATE_UP || + if ( (sc->sc_naddrs == 0 && sc->sc_naddrs6 == 0)) return; @@ -2001,27 +2000,11 @@ CARP_LOCK_ASSERT(sc); - if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) { - callout_stop(&sc->sc_ad_tmo); -#ifdef INET - callout_stop(&sc->sc_md_tmo); -#endif -#ifdef INET6 - callout_stop(&sc->sc_md6_tmo); -#endif - carp_set_state(sc, INIT); - carp_setrun(sc, 0); - if (!sc->sc_suppress) - carp_demote_adj(V_carp_ifdown_adj, "interface down"); - sc->sc_suppress = 1; - } else { carp_set_state(sc, INIT); carp_setrun(sc, 0); if (sc->sc_suppress) carp_demote_adj(-V_carp_ifdown_adj, "interface up"); sc->sc_suppress = 0; - } } static voidI'm not experienced enough with FreeBSD kernel debugging / drivers to really take this all that much further without it being fairly efforty – but it looks like:
- if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) {is not behaving correctly under hyper-v's network drivers. As for the cause for pfsync's woes I may try to take a look later.
-
That's interesting (thank you!).
I'll look at 'why' a bit later.
-
Hrm pfsync actually appears to work just fine (I fell into the same trap as https://forum.pfsense.org/index.php?topic=24550.0). Bad config on my part. I did enable both MAC spoofing http://blogs.technet.com/b/jhoward/archive/2009/05/21/new-in-hyper-v-windows-server-2008-r2-part-2-mac-spoofing.aspx and I disabled "Microsoft Windows Filtering Platform" in the Extensions portion of the vswitch.. although I'm not certain that was necessary – I did it a while before I realized my config issue.. I'll try reverting it/cleaning up the configs later.
I'll also try putting some real connections / clients behind it and doing some testing (I can drop it in place of my existing home router setup with some work/cabling).I'd love to hear what you find re:the driver/state stuff.. Thanks!
-
I recalled the vtnet driver of virtio originally had the same problem as here. Though looking through the change log, I don't see any commit logs that mention fixing that issue. It was fixed and works fine now, was hoping to be able to easily pinpoint the commit that fixed that but it's not obvious at a glance.
http://svnweb.freebsd.org/base/releng/10.1/sys/dev/virtio/network/if_vtnet.c?view=logI agree that multicast doesn't seem to be completely broken, some things work. Hopefully I'll get some time in the next week to get back to a bit more testing here, but it'll likely be this weekend at best as I'm focused on getting 2.2 release out at the moment.
-
@cmb:
Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.That's something we'll get into with the folks at Microsoft,
That's awesome… everything what can't run in virtualized environments anymore is a lack of support and not in "our" (my collegues for example. ;) ) focus anymore.
So I was really excited to see the 2.2 snapshots running like a charm at the moment.
Thanks to you guys. thumps up.… but generally speaking for this type of use case, you don't do full VM backups at all.
I don't understand. Why?
Have HA systems in place, and config backups ready to restore, and you're good. Generally just a waste of disk/tape/cloud/whatever-you-backup-to space to get full backups of a system like pfSense where it's really easy and fast to rebuild from scratch and restore (probably faster than restoring a full disk VM backup actually).
In my case it doesn't. It would need a much more automatic install process I can't afford or setting up at the moment.
HA is a bit to much and a recovery run doesn't take as long as I would need to reinstall and setup all the small needed things. It integrates at the moment much better in this kind of environment I'm running. So there's no need for a full Hyper-V support at the moment for me but it will be… sooner or later. :)
I'm on DPM and use pfsense at the moment more for testing purposes.waste of drive space sounds a bit nineties. fortunately dedup and reliable backup solutions kills that argument at all. ;)
Btw.
I would like to spend some money but I couldn't get a way to support pfsense except buying the book.
-
FWIW throughput looks fine.. most of the time (>30MB/s to google hosts and others) I do have some odd exceptions which I will look into (valve used to provide at least 40MB/s for static content but now I see only 2MB/s).
Inside the lan looks good - iperf sees about 730Mb/s bi-directional with very little config - from a Linux box in hyper-v on a different machine:
lburton@downquark ~ # iperf -c lburton-router -i 10 -t 60 -d ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to lburton-router, TCP port 5001 TCP window size: 183 KByte (default) ------------------------------------------------------------ [ 3] local 44.44.92.2 port 48434 connected with 44.44.92.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 824 MBytes 691 Mbits/sec [ 3] 10.0-20.0 sec 905 MBytes 759 Mbits/sec [ 3] 20.0-30.0 sec 882 MBytes 740 Mbits/sec [ 3] 30.0-40.0 sec 878 MBytes 737 Mbits/sec [ 3] 40.0-50.0 sec 875 MBytes 734 Mbits/sec [ 3] 50.0-60.0 sec 866 MBytes 727 Mbits/sec [ 3] 0.0-60.0 sec 5.11 GBytes 731 Mbits/secAlso while I kinda hate it.. I figure I might as well: ;) :P
One thing I see in the hyper-v drivers that I don't see in any of the other drivers (looking at the intel NIC drivers for example) is that during init they manually set if_flags (not that it probably matters.. I'm guessing the flags are changed by the OS code when the interface is configured but I haven't looked)
ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c#L265
Edit:
external iperf is having similar issues to valve/others – I see good outbound throughput and poor inbound..------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to iperf.scottlinux.com, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 5] local 44.44.92.5 port 42344 connected with 173.230.156.66 port 5001 [ ID] Interval Transfer Bandwidth [ 5] 0.0-30.0 sec 2.76 GBytes 790 Mbits/sec [ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [ 4] local 44.44.92.5 port 5001 connected with 173.230.156.66 port 35736 [ 4] 0.0-30.1 sec 176 MBytes 49.1 Mbits/sec [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet) -
Can you elaborate on "That path leads to madness."?
I've been running a standard/basic pfsense 2.1.2 -> 2.1.3 -> 2.1.5 (2.1.3 and 2.1.5 were upgrades from 2.1.2) without any significant issues on Hyper-V Server 2012 R2.
I've considered trying the 2.2 betas but 2.1.x with these drivers has been stable enough for me that I figured that I'd wait for a 2.2 stable release
@gonzopancho:
I don't recommend even trying 2.1 on Hyper-V. That path leads to madness.
2.2 is in pretty good shape. There are 10 (count them, 10) outstanding bugs in the way of cutting a release candidate.
one of them concerns CARP.
try it out! (It's a VM environment, what do you have to lose?)
Jim
-
Can you elaborate on "That path leads to madness."?
The most stable option available for Hyper-V is 2.2-RC (or newer if available and you're reading this in the future). By far. You're effectively hacking together something on a base OS that isn't officially supported by Microsoft. I'm sure it works fine in some specific circumstances. It is, without question, a riskier proposition than running 2.2 and has been for some time.
-
Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.
I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.
Email me if you want to test this.
If it works well, we'll get it into pfSense 2.2
Jim
-
@gonzopancho:
In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.I can definitely confirm it's a Hyper-V NIC driver issue for carp but I don't believe it to multicast related in carp's case – it looks to be related to the NIC state information as a quick super-hacky-terrible patch I did to the carp kernel code has resulted in functional carp on a test setup for me in Hyper-V. (I haven't tried to get pfsync going in the same setup -- it doesn't seem to be syncing state properly according to pftop.) I added it to the existing ip_carp.c.diff in the patch list:
root@freebsd:~ # cat /home/pfsense/tools/patches/releng/10.1/ip_carp.c.diff diff --git a/sys/netinet/ip_carp.c b/sys/netinet/ip_carp.c index a170e34..0a3607e 100644 --- a/sys/netinet/ip_carp.c +++ b/sys/netinet/ip_carp.c @@ -532,8 +532,8 @@ carp6_input(struct mbuf **mp, int *offp, int proto) /* check if received on a valid carp interface */ if (m->m_pkthdr.rcvif->if_carp == NULL) { CARPSTATS_INC(carps_badif); - CARP_DEBUG("%s: packet received on non-carp interface: %s\n", - __func__, m->m_pkthdr.rcvif->if_xname); + //CARP_DEBUG("%s: packet received on non-carp interface: %s\n", + // __func__, m->m_pkthdr.rcvif->if_xname); m_freem(m); return (IPPROTO_DONE); } @@ -1195,8 +1195,7 @@ CARP_LOCK_ASSERT(sc); - if ((sc->sc_carpdev->if_flags & IFF_UP) == 0 || - sc->sc_carpdev->if_link_state != LINK_STATE_UP || + if ( (sc->sc_naddrs == 0 && sc->sc_naddrs6 == 0)) return; @@ -2001,27 +2000,11 @@ CARP_LOCK_ASSERT(sc); - if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) { - callout_stop(&sc->sc_ad_tmo); -#ifdef INET - callout_stop(&sc->sc_md_tmo); -#endif -#ifdef INET6 - callout_stop(&sc->sc_md6_tmo); -#endif - carp_set_state(sc, INIT); - carp_setrun(sc, 0); - if (!sc->sc_suppress) - carp_demote_adj(V_carp_ifdown_adj, "interface down"); - sc->sc_suppress = 1; - } else { carp_set_state(sc, INIT); carp_setrun(sc, 0); if (sc->sc_suppress) carp_demote_adj(-V_carp_ifdown_adj, "interface up"); sc->sc_suppress = 0; - } } static voidI'm not experienced enough with FreeBSD kernel debugging / drivers to really take this all that much further without it being fairly efforty – but it looks like:
- if (sc->sc_carpdev->if_link_state != LINK_STATE_UP || - !(sc->sc_carpdev->if_flags & IFF_UP)) {is not behaving correctly under hyper-v's network drivers. As for the cause for pfsync's woes I may try to take a look later.
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
-
@gonzopancho:
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.
-
@gonzopancho:
The patch is actually somewhat simpler than this, and is completely contained in hv_netvsc_drv_freebsd.c.
Agreed – I'm using the patch from the BSD integration team @ MS as of 3 days ago and it appears to resolve the NIC state issues as expected.
patch is now in the next snapshot. (So much for all the key4ce bullshit.)
-
I can confirm that CARP is fully functional in Hyper-V with the most recent 2.2 snapshot available! Thanks to the folks at Microsoft for fixing it and getting us the patch. Those who'd like to use CARP in Hyper-V, check out the most recent 2.2 from snapshots.pfsense.org and let us know your experiences.
-
@cmb:
That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.
It's pretty clear from your code changes you have no idea what you're doing. Outside of the Hyper-V NIC driver bug, I'm going to call you out on this one - open just ONE actual bug with CARP at redmine.pfsense.org. Just one would suffice. You're spreading FUD and won't be able to do so.
I'll re-issue my call out now that CARP in 2.2 is working in Hyper-V as well, just in case anyone thinks they have any degree of credibility. We haven't gotten a single bug report from them to prove their point (because they're lying). Ours works with IPv6 too! Which key4ce's hacked up mess does not.
-
@gonzopancho:
Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.
I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.
Email me if you want to test this.
If it works well, we'll get it into pfSense 2.2
Jim
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
-
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.
-
I'm trying to get CARP on FreeBSD 10.1 to work on Hyper-V. Is it possible to share the 45 line patch?
Just to close the loop on this, the patch can be found at https://bz-attachments.freebsd.org/attachment.cgi?id=151515. MAC spoofing also needs to be enabled on the virtual NIC.
You can look in the pfsense-tools repo for the same patch.