Hyper-V ICS 1.0 (w/Synthethic Network Driver) for pfSense 2.1 & 2.1.1

hege

@gonzopancho:

You claim, "We have modified CARP extensively to work on any virtual platform" - as if it didn't already. (It does!)

Sorry, but thats not true, you can't use CARP with Hyper-V with current 2.2 (stucks in init as key4ce said)

rainabba

@gonzopancho:

And in our experience, pfSense works fine on Hyper-V.

The Key4ce folks are allowed (of course) to make other claims.

Let the market decide.

Everywhere I've found a solution, it is to use legacy nics with a disclaimer that it should not be used in production. Is there another solution or is that advice wrong?

Afterthought: …or is this a reference to 2.2? I'm out of the loop and didn't realize it was available yet since I've only been looking at RELEASE. I realize it is considered BETA, but would anyone with the project be willing to call it stable enough for more basic use in production (NAT, Firewall, IPSEC)?

Thanks in advance

rainabba

@key4ce:

Jim, again we only licensed our code, your code stays in tact and so does your license.
The code as a whole is then known as dual licensed and it is shown very clearly (altho the legal page is still in the making to clear it out further).
Thats why where in some sections where we changed you will find your License.txt and our License-VirtualPF.txt.

Bit odd you can claim we don't care about licensing when we spend so much time argueing about it tho.
also not something i care to do again. we checked it over with multiple legal shavy people who all said what we did is actually over doing what is required.

Which makes me wonder 1 or 2 things:

Your either not fully aware of legal laws and limitations of your own legal aspects

Your very aware but pretend not to know and scare people out of it and/or to see what it is they exactly know.

That i don't have to put credits on every website page: that is correct. you can ask your lawyers and they will tell you that claim is not possible under international laws. (and is also not written in your Legal section.. please read your own legal section careful.)

We did however keep it in the license page within the software and everyone can see it in full view before download.

We did not release any git code yet as it's still fully compatible with PFSense (including the builder).
I doubt you want agreements and CLA's to get crossed.

However we expect that part to be 100% available for public within about 3 weeks (after we fully integrated freebsd 10.1 stable)

The agreement Angelo signed does not limit the use or fork of any code so we still comply to that too.

As far a tech:
Ours already works and tested with over 40 Gbit's speeds.

The gui is indeed not finished, if you check the roadmap –> it was only made in 5 days and got quite alot left to redo (it's why we only made a Beta release their not intended on being perfect but atleast function wise you can test)

The Ejecting CD on what Virtual Environment is it tested? (version)
as we confirmed it working on vmware esxi, and vmware desktop (both latest versions).

That our release is effectively 2.2 is total BS and you know it.
CARP was just one, supporting 10 Gbit+, supporting AND automatically installing any virtual platform driver automatically based up on platform detection would be another. gui improvements is one, and translations to Chinese, Dutch, German are on their way.

Much more features have been added and will be carefully detailed before November 30th

It's just the first release, it won't truly define our selfs just yet. that will come with stable 2.

That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.
But ofcourse feel free to believe it is, I personally think our testing grounds are a bit more advanced then yours (as below 1 Gbit speeds won't be accepted by any of our customers, not even our own environment can go below 20 Gbit else we pinch our customers off lol).

Regards,
Marco

For someone who wouldn't have a project without everything provided in the pfSense base, you come across very rude, unappreciative and ungrateful. You sound like you're using open-source ideals as a defense, rather than being a good player in the scene. If the claims made by other on the PFSENSE forums aren't true, what do you gain by arguing; especially when they're providing evidence?

Generally, only guilty parties or those in harm of penalty feel the need to continuously justify themselves because the truth speaks for itself.

Furthermore, you're enforcing what I said earlier; "I" am the type of customer you're going to be pursuing if you are actually trying to build a business, and you're doing everything in your power to ensure that I don't go anywhere near testing, let alone production use of your build. If your build is so different and superior, why not do it from scratch and stay away from these forums so that you can have more control, less licensing issues and more respect for the finished product. Of course, I'm just "a customer" so what does my opinion matter? :)

I will step out of the drama on that note and continue my search for how to run pfSense, stable, on Hyper-V.

@hege:

@gonzopancho:

You claim, "We have modified CARP extensively to work on any virtual platform" - as if it didn't already. (It does!)

Sorry, but thats not true, you can't use CARP with Hyper-V with current 2.2 (stucks in init as key4ce said)

We're not done with CARP in 2.2.

I don't recommend even trying 2.1 on Hyper-V. That path leads to madness.

2.2 is in pretty good shape. There are 10 (count them, 10) outstanding bugs in the way of cutting a release candidate.

one of them concerns CARP.

try it out! (It's a VM environment, what do you have to lose?)

Jim

@key4ce:

Jim, again we only licensed our code, your code stays in tact and so does your license.
The code as a whole is then known as dual licensed and it is shown very clearly (altho the legal page is still in the making to clear it out further).

Your continued assertion is that you can relicense our code. You. Can. Not.

@key4ce:

Thats why where in some sections where we changed you will find your License.txt and our License-VirtualPF.txt.

I don't care about your license. I care about mine.

@key4ce:

Bit odd you can claim we don't care about licensing when we spend so much time argueing about it tho.
also not something i care to do again. we checked it over with multiple legal shavy people who all said what we did is actually over doing what is required.

I don't claim that you "don't care about licensing". I claim that you have breeched the license. Perhaps you don't care that you've breeched.

@key4ce:

Which makes me wonder 1 or 2 things:

Your either not fully aware of legal laws and limitations of your own legal aspects

Your very aware but pretend not to know and scare people out of it and/or to see what it is they exactly know.

Begging the question via assertion of incorrect facts isn't going to work. I'm very aware of the law, and pay attorneys to provide cover.

@key4ce:

That i don't have to put credits on every website page: that is correct. you can ask your lawyers and they will tell you that claim is not possible under international laws. (and is also not written in your Legal section.. please read your own legal section careful.)

You 'signed' a license that comes with certain obligations. "International laws" (there is no such thing) certainly allow you to bind yourself to certain acts in exchange for consideration.

@key4ce:

We did not release any git code yet as it's still fully compatible with PFSense (including the builder).

Yes, you've forked a proprietary copy, and it's fully allowed. have fun with your proprietary clone, and stay off the pfSense forums with any further discussion about same.

@key4ce:

Ours already works and tested with over 40 Gbit's speeds.

You didn't make pf work at 40Gbps, except possibly with very large packets. Perhaps you mean that you've included support
for some 40Gbps adapter, in which case… BFD! pfSense 2.2 supports the Chelsio T5.

But again.. stay off the forums until your fork is open sourced. I'm not hosting people who aren't willing to play on that field, and make no mistake ... you are (or were) a guest here.

key4ce

Jim, keep it simple, check with your legal parties and see if i complied to the agreement AND license.
I checked with mine which simply said: we don't even need to give so much credit as we did.

If you want i can publish the entire report of their findings.

please note we aren't making use of any of it besides that we don't have to place your copyrights on every single website page of VirtualPF.

That i am a guest here: i am well aware of that.
I was just answering another guests question without accusation or anything else.
same as we politely still contributed some simple fixes to your repo which was bothering people on your forum.

Where you fly off again and take the opportunity to bash a little bit more.

That you think we did or not do something like 40 Gbit: We work with Datacenters you can trust them to do simple tests like that.
That you again think we aren't opensource: we can release your builders as is, but that means anyone can download them from us and use them on PFSense.

I was being polite by waiting untill we made those only work on VirtualPF so it doesn't affect PFSense.
If you don't care about that, then our source will be out in the open within end today, please remember that includes the builders and without a CLA, we only apply CLA for contributors not to view the source it self (as viewing source and using it has nothing to do with CLA).

Regards,
Marco

@key4ce:

Jim, keep it simple, check with your legal parties and see if i complied to the agreement AND license.
I checked with mine which simply said: we don't even need to give so much credit as we did.

You signed a license agreement, or you did not. It's simple.

@key4ce:

I was being polite by waiting untill we made those only work on VirtualPF so it doesn't affect PFSense.
If you don't care about that, then our source will be out in the open within end today, please remember that includes the builders and without a CLA, we only apply CLA for contributors not to view the source it self (as viewing source and using it has nothing to do with CLA).

I'm not sure what you're saying here.

In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.

This would fix the only real outstanding issues with Hyper-V, and would put both a pfSense and Microsoft "seal of approval" on the result.

As before, this project is scheduled after pfSense version 2.2 is released.

Rooter

That's awesome. Full hyper-v support…yeah.
At the moment i have several installations of the snapshots series running smooth like a charm.

Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.

cmb

Just in case anyone actually thinks these guys know what they're doing, and to debunk their FUD flinging. I installed their iso and spent a few minutes messing around.

@key4ce:

Ours already works and tested with over 40 Gbit's speeds.

There aren't frames big enough to get > 40 Gb through pf at this point. We've done a great deal of analysis internally here, and continue to do so. There is no way you've made any significant performance improvements beyond what we've already done (having multiple FreeBSD committers on the payroll, one on contract who's a FreeBSD core team member and co-author of The Design and Implementation of FreeBSD book).

@key4ce:

That our release is effectively 2.2 is total BS and you know it.

It's not BS at all. It looks like you changed a tiny fraction of 1 percent of the LOC excluding the web interface design changes, and what you did change there is broken. Out of curiosity I looked at what you're actually doing to "make CARP work", which was pretty entertaining (and scary for anyone who actually thought using your mess was a good idea).

It's complete with source code comments on their own changes including:

needs rework!

That's the only really accurate code change I saw.

Then there's this gem:

Very ugly bitch failsafe...

We're clearly dealing with professionals here, folks!

They replaced CARP with ucarp, which isn't a good idea for this type of use case, and did a poor implementation of it.

@key4ce:

That you think CARP is fully working: sorry we already confirmed this: it wasn't, even this week we found some new bugs with it.

It's pretty clear from your code changes you have no idea what you're doing. Outside of the Hyper-V NIC driver bug, I'm going to call you out on this one - open just ONE actual bug with CARP at redmine.pfsense.org. Just one would suffice. You're spreading FUD and won't be able to do so.

@key4ce:

I personally think our testing grounds are a bit more advanced then yours

That's hilarious given how much basic stuff you broke, like you trashed something in the interfaces code that creates an absurd slew of bugs. If you did any worthwhile degree of testing you wouldn't have thrown out such a mess to the world.

We've been doing this project years before your company even existed. Then for years while your company was fixing people's home computers. It's pretty clear who's competent at what they're doing here, the folks who are behind one of the most widely used network firewall distributions in the world and have over a decade of results to show for it (read: us), not the one that has some weird twisted definition of open source that matches nothing that exists but yet won't release the source to their "open source project."

@gonzopancho:

In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.

This would fix the only real outstanding issues with Hyper-V, and would put both a pfSense and Microsoft "seal of approval" on the result.

And…game over for key4ce. ;D I'm sure there are plenty of home computers they can go fix.

cmb

@Rooter:

Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.

That's something we'll get into with the folks at Microsoft, but generally speaking for this type of use case, you don't do full VM backups at all. Have HA systems in place, and config backups ready to restore, and you're good. Generally just a waste of disk/tape/cloud/whatever-you-backup-to space to get full backups of a system like pfSense where it's really easy and fast to rebuild from scratch and restore (probably faster than restoring a full disk VM backup actually).

lburton

@gonzopancho:

In any case, I spoke with the Hyper-V and Azure people from Microsoft at the FreeBSD developer summit earlier this week.
They are quite interested in working with us to develop a fully-certified Hyper-V image, including fixes (via Microsoft)
to the obvious multicast issues with CARP/pfSync in the underlying drivers. They're also interested in more extensive testing, including performance-related work, and tuning.

I can definitely confirm it's a Hyper-V NIC driver issue for carp but I don't believe it to multicast related in carp's case – it looks to be related to the NIC state information as a quick super-hacky-terrible patch I did to the carp kernel code has resulted in functional carp on a test setup for me in Hyper-V. (I haven't tried to get pfsync going in the same setup -- it doesn't seem to be syncing state properly according to pftop.) I added it to the existing ip_carp.c.diff in the patch list:


root@freebsd:~ # cat /home/pfsense/tools/patches/releng/10.1/ip_carp.c.diff
diff --git a/sys/netinet/ip_carp.c b/sys/netinet/ip_carp.c
index a170e34..0a3607e 100644
--- a/sys/netinet/ip_carp.c
+++ b/sys/netinet/ip_carp.c
@@ -532,8 +532,8 @@ carp6_input(struct mbuf **mp, int *offp, int proto)
        /* check if received on a valid carp interface */
        if (m->m_pkthdr.rcvif->if_carp == NULL) {
                CARPSTATS_INC(carps_badif);
-               CARP_DEBUG("%s: packet received on non-carp interface: %s\n",
-                   __func__, m->m_pkthdr.rcvif->if_xname);
+               //CARP_DEBUG("%s: packet received on non-carp interface: %s\n",
+               //    __func__, m->m_pkthdr.rcvif->if_xname);
                m_freem(m);
                return (IPPROTO_DONE);
        }
@@ -1195,8 +1195,7 @@

        CARP_LOCK_ASSERT(sc);

-       if ((sc->sc_carpdev->if_flags & IFF_UP) == 0 ||
-           sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
+       if (
            (sc->sc_naddrs == 0 && sc->sc_naddrs6 == 0))
                return;

@@ -2001,27 +2000,11 @@

        CARP_LOCK_ASSERT(sc);

-       if (sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
-           !(sc->sc_carpdev->if_flags & IFF_UP)) {
-               callout_stop(&sc->sc_ad_tmo);
-#ifdef INET
-               callout_stop(&sc->sc_md_tmo);
-#endif
-#ifdef INET6
-               callout_stop(&sc->sc_md6_tmo);
-#endif
-               carp_set_state(sc, INIT);
-               carp_setrun(sc, 0);
-               if (!sc->sc_suppress)
-                       carp_demote_adj(V_carp_ifdown_adj, "interface down");
-               sc->sc_suppress = 1;
-       } else {
                carp_set_state(sc, INIT);
                carp_setrun(sc, 0);
                if (sc->sc_suppress)
                        carp_demote_adj(-V_carp_ifdown_adj, "interface up");
                sc->sc_suppress = 0;
-       }
 }

 static void

I'm not experienced enough with FreeBSD kernel debugging / drivers to really take this all that much further without it being fairly efforty – but it looks like:


-       if (sc->sc_carpdev->if_link_state != LINK_STATE_UP ||
-           !(sc->sc_carpdev->if_flags & IFF_UP)) {

is not behaving correctly under hyper-v's network drivers. As for the cause for pfsync's woes I may try to take a look later.

That's interesting (thank you!).

I'll look at 'why' a bit later.

lburton

Hrm pfsync actually appears to work just fine (I fell into the same trap as https://forum.pfsense.org/index.php?topic=24550.0). Bad config on my part. I did enable both MAC spoofing http://blogs.technet.com/b/jhoward/archive/2009/05/21/new-in-hyper-v-windows-server-2008-r2-part-2-mac-spoofing.aspx and I disabled "Microsoft Windows Filtering Platform" in the Extensions portion of the vswitch.. although I'm not certain that was necessary – I did it a while before I realized my config issue.. I'll try reverting it/cleaning up the configs later.
I'll also try putting some real connections / clients behind it and doing some testing (I can drop it in place of my existing home router setup with some work/cabling).

I'd love to hear what you find re:the driver/state stuff.. Thanks!

cmb

I recalled the vtnet driver of virtio originally had the same problem as here. Though looking through the change log, I don't see any commit logs that mention fixing that issue. It was fixed and works fine now, was hoping to be able to easily pinpoint the commit that fixed that but it's not obvious at a glance.
http://svnweb.freebsd.org/base/releng/10.1/sys/dev/virtio/network/if_vtnet.c?view=log

I agree that multicast doesn't seem to be completely broken, some things work. Hopefully I'll get some time in the next week to get back to a bit more testing here, but it'll likely be this weekend at best as I'm focused on getting 2.2 release out at the moment.

Rooter

@cmb:

@Rooter:

Only worry is the lack of the kvp tools ( i guess).
Live backup can be interrupting the service.

That's something we'll get into with the folks at Microsoft,

That's awesome… everything what can't run in virtualized environments anymore is a lack of support and not in "our" (my collegues for example. ;) ) focus anymore.
So I was really excited to see the 2.2 snapshots running like a charm at the moment.
Thanks to you guys. thumps up.

… but generally speaking for this type of use case, you don't do full VM backups at all.

I don't understand. Why?

Have HA systems in place, and config backups ready to restore, and you're good. Generally just a waste of disk/tape/cloud/whatever-you-backup-to space to get full backups of a system like pfSense where it's really easy and fast to rebuild from scratch and restore (probably faster than restoring a full disk VM backup actually).

In my case it doesn't. It would need a much more automatic install process I can't afford or setting up at the moment.
HA is a bit to much and a recovery run doesn't take as long as I would need to reinstall and setup all the small needed things. It integrates at the moment much better in this kind of environment I'm running. So there's no need for a full Hyper-V support at the moment for me but it will be… sooner or later. :)
I'm on DPM and use pfsense at the moment more for testing purposes.

waste of drive space sounds a bit nineties. fortunately dedup and reliable backup solutions kills that argument at all. ;)

Btw.

I would like to spend some money but I couldn't get a way to support pfsense except buying the book.

lburton

FWIW throughput looks fine.. most of the time (>30MB/s to google hosts and others) I do have some odd exceptions which I will look into (valve used to provide at least 40MB/s for static content but now I see only 2MB/s).

Inside the lan looks good - iperf sees about 730Mb/s bi-directional with very little config - from a Linux box in hyper-v on a different machine:


lburton@downquark ~ # iperf -c lburton-router -i 10 -t 60 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to lburton-router, TCP port 5001
TCP window size:  183 KByte (default)
------------------------------------------------------------
[  3] local 44.44.92.2 port 48434 connected with 44.44.92.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   824 MBytes   691 Mbits/sec
[  3] 10.0-20.0 sec   905 MBytes   759 Mbits/sec
[  3] 20.0-30.0 sec   882 MBytes   740 Mbits/sec
[  3] 30.0-40.0 sec   878 MBytes   737 Mbits/sec
[  3] 40.0-50.0 sec   875 MBytes   734 Mbits/sec
[  3] 50.0-60.0 sec   866 MBytes   727 Mbits/sec
[  3]  0.0-60.0 sec  5.11 GBytes   731 Mbits/sec

Also while I kinda hate it.. I figure I might as well: ;) :P

One thing I see in the hyper-v drivers that I don't see in any of the other drivers (looking at the intel NIC drivers for example) is that during init they manually set if_flags (not that it probably matters.. I'm guessing the flags are changed by the OS code when the interface is configured but I haven't looked)


ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST;

https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c#L265

Edit:
external iperf is having similar issues to valve/others – I see good outbound throughput and poor inbound..


------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to iperf.scottlinux.com, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  5] local 44.44.92.5 port 42344 connected with 173.230.156.66 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-30.0 sec  2.76 GBytes   790 Mbits/sec
[  5] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
[  4] local 44.44.92.5 port 5001 connected with 173.230.156.66 port 35736
[  4]  0.0-30.1 sec   176 MBytes  49.1 Mbits/sec
[  4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

cpresco

Can you elaborate on "That path leads to madness."?

I've been running a standard/basic pfsense 2.1.2 -> 2.1.3 -> 2.1.5 (2.1.3 and 2.1.5 were upgrades from 2.1.2) without any significant issues on Hyper-V Server 2012 R2.

I've considered trying the 2.2 betas but 2.1.x with these drivers has been stable enough for me that I figured that I'd wait for a 2.2 stable release

@gonzopancho:

I don't recommend even trying 2.1 on Hyper-V. That path leads to madness.

2.2 is in pretty good shape. There are 10 (count them, 10) outstanding bugs in the way of cutting a release candidate.

one of them concerns CARP.

try it out! (It's a VM environment, what do you have to lose?)

Jim

cmb

@cpresco:

Can you elaborate on "That path leads to madness."?

The most stable option available for Hyper-V is 2.2-RC (or newer if available and you're reading this in the future). By far. You're effectively hacking together something on a base OS that isn't officially supported by Microsoft. I'm sure it works fine in some specific circumstances. It is, without question, a riskier proposition than running 2.2 and has been for some time.

Microsoft let me know last night that they've developed a simple (45 line) patch to FreeBSD's hyper-v net driver (https://github.com/freebsd/freebsd/blob/master/sys/dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c) that fixes the issue with CARP.

I'm looking for no more than 5 well-qualified volunteers to test a custom-build with this patch included, on Hyper-V.

Email me if you want to test this.

If it works well, we'll get it into pfSense 2.2

Jim