[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

jamroz

@rmh-0 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

RSC

I am complete pfSense noob. I installed it for the first time about an hour ago. The first thing that I noticed was the terrible throughput. Instead of using Google I decided to try the Community here and your solution worked. I was expecting this to be one of those one you wrestle with for hours but you nailed. Thanks!

inmarket

Here is the cause, the outcomes, and how to solve it if you have problems...

The cause:
It appears that Hyper-V on Server 2019 has a bug in its RSC code that causes packets to be corrupted. There are reports that this problem is solved in Server 2022 but I have not confirmed this.
This bug causes problems for ALL virtual machines provided they try to implement RSC code in their ethernet drivers. This means that Linux, Windows and FreeBSD are all affected if the version of the operating system supports RSC. For Linux this is all recent kernels and for Windows this is Windows 10, 11, Server 2019, Server 2022 at least.
Talking about pfSense, it is based on FreeBSD. FreeBSD starting supporting RSC in the hn driver as of 12.3 and 13.1. This therefore affects pfSense from Version 2.6

As from FreeBSD kernel 13.X, RSC has been turned off by default (thus fixing the problem) but can optionally be turned on. Also, it appears that pfSense 2.7 turns off RSC thus also fixing the problem. Note: these fixes do not solve the problem for Linux or Windows Guest VM's.

The outcomes:
The problem exhibits in one of two ways, either the corrupted packets go up the IP stack and cause "interesting" affects, or the kernel drops the corrupted packets. How this is handled changes from Operating system to operating system and even kernel version to kernel version (in particular Linux). It is this behaviour that causes the slow network performance as packets need to be retried etc.

Solutions:

The best solution is to turn off Hyper-V RSC on each virtual switch. The powershell commands to do this are listed in previous posts. This solves it for pfSense as well as Linux and Windows guests.
An alternative is to turn RSC off on each virtual ethernet adapter on each virtual machine. This is a poor solution because it requires a lot of systems to be touched AND also doesn't stick across a Hyper-V reboot due to another bug in Hyper-V (even though the UI indicates that it has stuck).
Another alternative is to upgrade pfSense to the 2.7 stream. For FreeBSD upgrade to the very latest kernel which turns off RSC by default. For Linux and Windows, no current solution is known. Note: this solution does not solve it for all operating systems.
Another alternative is to upgrade Hyper-V to a version that doesn't have the problem. I have heard that Server 2022 Hyper-V doesn't have the problem but I have not personally confirmed that.

Conclusion
This is a right royal stuff-up by Microsoft (not that they will admit it). Unfortunately the problem has probably existed for a long time but it has only become noticeable as operating systems have started implementing RSC support in their ethernet drivers.
Fortunately, there is a simple workaround (#1 above). I suspect however that this problem has probably killed RSC as a reliable performance enhancing strategy especially now that kernels are turning it off by default.

deanfourie

@inmarket great summary, thanks!

I must be honest, I tried disabling RSC with pfSense 2.6 and it didn't work for me?

I have now however installed pfSense + 22.05 and the issue appears to be gone however I would still like it turned off for future VMs.

What's the correct command for this? I know it's posted above but like I say, it did not work for me? Maybe I'm missing something.

Cheers

gogglespisano

@deanfourie Same here for Win 10 + 2.6.

I don't have -RSCEnabled on the VMNetworkAdapters and -SoftwareRSCEnabled on the VMSwitches were already $false.

I need more clues.
Does it need to be toggled regardless of current state?
Does the VMSwitch need to be not in use when setting it?

inmarket

Just as a quick note for those who think the solutions above don't work...

After updating RSC at the virtual switch (and on every host machine if you are using a cluster), you will need to restart each virtual machine. You may also have to restart Hyper-V (or rather Server 2019) for it to show its affect.

There may be other configuration problems that also result in slow performance but that is not what this is post is about. This post is specifically about the performance issues caused by RSC in Hyper-V Server 2019 in pfSense 2.6 that weren't there in pfSense 2.5

murdof

@inmarket I’m running windows server 2022.
The problem is there and tried all the recommended “solutions”.
Only downgrading to 2.5 or upgrading to 2.7 fixed the performance issue.

emigu

@werter said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

Decision: do not use hyper-v as virtualization platform ))
Better try Proxmox VE (open source)

Netgate creating a buggy version is not the virtualization plattforms fault.

Bob.Dig

@emigu It was a change in FreeBSD and with that an incompatibility of pfSense with Hyper-V. Netgate is not selling any device with Hyper-V on it, so it is nobody's fault to begin with.

johnpoz

@bob-dig said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

It was a change in FreeBSD and with that an incompatibility of pfSense with Hyper-V.

Its not just freebsd - if you do a google for hyper-v and RSC you will find lots of issues where it can cause slow network, etc.

emigu

@bob-dig Something that works in 2.5.2 splendid, while boasting in the official documentation that it supports Hyper-V:
https://docs.netgate.com/pfsense/en/latest/virtualization/index.html - https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

Which then stops working in 2.6.0 is most certainly Netgate's fault, more specifically the QA department.

Clearly there were 0.0% tests done with Hyper-V (One of the largest hypervisors in the world).

If you think it's acceptable not testing and verifying (A very simple test that can be automated) a new major version on a hypervisor listed in the documentation that happens to be the 2nd or 3rd most used hypervisor in the world, AND the one that's powering one of the biggest public clouds in the world - I don't really know how to counter that other than let's agree to disagree.

I certainly hope this lack of quality doesn't extend to netgates paid versions.

Bob.Dig

@johnpoz But the change in FreeBSD was that you can not disable it, so FreeBSD had to be changed again and now "patched" pfSense versions disable it by default because they can.

Bob.Dig

@emigu said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

Clearly there were 0.0% tests done with Hyper-V (One of the largest hypervisors in the world).

I am with you on that but Netgate doesn't make money with hyper-V. And Azure has no similarities with anything MS is selling for bare metal installations. So in the end you can be happy if it is running on your own "hardware" anyways, they don't have to support it.
And I think that hyper-V probably will be let down even from MS like everything else that is not running in their cloud. Let's see.

johnpoz

@emigu so with your logic - what about vmware, one of the top virtual hosting platforms on the planet.. They did not do full testing? Because they have the same issue with RSC that had to be fixed with an update.

https://kb.vmware.com/s/article/2129176

You could also put blame on MS with not fully testing hyper-v and all their possible guests and having issues with a new RSC stuff they were doing, etc.

emigu

@johnpoz said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

@emigu so with your logic - what about vmware, one of the top virtual hosting platforms on the planet.. They did not do full testing? Because they have the same issue with RSC that had to be fixed with an update.

https://kb.vmware.com/s/article/2129176

You could also put blame on MS with not fully testing hyper-v and all their possible guests and having issues with a new RSC stuff they were doing, etc.

Last time I checked VMWare isn't developing the product we're talking about that claims to support VMWare and Hyper-V.

Last time I checked Netgate develops pfSense and is responsible for the QA process of it's new releases.

Do you think otherwise?

johnpoz

@emigu you blame whoever you want to blame.. If it makes you feel better..

Bob.Dig

@emigu Next time check the beta and file a bug report if you depend upon running it on hyper-v I guess.
But I am not sure if that would stop them to release a new final, I really have no clue.

emigu

@bob-dig said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

@emigu said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

Clearly there were 0.0% tests done with Hyper-V (One of the largest hypervisors in the world).

I am with you on that but Netgate doesn't make money with hyper-V. And Azure has no similarities with anything MS is selling for bare metal installations. So in the end you can be happy if it is running on your own "hardware" anyways, they don't have to support it.
And I think that hyper-V probably will be let down even from MS like everything else that is not running in their cloud. Let's see.

What are you on about?

Azure runs on Hyper-V.

They've ran a modified version of Hyper-V (Obviously, same as AWS runs on an inhouse modified version of KVW, and previous XEN) - GCP has their complete own Hypervisor, Borg I believe it's called.

These days MS has released said modified Hyper-V: Azure Stack HCI.

They're obviously going to stop developing Hyper-V in the future, and leave it as is right now as a very lightweight hypervisor - which is why they're making HCI trial free for customers whom want to run their own HW with the same hypervisor Azure does for Azure integration.

Bob.Dig

@emigu Azure was tested if I remember correctly and didn't had that problem they said.

emigu

@bob-dig said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

@emigu Next time check the beta and file a bug report if you depend upon running it on hyper-v I guess.
But I am not sure if that would stop them to release a new final, I really have no clue.

No, I won't. My intention was to pay for pfSense and thus expect compability and testing to be done by the developers (Which is a big reason why you pay for software) but if such a basic test as automating a pfsense setup on each hypervisor, creating a random VM and running speedtest-cli isn't done - I wouldn't trust that product in production even when it's free.

My time is valuable and I do not intend on spending it doing basic testing for a company too lazy or incompetent (In the case def. the former) to QA their software. I prefer paying for that service and keeping my time to business values.

stephenw10

Azure was unaffected because it doesn't support the RSC vswitches that caused the problem as I understand it. Hence none of the testing we did there revealed this issue.

Steve