Performance on Hyper-V broken by Microsoft Dec 13 patches?
-
I've got a Hyper-V system that's been working well under 2012r2. I had started upgrading to Server 2022 and initial tests worked fine. But on Dec 13 M$ patch set including KB5021249 appears to throttle somehow throughput down to 1-3 mbit/sec. Has anyone else seen this and figured out a workaround?
M$ has flagged vm performance issues with this patch in some circumstances. https://support.microsoft.com/en-us/topic/december-13-2022-kb5021249-os-build-20348-1366-d5fe7608-bc9d-4055-a88c-fb2fd3d5fd45
-
@ksdehoff said in Performance on Hyper-V broken by Microsoft Dec 13 patches?:
Has anyone else seen this ?
No. But your problems might be related to this.
-
I checked that thread - I have all the pfsense hardware acceleration features disabled: Offloading, alt-q. Still sucks and I actually noted a 'huge' decrease in throughput doing this: from 1 mbit/sec to 0.3 mbit/sec!
-
@ksdehoff Are you using 2.6? Then you have that problem. There is no setting inside of pfSense which would help against that and also none in the Windows GUI.
-
@bob-dig Yes 2.6 stable. I just put a 2.7 development version together from last night and it works! I have to decide on sticking with 2.6 stable on a 2012r2 temporary platform or 2.7 development on my production hyperv platform….. Are development builds typically pretty robust for simple traffic?
-
@ksdehoff I am using the plus version. If you have to use the development version, don't update it regularly and have a snapshot ready anytime you do update to roll back... it is risky.
-
Well this is annoying. My 'path to production' was a 2.7 build on hyperv 2012r2 to confirm a baseline that works. I then migrated it to hyperv 2022 to confirm it worked still with the same image - which it did. But I did not reboot the OS once I migrated it and once I rebooted on 2022 I'm back to the 1 mbit/sec terrible throughput
-
@ksdehoff That sounds horrible, to me. I hope netgate has fixed it, hadn't any time to test it myself. Maybe I will test the new beta version of plus but I don't know how this compares to 2.7.
-
Check the RSC sysctl. It should default to off but it's still possible to enable it and hit this issue.
It is in 2.7: https://github.com/pfsense/FreeBSD-src/commit/80c3eb7bc64b745b2a0bda953e29a26e32dff9d7
Steve
-
You guys are terrific thanks. I've disabled software rsc on the server 2022 hyperv 2.7 development build and my throughput is back where it should be! As noted in the article I used set-vmswitch -enablesoftwarersc $false to get this done. Much appreciated!
-
@ksdehoff So you are talking about the command in windows, but this shouldn't be needed anymore if it is set in FreeBSD, as I understand it. Also it isn't persistent.
@stephenw10 Is this sysctl. in the pfSense GUI in System-Advanced-System Tunables presant? I don't see it in 22.05.
-
@bob-dig yes this disables software receive coalesce segment coalescing - a new feature that was added in 2019. Not sure why the recent patch changed pfsense' behavior but this setting disables rsc on the virtual switch. It worked well for me - I set it only on my lan virtual switch so the wan virtual switch still has it enabled.
-
@bob-dig Well I've hit another oddity. I used the command set-vmswitch -name * -enablesoftwarersc $false. Then I tested my throughput at speedtest.xfinity.com and got a very good number.
Then an hour later I noticed slowness, ran speedtest again, was getting terrible throughput (1 mbit/sec) and so reran the disable rsc command - and throughput is back up. The pfsense vm has been up the entire time - so is something in the os toggling software rsc? Maybe thats what happened in this latest patch.
-
@ksdehoff I have no problems here on latest Server 2022 with all regular updates, it is a FreeBSD problem in a Hyper-V vm, not a Windows or Hyper-V problem.
-
@bob-dig I wonder if it has to do with my lan being a server embedded team, a dual intel x710sfp network card.
-
@ksdehoff Try what Stephen has said, it should solve the problem but don't ask me how to do it.
-
@bob-dig ok I get it - I had disabled rsc at the hyperv level but stephen is talking about doing it just for the freebsd VM. I've made the change and it looks good but time will tell
The freebsd command is sysctl dev.hn.1.rsc_switch=0/1 - (hn1 is my wan in this case) and 1 appears to disable rsc handling in the vm (the boot value is 0).
In more weirdness rsc_switch is 0 when pfsense boots but I get poor performance. Changing it to 1 does nothing, but changing it BACK to 0 clears the issue. Very bizarre behavior. Also, the rsc errors (sysctl -a | grep rsc) accumulate only on the wan interface, not the lan side
-
@ksdehoff I guess I have to try the beta shortly.
Edit: no Speed problems there:
23.01-BETA (amd64)
built on Sat Dec 17 14:33:51 UTC 2022
FreeBSD 14.0-CURRENT -
@bob-dig said in Performance on Hyper-V broken by Microsoft Dec 13 patches?:
@stephenw10 Is this sysctl. in the pfSense GUI in System-Advanced-System Tunables presant? I don't see it in 22.05.
Not by default but you could add it.
It shouldn't be needed because the default values there should be to disable the RSC behaviour. It doesn't appear to be here for some reason.
Steve
-
S SteveITS referenced this topic on Dec 19, 2022, 4:42 PM