Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Performance on Hyper-V broken by Microsoft Dec 13 patches?

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 3 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      ksdehoff
      last edited by

      I've got a Hyper-V system that's been working well under 2012r2. I had started upgrading to Server 2022 and initial tests worked fine. But on Dec 13 M$ patch set including KB5021249 appears to throttle somehow throughput down to 1-3 mbit/sec. Has anyone else seen this and figured out a workaround?

      M$ has flagged vm performance issues with this patch in some circumstances. https://support.microsoft.com/en-us/topic/december-13-2022-kb5021249-os-build-20348-1366-d5fe7608-bc9d-4055-a88c-fb2fd3d5fd45

      Bob.DigB 1 Reply Last reply Reply Quote 0
      • Bob.DigB
        Bob.Dig LAYER 8 @ksdehoff
        last edited by Bob.Dig

        @ksdehoff said in Performance on Hyper-V broken by Microsoft Dec 13 patches?:

        Has anyone else seen this ?

        No. But your problems might be related to this.

        1 Reply Last reply Reply Quote 0
        • K
          ksdehoff
          last edited by

          I checked that thread - I have all the pfsense hardware acceleration features disabled: Offloading, alt-q. Still sucks and I actually noted a 'huge' decrease in throughput doing this: from 1 mbit/sec to 0.3 mbit/sec!

          Bob.DigB 1 Reply Last reply Reply Quote 0
          • Bob.DigB
            Bob.Dig LAYER 8 @ksdehoff
            last edited by Bob.Dig

            @ksdehoff Are you using 2.6? Then you have that problem. There is no setting inside of pfSense which would help against that and also none in the Windows GUI.

            K 1 Reply Last reply Reply Quote 0
            • K
              ksdehoff @Bob.Dig
              last edited by

              @bob-dig Yes 2.6 stable. I just put a 2.7 development version together from last night and it works! I have to decide on sticking with 2.6 stable on a 2012r2 temporary platform or 2.7 development on my production hyperv platform….. Are development builds typically pretty robust for simple traffic?

              Bob.DigB 1 Reply Last reply Reply Quote 0
              • Bob.DigB
                Bob.Dig LAYER 8 @ksdehoff
                last edited by

                @ksdehoff I am using the plus version. If you have to use the development version, don't update it regularly and have a snapshot ready anytime you do update to roll back... it is risky.

                K 1 Reply Last reply Reply Quote 1
                • K
                  ksdehoff @Bob.Dig
                  last edited by

                  Well this is annoying. My 'path to production' was a 2.7 build on hyperv 2012r2 to confirm a baseline that works. I then migrated it to hyperv 2022 to confirm it worked still with the same image - which it did. But I did not reboot the OS once I migrated it and once I rebooted on 2022 I'm back to the 1 mbit/sec terrible throughput

                  Bob.DigB 1 Reply Last reply Reply Quote 0
                  • Bob.DigB
                    Bob.Dig LAYER 8 @ksdehoff
                    last edited by Bob.Dig

                    @ksdehoff That sounds horrible, to me. I hope netgate has fixed it, hadn't any time to test it myself. Maybe I will test the new beta version of plus but I don't know how this compares to 2.7.

                    1 Reply Last reply Reply Quote 1
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Check the RSC sysctl. It should default to off but it's still possible to enable it and hit this issue.

                      It is in 2.7: https://github.com/pfsense/FreeBSD-src/commit/80c3eb7bc64b745b2a0bda953e29a26e32dff9d7

                      Steve

                      1 Reply Last reply Reply Quote 1
                      • K
                        ksdehoff
                        last edited by

                        You guys are terrific thanks. I've disabled software rsc on the server 2022 hyperv 2.7 development build and my throughput is back where it should be! As noted in the article I used set-vmswitch -enablesoftwarersc $false to get this done. Much appreciated!

                        Bob.DigB 1 Reply Last reply Reply Quote 0
                        • Bob.DigB
                          Bob.Dig LAYER 8 @ksdehoff
                          last edited by Bob.Dig

                          @ksdehoff So you are talking about the command in windows, but this shouldn't be needed anymore if it is set in FreeBSD, as I understand it. Also it isn't persistent.

                          @stephenw10 Is this sysctl. in the pfSense GUI in System-Advanced-System Tunables presant? I don't see it in 22.05.

                          K stephenw10S 2 Replies Last reply Reply Quote 0
                          • K
                            ksdehoff @Bob.Dig
                            last edited by

                            @bob-dig yes this disables software receive coalesce segment coalescing - a new feature that was added in 2019. Not sure why the recent patch changed pfsense' behavior but this setting disables rsc on the virtual switch. It worked well for me - I set it only on my lan virtual switch so the wan virtual switch still has it enabled.

                            K 1 Reply Last reply Reply Quote 0
                            • K
                              ksdehoff @ksdehoff
                              last edited by

                              @bob-dig Well I've hit another oddity. I used the command set-vmswitch -name * -enablesoftwarersc $false. Then I tested my throughput at speedtest.xfinity.com and got a very good number.

                              Then an hour later I noticed slowness, ran speedtest again, was getting terrible throughput (1 mbit/sec) and so reran the disable rsc command - and throughput is back up. The pfsense vm has been up the entire time - so is something in the os toggling software rsc? Maybe thats what happened in this latest patch.

                              Bob.DigB 1 Reply Last reply Reply Quote 0
                              • Bob.DigB
                                Bob.Dig LAYER 8 @ksdehoff
                                last edited by

                                @ksdehoff I have no problems here on latest Server 2022 with all regular updates, it is a FreeBSD problem in a Hyper-V vm, not a Windows or Hyper-V problem.

                                K 1 Reply Last reply Reply Quote 0
                                • K
                                  ksdehoff @Bob.Dig
                                  last edited by

                                  @bob-dig I wonder if it has to do with my lan being a server embedded team, a dual intel x710sfp network card.

                                  Bob.DigB 1 Reply Last reply Reply Quote 0
                                  • Bob.DigB
                                    Bob.Dig LAYER 8 @ksdehoff
                                    last edited by

                                    @ksdehoff Try what Stephen has said, it should solve the problem but don't ask me how to do it. 😁

                                    K 1 Reply Last reply Reply Quote 0
                                    • K
                                      ksdehoff @Bob.Dig
                                      last edited by ksdehoff

                                      @bob-dig ok I get it - I had disabled rsc at the hyperv level but stephen is talking about doing it just for the freebsd VM. I've made the change and it looks good but time will tell

                                      The freebsd command is sysctl dev.hn.1.rsc_switch=0/1 - (hn1 is my wan in this case) and 1 appears to disable rsc handling in the vm (the boot value is 0).

                                      In more weirdness rsc_switch is 0 when pfsense boots but I get poor performance. Changing it to 1 does nothing, but changing it BACK to 0 clears the issue. Very bizarre behavior. Also, the rsc errors (sysctl -a | grep rsc) accumulate only on the wan interface, not the lan side

                                      Bob.DigB 1 Reply Last reply Reply Quote 1
                                      • Bob.DigB
                                        Bob.Dig LAYER 8 @ksdehoff
                                        last edited by Bob.Dig

                                        @ksdehoff I guess I have to try the beta shortly.
                                        Edit: no Speed problems there:
                                        23.01-BETA (amd64)
                                        built on Sat Dec 17 14:33:51 UTC 2022
                                        FreeBSD 14.0-CURRENT

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator @Bob.Dig
                                          last edited by

                                          @bob-dig said in Performance on Hyper-V broken by Microsoft Dec 13 patches?:

                                          @stephenw10 Is this sysctl. in the pfSense GUI in System-Advanced-System Tunables presant? I don't see it in 22.05.

                                          Not by default but you could add it.

                                          It shouldn't be needed because the default values there should be to disable the RSC behaviour. It doesn't appear to be here for some reason.

                                          Steve

                                          1 Reply Last reply Reply Quote 1
                                          • S SteveITS referenced this topic on
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.