Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

    Scheduled Pinned Locked Moved Virtualization
    187 Posts 36 Posters 105.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H
      hendryjl @m0nji
      last edited by

      @m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

      EDIT: i think i will test tomorrow FreeBSD 12.2 (pfsense 2.5.x) and will see if RSC makes a difference there

      I still have my image of 2.5.2 before I applied 2.6. I reverted to it when I first had my issues (before I ever tried disabling RSC on the Hyper-V virtual switches). I didn't see any issues. Later tonight I will revert to it again and see what the sysctls say.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        RSC support wasn't added to hn(4) until 12.3 so I would expect to see no sysctls there.

        The driver looks to try to use them based on the VNIC revision. I'm speculating that under some set of circumstances the driver/host/platform maybe trying and failing to use RSC even when it's disabled in the vswitch.

        I'd like to check the sysctl and verbose boot logs from someone who is hitting this still with rsc seemingly disabled. Since we know they should be all zeros if it's actually disabled.

        Steve

        Bob.DigB 1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator @m0nji
          last edited by stephenw10

          @m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

          on FreeBSD 13.0, RSC was removed completly from the kernel/system, thats why the sysctl's doesn't exist and so we do not have these problems?

          Ah, nope! It's because FreeBSD 13 is actually older than 12.3 and doesn't have any of the RSC support patches:
          https://github.com/freebsd/freebsd-src/tree/releng/13.0/sys/dev/hyperv/netvsc

          I would expect an 13-stable snapshot to fail the same way.

          Steve

          1 Reply Last reply Reply Quote 1
          • Bob.DigB
            Bob.Dig LAYER 8 @stephenw10
            last edited by Bob.Dig

            @stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

            I'd like to check the sysctl and verbose boot logs from someone who is hitting this still with rsc seemingly disabled. Since we know they should be all zeros if it's actually disabled.

            Steve

            I would like to but I need description for dummies how to do it.

            m0njiM 1 Reply Last reply Reply Quote 0
            • m0njiM
              m0nji @Bob.Dig
              last edited by m0nji

              @bob-dig just open the VM preview window, reboot and on the boot screen hit the "boot options" and select verbose boot

              63e6e7d6-391f-42f5-b3ba-d3c8cde56986-image.png

              FreeBSD 12.3

              root@freebsd123:~ # grep -i "rsc" /var/log/messages
              Feb 25 08:48:02 freebsd123 kernel: hn0: hwcaps rsc: ip4 1 ip6 1
              Feb 25 08:48:02 freebsd123 kernel: hn0: offload rsc: ip4 2, ip6 2
              Feb 25 08:48:02 freebsd123 kernel: hn1: hwcaps rsc: ip4 1 ip6 1
              Feb 25 08:48:02 freebsd123 kernel: hn1: offload rsc: ip4 2, ip6 2
              Feb 25 08:48:02 freebsd123 kernel: hn2: hwcaps rsc: ip4 1 ip6 1
              Feb 25 08:48:02 freebsd123 kernel: hn2: offload rsc: ip4 2, ip6 2
              
              

              pfSense 2.6.0

              [2.6.0-RELEASE][admin@pfSense.home.arpa]/root: grep -i "rsc" /var/log/system.log                
              Feb 25 07:56:22 pfSense kernel: hn0: hwcaps rsc: ip4 1 ip6 1
              Feb 25 07:56:22 pfSense kernel: hn0: offload rsc: ip4 2, ip6 2
              Feb 25 07:56:22 pfSense kernel: hn1: hwcaps rsc: ip4 1 ip6 1
              Feb 25 07:56:22 pfSense kernel: hn1: offload rsc: ip4 2, ip6 2
              Feb 25 07:56:22 pfSense kernel: hn2: hwcaps rsc: ip4 1 ip6 1
              Feb 25 07:56:22 pfSense kernel: hn2: offload rsc: ip4 2, ip6 2
              
              

              Intel i3-N305 / 4 x 2.5Gbe LAN @2.7.2-Release
              WAN: Vodafone 1000/50, Telekom 250/40; Switch: USW Enterprise 8 PoE, USW Flex XG, US-8-60W; Wifi: Unifi 6 Lite AP, U6 Mesh

              Bob.DigB 1 Reply Last reply Reply Quote 1
              • Bob.DigB
                Bob.Dig LAYER 8 @m0nji
                last edited by Bob.Dig

                @m0nji I tried this with 2.5.2. I don't see anything regarding rsc in the logs or OS Boot log.
                Also where to put those commands in? If I try with putty, it gives my permission denied. In the WebUI there is no output as well.

                So I guess not noob friendly enough. 😓

                
                Enter an option: 8
                
                [2.5.2-RELEASE][admin@pfSense.home.arpa]/root: # grep -i "rsc" /var/log/messages                                      
                #: Command not found.
                [2.5.2-RELEASE][admin@pfSense.home.arpa]/root: grep -i "rsc" /var/log/system.log                                      
                [2.5.2-RELEASE][admin@pfSense.home.arpa]/root: ~ # grep -i "rsc" /var/log/messages
                /root: Permission denied.
                [2.5.2-RELEASE][admin@pfSense.home.arpa]/root:
                
                
                m0njiM 1 Reply Last reply Reply Quote 0
                • m0njiM
                  m0nji @Bob.Dig
                  last edited by

                  @bob-dig i think your output is correct. you should not see anything RSC related on pfSense 2.5.x

                  This is what @stephenw10 said:
                  "RSC support wasn't added to hn(4) until 12.3 so I would expect to see no sysctls there."

                  Intel i3-N305 / 4 x 2.5Gbe LAN @2.7.2-Release
                  WAN: Vodafone 1000/50, Telekom 250/40; Switch: USW Enterprise 8 PoE, USW Flex XG, US-8-60W; Wifi: Unifi 6 Lite AP, U6 Mesh

                  Bob.DigB 1 Reply Last reply Reply Quote 1
                  • Bob.DigB
                    Bob.Dig LAYER 8 @m0nji
                    last edited by Bob.Dig

                    This post is deleted!
                    m0njiM 1 Reply Last reply Reply Quote 0
                    • m0njiM
                      m0nji @Bob.Dig
                      last edited by m0nji

                      @bob-dig you missed a "."
                      2a56818d-af59-4334-a94b-62baab96adfe-image.png

                      hn.0 is probably the wan interface. you should also try hn.1, hn.2 and hn.3

                      EDIT: you already fixed your post ;) try it also for hn1, hn2 and hn3. it depends which hn adapter you use

                      EDIT2: if i understood it correctly, then none hn adapter should give you a value >0 if RSC is disabled. if it tells you a value >0 then the setting RSC disabled does not realy work

                      Intel i3-N305 / 4 x 2.5Gbe LAN @2.7.2-Release
                      WAN: Vodafone 1000/50, Telekom 250/40; Switch: USW Enterprise 8 PoE, USW Flex XG, US-8-60W; Wifi: Unifi 6 Lite AP, U6 Mesh

                      Bob.DigB 1 Reply Last reply Reply Quote 1
                      • Bob.DigB
                        Bob.Dig LAYER 8 @m0nji
                        last edited by Bob.Dig

                        2.6:

                        SoftwareRscEnabled RscOffloadEnabled
                        ------------------ -----------------
                                     False             False
                                     False             False
                                     False             False
                        
                        grep -i "rsc" /var/log/system.log
                        
                        Feb 25 13:18:24 pfSense kernel: hn0: hwcaps rsc: ip4 1 ip6 1
                        Feb 25 13:18:24 pfSense kernel: hn0: offload rsc: ip4 2, ip6 2
                        Feb 25 13:18:24 pfSense kernel: hn1: hwcaps rsc: ip4 1 ip6 1
                        Feb 25 13:18:24 pfSense kernel: hn1: offload rsc: ip4 2, ip6 2
                        Feb 25 13:18:24 pfSense kernel: hn2: hwcaps rsc: ip4 1 ip6 1
                        Feb 25 13:18:24 pfSense kernel: hn2: offload rsc: ip4 2, ip6 2
                        

                        While "copying":

                        sysctl dev.hn.1.rx.0.rsc_drop
                        sysctl dev.hn.1.rx.0.rsc_pkts
                        sysctl dev.hn.2.rx.0.rsc_drop
                        sysctl dev.hn.2.rx.0.rsc_pkts
                        
                        [2.6.0-RELEASE][root@pfSense.home.arpa]/root: sysctl dev.hn.1.rx.0.rsc_drop
                        dev.hn.1.rx.0.rsc_drop: 0
                        [2.6.0-RELEASE][root@pfSense.home.arpa]/root: sysctl dev.hn.1.rx.0.rsc_pkts
                        dev.hn.1.rx.0.rsc_pkts: 0
                        [2.6.0-RELEASE][root@pfSense.home.arpa]/root: sysctl dev.hn.2.rx.0.rsc_drop
                        dev.hn.2.rx.0.rsc_drop: 0
                        [2.6.0-RELEASE][root@pfSense.home.arpa]/root: sysctl dev.hn.2.rx.0.rsc_pkts
                        dev.hn.2.rx.0.rsc_pkts: 107
                        

                        The last number constantly rises.

                        Everything stays the same for me if I re-enable rsc in Windows, no difference at all.

                        SoftwareRscEnabled RscOffloadEnabled
                        ------------------ -----------------
                                      True             False
                                      True             False
                                      True             False
                        
                        1 Reply Last reply Reply Quote 1
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Hmm, so just to be clear you are seeing the RSC packets counter increment whether or not you have disabled RSC on the vswitch that interface is connected to?

                          That seems like a different result to those for whom disabling RSC solved the issue. And seems to support my conjecture... 🤔

                          Unclear what we can do about it though if that is the case. Yet.

                          Steve

                          Bob.DigB 1 Reply Last reply Reply Quote 0
                          • Bob.DigB
                            Bob.Dig LAYER 8 @stephenw10
                            last edited by

                            @stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

                            Hmm, so just to be clear you are seeing the RSC packets counter increment whether or not you have disabled RSC on the vswitch that interface is connected to?

                            True

                            Unclear what we can do about it though if that is the case. Yet.

                            I hope you guys figure it out.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Ok, so we need more data points here to be sure it's actually what's happening.

                              But assuming that's true it appears:
                              There's an issue with the RSC code added in FreeBSD.
                              In some situations the vswitches in hyper-v do not respect the disable RSC setting.

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Ok, so it looks like our European friends in fact already hit this because they are actually building on 13-stable and came to the same conclusions. I have opened a bug report: https://redmine.pfsense.org/issues/12873

                                Steve

                                1 Reply Last reply Reply Quote 1
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  What do you guys see for these sysctls?:

                                  dev.hn.0.hwassist: 607<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP6_UDP,CSUM_IP6_TCP>
                                  dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                  dev.hn.0.ndis_version: 6.30
                                  dev.hn.0.nvs_version: 393217
                                  

                                  Please report whether or not you're hitting the issue with the values shown.

                                  Bob.DigB m0njiM 2 Replies Last reply Reply Quote 0
                                  • Bob.DigB
                                    Bob.Dig LAYER 8 @stephenw10
                                    last edited by Bob.Dig

                                    @stephenw10 Hittin it hard

                                    sysctl dev.hn.0.hwassist
                                    sysctl dev.hn.0.caps
                                    sysctl dev.hn.0.ndis_version
                                    sysctl dev.hn.0.nvs_version
                                    
                                    2.5.2
                                    dev.hn.0.hwassist: 1617<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO,CSUM_IP6_UDP,CSUM_IP6_TCP,CSUM_IP6_TSO>
                                    dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                    dev.hn.0.ndis_version: 6.30
                                    dev.hn.0.nvs_version: 327680
                                    
                                    2.6.0
                                    dev.hn.0.hwassist: 607<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP6_UDP,CSUM_IP6_TCP>
                                    dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                    dev.hn.0.ndis_version: 6.30
                                    dev.hn.0.nvs_version: 393217
                                    
                                    

                                    Although this time I haven't deactivated RSC in Windows Host.

                                    1 Reply Last reply Reply Quote 1
                                    • m0njiM
                                      m0nji @stephenw10
                                      last edited by

                                      @stephenw10

                                      RSC disabled on Hyper-V Host

                                      PS C:\Users\m0nji\Downloads\iperf-3.1.3-win64\iperf-3.1.3-win64> .\iperf3.exe -c 192.168.187.11 -R
                                      Connecting to host 192.168.187.11, port 5201
                                      Reverse mode, remote host 192.168.187.11 is sending
                                      [  4] local 192.168.189.10 port 49995 connected to 192.168.187.11 port 5201
                                      [ ID] Interval           Transfer     Bandwidth
                                      [  4]   0.00-1.00   sec  55.6 KBytes   454 Kbits/sec
                                      [  4]   1.00-2.01   sec  21.4 KBytes   174 Kbits/sec
                                      [  4]   2.01-3.00   sec  21.4 KBytes   176 Kbits/sec
                                      [  4]   3.00-4.00   sec  21.4 KBytes   175 Kbits/sec
                                      [  4]   4.00-5.00   sec  17.1 KBytes   140 Kbits/sec
                                      [  4]   5.00-6.00   sec  21.4 KBytes   175 Kbits/sec
                                      [  4]   6.00-7.00   sec  21.4 KBytes   175 Kbits/sec
                                      [  4]   7.00-8.01   sec  15.7 KBytes   128 Kbits/sec
                                      [  4]   8.01-9.00   sec  20.0 KBytes   165 Kbits/sec
                                      [  4]   9.00-10.00  sec  21.4 KBytes   175 Kbits/sec
                                      - - - - - - - - - - - - - - - - - - - - - - - - -
                                      [ ID] Interval           Transfer     Bandwidth
                                      [  4]   0.00-10.00  sec   384 KBytes   314 Kbits/sec                  sender
                                      [  4]   0.00-10.00  sec   237 KBytes   194 Kbits/sec                  receiver
                                      
                                      iperf Done.
                                      

                                      pfSense 2.6.0 (hitting the issue)

                                      dev.hn.0.hwassist: 607<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP6_UDP,CSUM_IP6_TCP>
                                      dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.0.ndis_version: 6.30
                                      dev.hn.0.nvs_version: 393217
                                      dev.hn.1.hwassist: 607<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP6_UDP,CSUM_IP6_TCP>
                                      dev.hn.1.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.1.ndis_version: 6.30
                                      dev.hn.1.nvs_version: 393217
                                      dev.hn.2.hwassist: 607<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP6_UDP,CSUM_IP6_TCP>
                                      dev.hn.2.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.2.ndis_version: 6.30
                                      dev.hn.2.nvs_version: 393217
                                      

                                      for comparision FreeBSD 12.3 (hitting the issue)

                                      dev.hn.0.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.0.ndis_version: 6.30
                                      dev.hn.0.nvs_version: 393217
                                      dev.hn.1.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.1.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.1.ndis_version: 6.30
                                      dev.hn.1.nvs_version: 393217
                                      dev.hn.2.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.2.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.2.ndis_version: 6.30
                                      dev.hn.2.nvs_version: 393217
                                      

                                      FreeBSD 13.0 (not hitting the issue but obviously not STABLE version)

                                      dev.hn.0.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.0.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.0.ndis_version: 6.30
                                      dev.hn.0.nvs_version: 327680
                                      dev.hn.1.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.1.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.1.ndis_version: 6.30
                                      dev.hn.1.nvs_version: 327680
                                      dev.hn.2.hwassist: 17<CSUM_IP,CSUM_IP_UDP,CSUM_IP_TCP,CSUM_IP_TSO>
                                      dev.hn.2.caps: 7ff<VLAN,MTU,IPCS,TCP4CS,TCP6CS,UDP4CS,UDP6CS,TSO4,TSO6,HASHVAL,UDPHASH>
                                      dev.hn.2.ndis_version: 6.30
                                      dev.hn.2.nvs_version: 327680
                                      

                                      Intel i3-N305 / 4 x 2.5Gbe LAN @2.7.2-Release
                                      WAN: Vodafone 1000/50, Telekom 250/40; Switch: USW Enterprise 8 PoE, USW Flex XG, US-8-60W; Wifi: Unifi 6 Lite AP, U6 Mesh

                                      1 Reply Last reply Reply Quote 1
                                      • Bob.DigB
                                        Bob.Dig LAYER 8
                                        last edited by Bob.Dig

                                        I figured out an interim solution for me. I created two external Switches, one for pfSense and one for all the other VMs. With that it does work, no slow speed anymore. Drawback is, it is using one more port and everything goes through a physical Switch.

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          So that's no routing between VMs in the same host? That seems like it's what should trigger this.

                                          Bob.DigB 1 Reply Last reply Reply Quote 0
                                          • Bob.DigB
                                            Bob.Dig LAYER 8 @stephenw10
                                            last edited by Bob.Dig

                                            @stephenw10 I think the key point is that pfsense is not using the same vSwitch then the others. This is an untypical setup and no one with a right mind would do it like this, but I did. And it does work here. I think I simulated "having two (vm) hosts", where it is natural, that the VMs can't use the same vSwitch.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.