[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

i386DX

@m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

one important side note: right now, you have to enable & disable RSC on the VM after every VM reboot even when the value is still $false!
at least on my host...

Yes, same behaviour on my host too (Windows server 2022, gen2 vm, I350-T2 adapter, sr-iov unavailable, vmq disabled).
In my case I could even leave -SoftwareRscEnabled enabled on the vswitches and just flip -RscEnabled on and off on the vm network adapter(s) to restore normal bandwidth.

Unfortunately I'm not aware if there are ways to enforce nvs version 5 (327680) on the hn driver.

stephenw10

Indeed I was hoping there would be some way to force it but I don't see any way.

m0nji

@stephenw10
we should keep an eye on this: https://reviews.freebsd.org/D29075?id=85183

Drumdevil

I have two Server 2019 hosts (Host1 & Host2) in my homelab, with both the same hardware. Both experienced slow WAN speeds since the update to 2.6. Normally WAN would reach about 1 Gbit, but after the update it was only about 30 Mbit. The servers each have two separate virtual switches for LAN and WAN. Inter-VLAN communication was as expected in 2.6.

The virtual switches are attached as follows:

WAN vSwitch -> Intel Pro/1000 PT, not shared with host
LAN vSwitch -> Microsoft Network Adapter Multiplexer Driver -> NIC Team, static teaming, dynamic load balance

On both hosts I did:

Get-VMSwitch | Set-VMSwitch -SoftwareRscEnabled $false

Then I confirmed both servers returned False for both switches using:

Get-VMSwitch | Select-Object *RSC*

This fixed the issue on Host1. On Host2, now the WAN speeds were OK too, BUT: Inter-VRF traffic became insanely slow. After troubleshooting I found that the only difference was that Host2 was more recently installed and had yet to get a bunch of Windows updates. I couldn't tell you which one it was, but the updates fixed it.

So if you have inter-VLAN performance issues after disabling RSC, try installing Windows updates.

fiblimitless

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

m0nji

@fiblimitless said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

Cannot confirm this. I still need to enable & disable RSC on the VM after every VM Reboot!
Just for Clarification: I disabled Large Send Offload on the virtual Switch and the physical NIC. But i didnt reboot the host yet.

Bob.Dig

@m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

Just for Clarification: I disabled Large Send Offload on the virtual Switch and the physical NIC. But i didnt reboot the host yet.

I think he meant the NIC in the vms, pretty easy to do that for a windows vm, more hard on a linux machine without GUI I guess. That's why I have not tested this.

I have my wacky solution for now using two external switches and wait for a permanent solution, probably coming with 2.7... lets hope.

bimmerdriver

@fiblimitless said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

I tried your suggestion, but it made no difference. I'm experiencing erratic dead slow uploads and downloads. Is it necessary to reboot after making this change?

I'm running pfSense 2.6.0 on Windows Server 2019 in a Generation 2 VM. There are virtual switches on the WAN and LAN. The WAN and LAN NICs are Intel I350. Not using VLANs. VMQ, IPsec task offloading, SR-IOV enabled.

fiblimitless

@bimmerdriver
I have same system as you except for SR-IOV I have it disabled.

For the download as mentioned before disable RSC via powershell.
For the uploads: Make sure you disable Large Send Offload v2 on the
virtual Switch and the physical NIC on the Host Machine.

No boot needed, maybe disable/enable NICs.

If that doesn't work, start disabling all these offload options you'll find in the v/NICs properties. and see if the speed goes back to normal. then work your way through by enabling them one by one. until you find the cause.

I'v had a previous experience with win2016 when it was in preview, with regular Windows VM on Hyper-V. The speed was so slow, and had this offloading disabled, until Microsoft pushed an update and fixed it, or it was a NIC driver update. I don't remember.

.

i386DX

@bimmerdriver did you try flipping -RscEnabled on and off on your pfsense vm network adapters?
This fixes both wan upload and inter vlan bandwidth for me on ws 2022.

I ended up scheduling this at startup until a fix is available:

Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

Bob.Dig

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I ended up scheduling this at startup until a fix is available:

Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

~~This works great~~, or not I am getting mixed results here, but how do you start that with Windows exactly?

bimmerdriver

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@bimmerdriver did you try flipping -RscEnabled on and off on your pfsense vm network adapters?
This fixes both wan upload and inter vlan bandwidth for me on ws 2022.

I ended up scheduling this at startup until a fix is available:
Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

I did this yesterday. I didn't have a chance to post an update. It seems to have fixed the problem. I also noticed in the advanced settings for the virtual ethernet adapter that there are settings to enable and disable RSC for Ipv4 and IPv6. I will enable RSC on the switches and try disabling it in the driver to see if it accomplishes the same thing or not.

bimmerdriver

Here are the settings I'm using for Windows Server 2019 with a Generation 2 VM. I'm not suggesting these settings are optimal for everyone, but in my case, pfSense is working properly with them, unlike before I disabled RSC in the virtual switches.

My LAN and WAN NICs are I350-T2.

I'm not using VLANs.

In my system, the LAN and WAN NICs are connected to virtual switches. pfSense NICs are virtual, connected to LAN and WAN virtual switches.

Virtual switches: RSC disabled, SR-IOV enabled in external switches.

In all NICs (physical and virtual), VMQ, IPsec task offloading and SR-IOV are enabled. The only advanced feature I have enabled is port mirroring on WAN NICs so I can capture packets.

NIC Drivers: Jumbo packets (9014) are enabled. All possible offloading is enabled. IRRC, the only non-default setting is enabling jumbo packets.

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

I have not tried disabling RSC (IPv4 and IPv6) in the virtual NIC driver settings. Perhaps that would have the same effect as disabling RSC in the virtual switches.

YMMV.

i386DX

@bob-dig said in [After Upgrade inter (V)LAN communication is very slow (on Hyper-V).]

but how do you start that with Windows exactly?

A scheduled task that executes powershell.exe C:\path\to\script.ps1 as SYSTEM on startup works fine.
It should also be possible to trigger the task on vm restart by creating a custom eventlog query (look for event 18601 on Microsoft-Windows-Hyper-V-Worker/Admin).

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I'm not using VLANs.

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough. My upload was fine too after disabling it, but in my case it made more sense to operate on -RscEnabled on both vmnetworkadapters since I was going to enable/disable it anyway on the lan in order to restore inter vlan bandwidth.

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

Same for me, except ALTQ support which is enabled to allow FQ_CODEL usage.

Bob.Dig

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough.

I think on ws2019 and earlier it might be enough but not on ws2022.

inmarket

I have done some testing that might shed some light on this (in some circumstances). For me turning off RSC on the virtual switch solved the problem. This problem also occurs with raw FreeBSD 13 STABLE.

With RSC turned on I can successfully do...

ping -s 1400 host

(where the host is either on the same virtual switch or via a hardware port) PROVIDED THE PORT IS ON A LOCAL SEGMENT.
However,

ping -s 1400 www.google.com

comes back with...

76 bytes from XX.XXX.XX.XX ....
wrong total length 96 instead of 1428

Note: this is on the same virtual adapter, the same virtual switch, the same network port as the successful operation.

From this I conclude that the new RSC capabilities are probably broken for packets that are fragmented. The problem results in the packet received truncated to 96 bytes!! This occurs regardless of TCP/UDP/ICMP.

stephenw10

The patch for this that disables RSC by default and adds a sysctl to enable it should now be in new 2.7 snapshots:
https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Testing and feedback required there from anyone who can.

Steve

murdof

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Had tried all workarounds - nothing had worked for me.
Windows Server 2022 (AMD EPYC)
pfsense 2.6.0 (no issues with 2.5.2)

I upgraded from within the web ui to latest build (2.7.0.a.20220513.0600) and now speeds are back to normal.
While I was getting for my IPSec traffic about 350kb/sec now I'm getting 123Mbits/sec

Thanks for the fix!
Any chance when we should expect 2.7.0 to come out soon, or even maybe this fix to be backport to a 2.6.1 version?

Bob.Dig

I also can confirm, problem is gone!

22.05-BETA (amd64)
built on Fri May 13 06:28:24 UTC 2022
FreeBSD 12.3-STABLE

stephenw10

@murdof said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

maybe this fix to be backport to a 2.6.1 version?

We may be able to include it in a point release if one happened but that would only happen if a security issue required it. I'm not aware of one currently.

Great news that it fixed it as expected though.

Steve