[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

cannyit

@m0nji WAN speed or better speed between all interfaces is fine again after disable RSC. I didnt had an issue with inter VLAN routing. I have vlans tagged within pfsense but speed is OK.

stephenw10

Is the VLAN issue only happening on gen 1 VMs?
That was an early theory but I've not seen definite conclusion.

Bob.Dig

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

Is the VLAN issue only happening on gen 1 VMs?
That was an early theory but I've not seen definite conclusion.

It is not VLAN related but related to hn interfaces and or virtual switches I would say.
Just for the record:

First who encountered it
Server 2022 Gen2 VM
AMD System
WAN-Speed never was affected
Even just using virtual adapters on private switch without VLAN it happened
Most probably a bug in FreeBSD 12.3 regarding hyper-v

stephenw10

Mmm, I'd bet it's going to be one of these changes from Mar 2021:
https://github.com/pfsense/FreeBSD-src/commits/RELENG_2_6_0/sys/dev/hyperv/netvsc

And given that one of those enabled RSC support....

hendryjl

+1 for Disabling RSC. My setup...

Home "power user"
Fitlet2 with three network interfaces
Windows Server 2019
Hyper-V Gen 1 VM.
** Three virtual switches (one for each ISP, one for switch)
** 7 network interfaces at VM level (one for each ISP, 5 for 5 VLANs, set with VLAN tagging at virtual NIC level in HyperV)
** All virtual nics with every check box disabled except VLAN (no VMQ, etc)

WAN speed was terrible. Changed software receive side coalescing on the VM switches, and all good.

Commands used:

Check:
Get-VMSwitch | % { $_ | Select-Object *RSC* }

Change:
Get-VMSwitch | % { $_ | Set-VMSwitch -EnableSoftwareRsc $false }

Re-Check:
Get-VMSwitch | % { $_ | Select-Object *RSC* }

Bob.Dig

Yup, that is the main difference, it is not about WAN-Speed in my and some few others cases.
If WAN then RSC.

I had disabled RSC today a second time, even rebooted the host, it again didn't helped in my case.

stephenw10

Looking at that commit there are some things adds we should be able to check. Some sysctls:

dev.hn.0.rx.0.rsc_drop: 0
dev.hn.0.rx.0.rsc_pkts: 0

And if you boot verbose the logs show:

Feb 24 18:51:42 	kernel 		hn0: hwcaps rsc: ip4 1 ip6 1
Feb 24 18:51:42 	kernel 		hn0: offload rsc: ip4 2, ip6 2

So I wonder if this is somehow not being disabled because it still seems to fit:
It only affects TCP.
It only affects traffic between VMs in the same host.

Steve

hendryjl

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

It only affects TCP.

Confirmed for me. UDP streams were fine. RSC affects only with TCP packets (by definition).

It only affects traffic between VMs in the same host.

In my case, because my traffic comes in on other physcial nics, and those physical nics tie to a physical nic with virtual nics (vlan tagged by hyper-v) and the clients live on those VLANs, it definitely affects traffic outside of the host.

stephenw10

Be good to get those values from an affected VM. I grabbed those from Azure which isn't.

Steve

hendryjl

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

Be good to get those values from an affected VM

I switched RSC back on, booted the VM back up, and verified the problem existed. Here are the ctls (attached).sysctls.txt

I booted verbose, but missed the output. I couldn't find it in the logs. Am I going to have to do a serial console to see them?

stephenw10

Nice, so you see traffic in those counters with RSC enabled in the vswitches when the problem exists. Do you stil see data there when RSC is disabled in the switches?

It would be interesting to see if that varies for VMs that are still hitting issues even with RSC disabled.

Steve

ttmcmurry

I still think there's more than one issue. My post several days ago is with a pfSense on hyper-v, gen 2 vm, with no vlans defined in pfSense, vnic, or vswitch, and RSC is disabled. And I had (well, have if I turn the pfSense 2.6 vm back on) the slow performance issue.

Perhaps for hyper-v configuration with native interfaces throughout, RSC is not a factor. Perhaps at best a clue.

For those using hyper-v with vlans and disabling RSC is the fix, that's great, and also a clue.

My mind keeps going back to the troubleshooting where migrating the pfSense VM to another host fixing slow network performance. I'm beginning to think the issue is how FreeBSD is interfacing with the hn driver. In my physical setup, technically my LAN is a directly connected native vSwitch to the hyper-v host. Any device connected to hn1 (in my case) is connected to the same physical hyper-v host. pfSense is routing data between different vNICs which are connected to different vSwitches which are bound to unique physical uplinks.

Unfortunately I haven't time to prove this or packet cap it. This seems OS/Driver related to me.

m0nji

VMQ disabled on all VMs
RSC disabled on Hyper-V Host

PS C:\WINDOWS\system32> Get-VMSwitch -Name "Bridged_LAN" | Select-Object *RSC*

SoftwareRscEnabled RscOffloadEnabled
------------------ -----------------
             False             False

fresh booted FreeBSD 12.3 after 120 seconds iperf test (with speed problems)

dev.hn.0.rx.0.rsc_drop: 0
dev.hn.1.rx.0.rsc_drop: 0
dev.hn.2.rx.0.rsc_drop: 0
dev.hn.2.rx.0.rsc_pkts: 0
dev.hn.1.rx.0.rsc_pkts: 0
dev.hn.0.rx.0.rsc_pkts: 321

fresh booted FreeBSD 13.0 after 120s iperf test (speed problem does not exist)
sysctl's do not exist!

root@freebsd130:~ # sysctl dev.hn.0.rx.0.rsc_drop
sysctl: unknown oid 'dev.hn.0.rx.0.rsc_drop'
root@freebsd130:~ # sysctl dev.hn.1rx.0.rsc_drop
sysctl: unknown oid 'dev.hn.1.rx.0.rsc_drop'
root@freebsd130:~ # sysctl dev.hn.2rx.0.rsc_drop
sysctl: unknown oid 'dev.hn.2.rx.0.rsc_drop'
root@freebsd130:~ # sysctl dev.hn.0rx.0.rsc_pkts
sysctl: unknown oid 'dev.hn.0.rx.0.rsc_pkts'
root@freebsd130:~ # sysctl dev.hn.1rx.0.rsc_pkts
sysctl: unknown oid 'dev.hn.1.rx.0.rsc_pkts'
root@freebsd130:~ # sysctl dev.hn.2rx.0.rsc_pkts
sysctl: unknown oid 'dev.hn.2.rx.0.rsc_pkts'

hendryjl

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

It would be interesting to see if that varies for VMs that are still hitting issues even with RSC disabled.

Shutdown, Disable RCS on virtual switches, Boot, Test (success), Check CTLs... results attached (all 0's - good).

sysctls_rcsdisabled.txt

Definitely two different issues.

m0nji

just a theory:

on FreeBSD 13.0, RSC was removed completly from the kernel/system, thats why the sysctl's doesn't exist and so we do not have these problems?

on FreeBSD 12.3, RSC can't be disabled completly under some circumstances (drivers, nic's), thats why some people do see these problems?

if i interpret my values correctly, then even with RSC disabled, some packages hit RSC?! (321 packages) at least if "pkts" means packages ;)

EDIT: i think i will test tomorrow FreeBSD 12.2 (pfsense 2.5.x) and will see if RSC makes a difference there

hendryjl

@m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

EDIT: i think i will test tomorrow FreeBSD 12.2 (pfsense 2.5.x) and will see if RSC makes a difference there

I still have my image of 2.5.2 before I applied 2.6. I reverted to it when I first had my issues (before I ever tried disabling RSC on the Hyper-V virtual switches). I didn't see any issues. Later tonight I will revert to it again and see what the sysctls say.

stephenw10

RSC support wasn't added to hn(4) until 12.3 so I would expect to see no sysctls there.

The driver looks to try to use them based on the VNIC revision. I'm speculating that under some set of circumstances the driver/host/platform maybe trying and failing to use RSC even when it's disabled in the vswitch.

I'd like to check the sysctl and verbose boot logs from someone who is hitting this still with rsc seemingly disabled. Since we know they should be all zeros if it's actually disabled.

Steve

stephenw10

@m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

on FreeBSD 13.0, RSC was removed completly from the kernel/system, thats why the sysctl's doesn't exist and so we do not have these problems?

Ah, nope! It's because FreeBSD 13 is actually older than 12.3 and doesn't have any of the RSC support patches:
https://github.com/freebsd/freebsd-src/tree/releng/13.0/sys/dev/hyperv/netvsc

I would expect an 13-stable snapshot to fail the same way.

Steve

Bob.Dig

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I'd like to check the sysctl and verbose boot logs from someone who is hitting this still with rsc seemingly disabled. Since we know they should be all zeros if it's actually disabled.

Steve

I would like to but I need description for dummies how to do it.

m0nji

@bob-dig just open the VM preview window, reboot and on the boot screen hit the "boot options" and select verbose boot

FreeBSD 12.3

root@freebsd123:~ # grep -i "rsc" /var/log/messages
Feb 25 08:48:02 freebsd123 kernel: hn0: hwcaps rsc: ip4 1 ip6 1
Feb 25 08:48:02 freebsd123 kernel: hn0: offload rsc: ip4 2, ip6 2
Feb 25 08:48:02 freebsd123 kernel: hn1: hwcaps rsc: ip4 1 ip6 1
Feb 25 08:48:02 freebsd123 kernel: hn1: offload rsc: ip4 2, ip6 2
Feb 25 08:48:02 freebsd123 kernel: hn2: hwcaps rsc: ip4 1 ip6 1
Feb 25 08:48:02 freebsd123 kernel: hn2: offload rsc: ip4 2, ip6 2

pfSense 2.6.0

[2.6.0-RELEASE][admin@pfSense.home.arpa]/root: grep -i "rsc" /var/log/system.log                
Feb 25 07:56:22 pfSense kernel: hn0: hwcaps rsc: ip4 1 ip6 1
Feb 25 07:56:22 pfSense kernel: hn0: offload rsc: ip4 2, ip6 2
Feb 25 07:56:22 pfSense kernel: hn1: hwcaps rsc: ip4 1 ip6 1
Feb 25 07:56:22 pfSense kernel: hn1: offload rsc: ip4 2, ip6 2
Feb 25 07:56:22 pfSense kernel: hn2: hwcaps rsc: ip4 1 ip6 1
Feb 25 07:56:22 pfSense kernel: hn2: offload rsc: ip4 2, ip6 2