[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

m0nji

@stephenw10
we should keep an eye on this: https://reviews.freebsd.org/D29075?id=85183

Drumdevil

I have two Server 2019 hosts (Host1 & Host2) in my homelab, with both the same hardware. Both experienced slow WAN speeds since the update to 2.6. Normally WAN would reach about 1 Gbit, but after the update it was only about 30 Mbit. The servers each have two separate virtual switches for LAN and WAN. Inter-VLAN communication was as expected in 2.6.

The virtual switches are attached as follows:

WAN vSwitch -> Intel Pro/1000 PT, not shared with host
LAN vSwitch -> Microsoft Network Adapter Multiplexer Driver -> NIC Team, static teaming, dynamic load balance

On both hosts I did:

Get-VMSwitch | Set-VMSwitch -SoftwareRscEnabled $false

Then I confirmed both servers returned False for both switches using:

Get-VMSwitch | Select-Object *RSC*

This fixed the issue on Host1. On Host2, now the WAN speeds were OK too, BUT: Inter-VRF traffic became insanely slow. After troubleshooting I found that the only difference was that Host2 was more recently installed and had yet to get a bunch of Windows updates. I couldn't tell you which one it was, but the updates fixed it.

So if you have inter-VLAN performance issues after disabling RSC, try installing Windows updates.

fiblimitless

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

m0nji

@fiblimitless said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

Cannot confirm this. I still need to enable & disable RSC on the VM after every VM Reboot!
Just for Clarification: I disabled Large Send Offload on the virtual Switch and the physical NIC. But i didnt reboot the host yet.

Bob.Dig

@m0nji said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

Just for Clarification: I disabled Large Send Offload on the virtual Switch and the physical NIC. But i didnt reboot the host yet.

I think he meant the NIC in the vms, pretty easy to do that for a windows vm, more hard on a linux machine without GUI I guess. That's why I have not tested this.

I have my wacky solution for now using two external switches and wait for a permanent solution, probably coming with 2.7... lets hope.

bimmerdriver

@fiblimitless said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@drumdevil said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

installing Windows updates.

For anyone who is having throttled uploads. I found a fix.

Go to the your vNIC Properties > Configure > Advanced, and disable "Large Send Offload Version 2" for IPv4/ IPv6, and the upload speed goes back to normal.

I think this also will fix your inter-VLAN performance after disabling RSC.

I tried your suggestion, but it made no difference. I'm experiencing erratic dead slow uploads and downloads. Is it necessary to reboot after making this change?

I'm running pfSense 2.6.0 on Windows Server 2019 in a Generation 2 VM. There are virtual switches on the WAN and LAN. The WAN and LAN NICs are Intel I350. Not using VLANs. VMQ, IPsec task offloading, SR-IOV enabled.

fiblimitless

@bimmerdriver
I have same system as you except for SR-IOV I have it disabled.

For the download as mentioned before disable RSC via powershell.
For the uploads: Make sure you disable Large Send Offload v2 on the
virtual Switch and the physical NIC on the Host Machine.

No boot needed, maybe disable/enable NICs.

If that doesn't work, start disabling all these offload options you'll find in the v/NICs properties. and see if the speed goes back to normal. then work your way through by enabling them one by one. until you find the cause.

I'v had a previous experience with win2016 when it was in preview, with regular Windows VM on Hyper-V. The speed was so slow, and had this offloading disabled, until Microsoft pushed an update and fixed it, or it was a NIC driver update. I don't remember.

.

i386DX

@bimmerdriver did you try flipping -RscEnabled on and off on your pfsense vm network adapters?
This fixes both wan upload and inter vlan bandwidth for me on ws 2022.

I ended up scheduling this at startup until a fix is available:

Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

Bob.Dig

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I ended up scheduling this at startup until a fix is available:

Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

~~This works great~~, or not I am getting mixed results here, but how do you start that with Windows exactly?

bimmerdriver

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

@bimmerdriver did you try flipping -RscEnabled on and off on your pfsense vm network adapters?
This fixes both wan upload and inter vlan bandwidth for me on ws 2022.

I ended up scheduling this at startup until a fix is available:
Start-Sleep -Seconds 120 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $true ; `
Start-Sleep -Seconds 10 ; `
Get-VMNetworkAdapter -VMName "yourvmname" | Set-VMNetworkAdapter -RscEnabled $false

I did this yesterday. I didn't have a chance to post an update. It seems to have fixed the problem. I also noticed in the advanced settings for the virtual ethernet adapter that there are settings to enable and disable RSC for Ipv4 and IPv6. I will enable RSC on the switches and try disabling it in the driver to see if it accomplishes the same thing or not.

bimmerdriver

Here are the settings I'm using for Windows Server 2019 with a Generation 2 VM. I'm not suggesting these settings are optimal for everyone, but in my case, pfSense is working properly with them, unlike before I disabled RSC in the virtual switches.

My LAN and WAN NICs are I350-T2.

I'm not using VLANs.

In my system, the LAN and WAN NICs are connected to virtual switches. pfSense NICs are virtual, connected to LAN and WAN virtual switches.

Virtual switches: RSC disabled, SR-IOV enabled in external switches.

In all NICs (physical and virtual), VMQ, IPsec task offloading and SR-IOV are enabled. The only advanced feature I have enabled is port mirroring on WAN NICs so I can capture packets.

NIC Drivers: Jumbo packets (9014) are enabled. All possible offloading is enabled. IRRC, the only non-default setting is enabling jumbo packets.

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

I have not tried disabling RSC (IPv4 and IPv6) in the virtual NIC driver settings. Perhaps that would have the same effect as disabling RSC in the virtual switches.

YMMV.

i386DX

@bob-dig said in [After Upgrade inter (V)LAN communication is very slow (on Hyper-V).]

but how do you start that with Windows exactly?

A scheduled task that executes powershell.exe C:\path\to\script.ps1 as SYSTEM on startup works fine.
It should also be possible to trigger the task on vm restart by creating a custom eventlog query (look for event 18601 on Microsoft-Windows-Hyper-V-Worker/Admin).

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I'm not using VLANs.

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough. My upload was fine too after disabling it, but in my case it made more sense to operate on -RscEnabled on both vmnetworkadapters since I was going to enable/disable it anyway on the lan in order to restore inter vlan bandwidth.

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

Same for me, except ALTQ support which is enabled to allow FQ_CODEL usage.

Bob.Dig

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough.

I think on ws2019 and earlier it might be enough but not on ws2022.

inmarket

I have done some testing that might shed some light on this (in some circumstances). For me turning off RSC on the virtual switch solved the problem. This problem also occurs with raw FreeBSD 13 STABLE.

With RSC turned on I can successfully do...

ping -s 1400 host

(where the host is either on the same virtual switch or via a hardware port) PROVIDED THE PORT IS ON A LOCAL SEGMENT.
However,

ping -s 1400 www.google.com

comes back with...

76 bytes from XX.XXX.XX.XX ....
wrong total length 96 instead of 1428

Note: this is on the same virtual adapter, the same virtual switch, the same network port as the successful operation.

From this I conclude that the new RSC capabilities are probably broken for packets that are fragmented. The problem results in the packet received truncated to 96 bytes!! This occurs regardless of TCP/UDP/ICMP.

stephenw10

The patch for this that disables RSC by default and adds a sysctl to enable it should now be in new 2.7 snapshots:
https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Testing and feedback required there from anyone who can.

Steve

murdof

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Had tried all workarounds - nothing had worked for me.
Windows Server 2022 (AMD EPYC)
pfsense 2.6.0 (no issues with 2.5.2)

I upgraded from within the web ui to latest build (2.7.0.a.20220513.0600) and now speeds are back to normal.
While I was getting for my IPSec traffic about 350kb/sec now I'm getting 123Mbits/sec

Thanks for the fix!
Any chance when we should expect 2.7.0 to come out soon, or even maybe this fix to be backport to a 2.6.1 version?

Bob.Dig

I also can confirm, problem is gone!

22.05-BETA (amd64)
built on Fri May 13 06:28:24 UTC 2022
FreeBSD 12.3-STABLE

stephenw10

@murdof said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

maybe this fix to be backport to a 2.6.1 version?

We may be able to include it in a point release if one happened but that would only happen if a security issue required it. I'm not aware of one currently.

Great news that it fixed it as expected though.

Steve

benc

Issue:
Very low WAN throughput after upgrading to 2.6.0

Hardware:
Dell PowerEdge R730
Intel Gigabit 4P I350 -t rNDC (onboard quad gigabit NICs)
Dual Xeon E5-2620 v3 @ 2.4 Ghz
32 GB DDR3

Environment:
Windows Server 2019, Hyper-V
This server also hosts VMs for AD controller, Issabel PBX, and Unifi Video (Ubuntu 14).
pfSense 2.6.0 VM, Generation 2, 8 cores, 4 GB RAM, 16 GB VHDX, 2 vNICs

In the virtual NICs for pfSense,
SR-IOV is disabled,
VMQ is disabled,
and IPsec task offloading is disabled.

In pfSense web gui System > Advanced > Networking,
Hardware checksum offload, hardware TCP segmentation offload, and hardware large receive offload are disabled.

WAN:
Cable modem directly connected to NIC1 on the server. 450 Mbit down / 20 Mbit up. The bandwidth has always been consistent. Host OS is not allowed to use this NIC. pfSense virtual NIC is the only thing connected to it.

LAN:
8 port dumb gigabit switch. Two wireless access points and one desktop PC are physically connected to the switch. NIC2 on the server is connected to this switch also.

Issues:
After upgrading from pfSense 2.5.2 to 2.6.0, all traffic going through the firewall seemed to slow down and my OpenVPN connection to my workplace started having issues. Internet throughput was all over the place, from < 1 Mbit up to about 16 Mbits. For some reason, YouTube was still able to pull down video at over 60 Mbits. It seemed like a physical issue, like a bad cable or NIC or maybe a duplex mismatch.

Troubleshooting:
Rebooted cable modem.
Tested WAN with laptop using existing cabling. Everything is OK.
Rebooted cable modem again.
Rebooted Hyper-V host.
Set up new pfSense VM 2.6.0 from scratch, keeping as many defaults as possible. Same slow results as before.
Upgraded server firmware to latest version (2.13.0).
Upgraded server iDRAC firmware to latest (2.83.83.83).
Upgraded server network drivers to latest (19.5.12).
Toggled the 3 NIC offload settings in pfSense one at a time in all the different combinations, rebooting the VM each time.
Toggled the SR-IOV and VMQ settings in Hyper-V for both virtual NICs, rebooting VM each time.

None of this made any significant difference.

Set up another 2.6.0 from scratch and set up new virtual switches using NIC3 and NIC4. Same as before.

I set up a new internal switch in Hyper-V and added it to pfSense. I also set up a test Win10 VM and assigned its vNIC to the new internal switch. It had the same slowness as when using an external Hyper-V switch. I noticed that it wasn't just internet traffic, but pretty much any traffic being routed by pfSense had weird latency and throughput issues. Using iperf3 from a laptop, I could upload > 900 Mbits to this test VM, but when downloading (-R option) I was getting between 4 kilobits and 70 kilobits, and sometimes it would just drop out completely.

One of my tests involved downloading a large file over FTP from a server at work over a VPN connection. Until I upgraded to 2.6.0, I used to get the full bandwidth of the ISP during this transfer. With 2.6.0, I had no issues connecting to the FTP server or browsing through folders, but the max download speed was less than 100 kilobits.

All the issues I had were fixed immediately by upgrading to 2.7.0 DEV, and transfers went back up to ISP limited speeds.

Fix:
Install pfSense 2.5.2 or earlier from scratch -or- Upgrade existing installation to 2.7.0 DEV release.

Ultimately I decided upgrading to 2.7.0 was easier than further troubleshooting, and I never looked into disabling RSC. I didn't want to have to deal with things like running scripts every time the VM was rebooted like others have reported.

Tried backing up the config from 2.6.0 and restoring to a new install of 2.5.2 (yes, I should have made a checkpoint before doing the upgrade...), but this bricked the VM and it kept complaining about an XML error. I feel like I've backed up and restored to an older version in the past, but maybe I was mistaken.

Conclusion:
Maybe there should be a 2.6.1 release? I mean, if I were just getting into pfSense and I just happened to be using Hyper-V, and I just happened to be having weird bandwidth issues... It's just not a great user experience to have to troubleshoot right out of the box. The RSC fix (or whatever it was) that was added in 2.7.0 DEV really needs to be addressed in an official release soon, even if it's not security related.

Cool_Corona

@benc Couldnt agree more.