[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

bimmerdriver

Here are the settings I'm using for Windows Server 2019 with a Generation 2 VM. I'm not suggesting these settings are optimal for everyone, but in my case, pfSense is working properly with them, unlike before I disabled RSC in the virtual switches.

My LAN and WAN NICs are I350-T2.

I'm not using VLANs.

In my system, the LAN and WAN NICs are connected to virtual switches. pfSense NICs are virtual, connected to LAN and WAN virtual switches.

Virtual switches: RSC disabled, SR-IOV enabled in external switches.

In all NICs (physical and virtual), VMQ, IPsec task offloading and SR-IOV are enabled. The only advanced feature I have enabled is port mirroring on WAN NICs so I can capture packets.

NIC Drivers: Jumbo packets (9014) are enabled. All possible offloading is enabled. IRRC, the only non-default setting is enabling jumbo packets.

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

I have not tried disabling RSC (IPv4 and IPv6) in the virtual NIC driver settings. Perhaps that would have the same effect as disabling RSC in the virtual switches.

YMMV.

i386DX

@bob-dig said in [After Upgrade inter (V)LAN communication is very slow (on Hyper-V).]

but how do you start that with Windows exactly?

A scheduled task that executes powershell.exe C:\path\to\script.ps1 as SYSTEM on startup works fine.
It should also be possible to trigger the task on vm restart by creating a custom eventlog query (look for event 18601 on Microsoft-Windows-Hyper-V-Worker/Admin).

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

I'm not using VLANs.

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough. My upload was fine too after disabling it, but in my case it made more sense to operate on -RscEnabled on both vmnetworkadapters since I was going to enable/disable it anyway on the lan in order to restore inter vlan bandwidth.

@bimmerdriver said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

In pfSense:

Disable hardware checksum offload: not checked
Disable hardware TCP segmentation offload: checked
Disable hardware large receive offload: checked
Enable the ALTQ support for hn NICs: not checked

Same for me, except ALTQ support which is enabled to allow FQ_CODEL usage.

Bob.Dig

@i386dx said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

With no vlans involved -SoftwareRscEnabled $false on the vswitch could be enough.

I think on ws2019 and earlier it might be enough but not on ws2022.

inmarket

I have done some testing that might shed some light on this (in some circumstances). For me turning off RSC on the virtual switch solved the problem. This problem also occurs with raw FreeBSD 13 STABLE.

With RSC turned on I can successfully do...

ping -s 1400 host

(where the host is either on the same virtual switch or via a hardware port) PROVIDED THE PORT IS ON A LOCAL SEGMENT.
However,

ping -s 1400 www.google.com

comes back with...

76 bytes from XX.XXX.XX.XX ....
wrong total length 96 instead of 1428

Note: this is on the same virtual adapter, the same virtual switch, the same network port as the successful operation.

From this I conclude that the new RSC capabilities are probably broken for packets that are fragmented. The problem results in the packet received truncated to 96 bytes!! This occurs regardless of TCP/UDP/ICMP.

stephenw10

The patch for this that disables RSC by default and adds a sysctl to enable it should now be in new 2.7 snapshots:
https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Testing and feedback required there from anyone who can.

Steve

murdof

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

https://github.com/pfsense/FreeBSD-src/commit/b3efc50b66a446afe51f6502bf06b7450bac2ba7

Had tried all workarounds - nothing had worked for me.
Windows Server 2022 (AMD EPYC)
pfsense 2.6.0 (no issues with 2.5.2)

I upgraded from within the web ui to latest build (2.7.0.a.20220513.0600) and now speeds are back to normal.
While I was getting for my IPSec traffic about 350kb/sec now I'm getting 123Mbits/sec

Thanks for the fix!
Any chance when we should expect 2.7.0 to come out soon, or even maybe this fix to be backport to a 2.6.1 version?

Bob.Dig

I also can confirm, problem is gone!

22.05-BETA (amd64)
built on Fri May 13 06:28:24 UTC 2022
FreeBSD 12.3-STABLE

stephenw10

@murdof said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

maybe this fix to be backport to a 2.6.1 version?

We may be able to include it in a point release if one happened but that would only happen if a security issue required it. I'm not aware of one currently.

Great news that it fixed it as expected though.

Steve

benc

Issue:
Very low WAN throughput after upgrading to 2.6.0

Hardware:
Dell PowerEdge R730
Intel Gigabit 4P I350 -t rNDC (onboard quad gigabit NICs)
Dual Xeon E5-2620 v3 @ 2.4 Ghz
32 GB DDR3

Environment:
Windows Server 2019, Hyper-V
This server also hosts VMs for AD controller, Issabel PBX, and Unifi Video (Ubuntu 14).
pfSense 2.6.0 VM, Generation 2, 8 cores, 4 GB RAM, 16 GB VHDX, 2 vNICs

In the virtual NICs for pfSense,
SR-IOV is disabled,
VMQ is disabled,
and IPsec task offloading is disabled.

In pfSense web gui System > Advanced > Networking,
Hardware checksum offload, hardware TCP segmentation offload, and hardware large receive offload are disabled.

WAN:
Cable modem directly connected to NIC1 on the server. 450 Mbit down / 20 Mbit up. The bandwidth has always been consistent. Host OS is not allowed to use this NIC. pfSense virtual NIC is the only thing connected to it.

LAN:
8 port dumb gigabit switch. Two wireless access points and one desktop PC are physically connected to the switch. NIC2 on the server is connected to this switch also.

Issues:
After upgrading from pfSense 2.5.2 to 2.6.0, all traffic going through the firewall seemed to slow down and my OpenVPN connection to my workplace started having issues. Internet throughput was all over the place, from < 1 Mbit up to about 16 Mbits. For some reason, YouTube was still able to pull down video at over 60 Mbits. It seemed like a physical issue, like a bad cable or NIC or maybe a duplex mismatch.

Troubleshooting:
Rebooted cable modem.
Tested WAN with laptop using existing cabling. Everything is OK.
Rebooted cable modem again.
Rebooted Hyper-V host.
Set up new pfSense VM 2.6.0 from scratch, keeping as many defaults as possible. Same slow results as before.
Upgraded server firmware to latest version (2.13.0).
Upgraded server iDRAC firmware to latest (2.83.83.83).
Upgraded server network drivers to latest (19.5.12).
Toggled the 3 NIC offload settings in pfSense one at a time in all the different combinations, rebooting the VM each time.
Toggled the SR-IOV and VMQ settings in Hyper-V for both virtual NICs, rebooting VM each time.

None of this made any significant difference.

Set up another 2.6.0 from scratch and set up new virtual switches using NIC3 and NIC4. Same as before.

I set up a new internal switch in Hyper-V and added it to pfSense. I also set up a test Win10 VM and assigned its vNIC to the new internal switch. It had the same slowness as when using an external Hyper-V switch. I noticed that it wasn't just internet traffic, but pretty much any traffic being routed by pfSense had weird latency and throughput issues. Using iperf3 from a laptop, I could upload > 900 Mbits to this test VM, but when downloading (-R option) I was getting between 4 kilobits and 70 kilobits, and sometimes it would just drop out completely.

One of my tests involved downloading a large file over FTP from a server at work over a VPN connection. Until I upgraded to 2.6.0, I used to get the full bandwidth of the ISP during this transfer. With 2.6.0, I had no issues connecting to the FTP server or browsing through folders, but the max download speed was less than 100 kilobits.

All the issues I had were fixed immediately by upgrading to 2.7.0 DEV, and transfers went back up to ISP limited speeds.

Fix:
Install pfSense 2.5.2 or earlier from scratch -or- Upgrade existing installation to 2.7.0 DEV release.

Ultimately I decided upgrading to 2.7.0 was easier than further troubleshooting, and I never looked into disabling RSC. I didn't want to have to deal with things like running scripts every time the VM was rebooted like others have reported.

Tried backing up the config from 2.6.0 and restoring to a new install of 2.5.2 (yes, I should have made a checkpoint before doing the upgrade...), but this bricked the VM and it kept complaining about an XML error. I feel like I've backed up and restored to an older version in the past, but maybe I was mistaken.

Conclusion:
Maybe there should be a 2.6.1 release? I mean, if I were just getting into pfSense and I just happened to be using Hyper-V, and I just happened to be having weird bandwidth issues... It's just not a great user experience to have to troubleshoot right out of the box. The RSC fix (or whatever it was) that was added in 2.7.0 DEV really needs to be addressed in an official release soon, even if it's not security related.

Cool_Corona

@benc Couldnt agree more.

murdof

@benc I was on the same boat as you.
I was wondering if I could downgrade from 2.6.0 to 2.5.2.

I just got the backup and restored and it worked.

Maybe you need to manually check your XML file and figure out what is causing the problem and remove it.

The reason I didn't stick with the nightly 2.7.0 was because of haproxy. It wasn't handling the certificates properly and was getting warning. Makes sense I guess since pfsense was nightly and haproxy was still 2.6.0...

stephenw10

If you're running a 2.7 snapshot it will pull the current HAProxy package. There shouldn't be any compatibility issues there.

murdof

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

If you're running a 2.7 snapshot it will pull the current HAProxy package. There shouldn't be any compatibility issues there.

Hm... I should had raised a bug report then...

im.thatoneguy

Although this appears to have been resolved in an upcoming update for the sake of data collection thoroughness:

Hyper-V 2019 (pending updates) multiple VLANs on Connect-X 3: Slow
(VMQ enabled, SR-IOV enabled in Hyper-V)
Server 2022 (fully up to date) multiple VLANs on Broadcom 10gbe (Supermicro): Fast
(VMQ enabled (Maybe not supported), SR-IOV disabled in Hyper-V)

Disabling RSC on the HV2019 host's vSwitch matched performance between the two.

fsr

@hendryjl Thanks a lot, that worked perfectly!!
Microsoft has a curious way of developing optimizations....
In my case, the internet connection worked horribly slow. Downloading some less than 200 MB file took more than 1 hour when using the Hyper-V virtualized pfsense, and less than 20 seconds when downloading directly from the hardware router. After disabling RSC, it downloads as fast as the hardware router.

jba

I have had a problem updating the pfsesne version in virtual machines (2019 server) the download works slowly, I have found this solution to create the virtual machine in version 5.0
PowerShell
New-VM -Name "Pfsesne" -Version 5.0
speed works fine

Spanish
He tenido un problema al actualizar la version pfsesne en maquinas virtuales (2019 server) funciona lenta la descarga, he encontrado esta solucion crear la maquina virtual en version 5.0
PowerShell
New-VM -Name "Pfsesne" -Version 5.0
La velocidad funciona correctamente

deanfourie

@jba So ive been having the same issues!

My WAN speeds are super slow! If I do a speedtest on any LAN side client, its 2mbps down and 0.17 up. Totally dies!

If I do a speedtest on the HyperV Host itself, I am getting my full throughput of around 80mbps down and 7 up. Which is normal for me.

I have my WAN interface setup, and set to "Do not share this interface with the host" option ticked. Which still leaves me only 1 interface for WAN. In this interface properties on the host directly the only option I have ticked is "HyperV Extensible Virtual Switch", I have all other options including "Client for Microsoft Windows" and TCPIP v4 / Link Layer, etc etc disabled.

When this is the case, I am getting issues with my speed. However, If I enable IPv4 on the WAN interface and allow the interface to actually obtain DHCP, the issues goes away and all LAN clients get the full throughput. Enabling IPv4 on the WAN interface gives the HOST internet. (just to help you understand my setup a bit).

If I disable it again, it drops down to the miserable 2mbps download etc.

I usually have the HOST IPv4 on the WAN option disabled to take the HOST off the internet directly, thus pfSense is the only device connected to, in my case, a WAN upstream gateway.

deanfourie

@deanfourie here is an example of the behavior I am talking about.

https://youtu.be/VLq4vMvIGgE

jamroz

@rmh-0 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

RSC

I am complete pfSense noob. I installed it for the first time about an hour ago. The first thing that I noticed was the terrible throughput. Instead of using Google I decided to try the Community here and your solution worked. I was expecting this to be one of those one you wrestle with for hours but you nailed. Thanks!

inmarket

Here is the cause, the outcomes, and how to solve it if you have problems...

The cause:
It appears that Hyper-V on Server 2019 has a bug in its RSC code that causes packets to be corrupted. There are reports that this problem is solved in Server 2022 but I have not confirmed this.
This bug causes problems for ALL virtual machines provided they try to implement RSC code in their ethernet drivers. This means that Linux, Windows and FreeBSD are all affected if the version of the operating system supports RSC. For Linux this is all recent kernels and for Windows this is Windows 10, 11, Server 2019, Server 2022 at least.
Talking about pfSense, it is based on FreeBSD. FreeBSD starting supporting RSC in the hn driver as of 12.3 and 13.1. This therefore affects pfSense from Version 2.6

As from FreeBSD kernel 13.X, RSC has been turned off by default (thus fixing the problem) but can optionally be turned on. Also, it appears that pfSense 2.7 turns off RSC thus also fixing the problem. Note: these fixes do not solve the problem for Linux or Windows Guest VM's.

The outcomes:
The problem exhibits in one of two ways, either the corrupted packets go up the IP stack and cause "interesting" affects, or the kernel drops the corrupted packets. How this is handled changes from Operating system to operating system and even kernel version to kernel version (in particular Linux). It is this behaviour that causes the slow network performance as packets need to be retried etc.

Solutions:

The best solution is to turn off Hyper-V RSC on each virtual switch. The powershell commands to do this are listed in previous posts. This solves it for pfSense as well as Linux and Windows guests.
An alternative is to turn RSC off on each virtual ethernet adapter on each virtual machine. This is a poor solution because it requires a lot of systems to be touched AND also doesn't stick across a Hyper-V reboot due to another bug in Hyper-V (even though the UI indicates that it has stuck).
Another alternative is to upgrade pfSense to the 2.7 stream. For FreeBSD upgrade to the very latest kernel which turns off RSC by default. For Linux and Windows, no current solution is known. Note: this solution does not solve it for all operating systems.
Another alternative is to upgrade Hyper-V to a version that doesn't have the problem. I have heard that Server 2022 Hyper-V doesn't have the problem but I have not personally confirmed that.

Conclusion
This is a right royal stuff-up by Microsoft (not that they will admit it). Unfortunately the problem has probably existed for a long time but it has only become noticeable as operating systems have started implementing RSC support in their ethernet drivers.
Fortunately, there is a simple workaround (#1 above). I suspect however that this problem has probably killed RSC as a reliable performance enhancing strategy especially now that kernels are turning it off by default.