[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

benc

Issue:
Very low WAN throughput after upgrading to 2.6.0

Hardware:
Dell PowerEdge R730
Intel Gigabit 4P I350 -t rNDC (onboard quad gigabit NICs)
Dual Xeon E5-2620 v3 @ 2.4 Ghz
32 GB DDR3

Environment:
Windows Server 2019, Hyper-V
This server also hosts VMs for AD controller, Issabel PBX, and Unifi Video (Ubuntu 14).
pfSense 2.6.0 VM, Generation 2, 8 cores, 4 GB RAM, 16 GB VHDX, 2 vNICs

In the virtual NICs for pfSense,
SR-IOV is disabled,
VMQ is disabled,
and IPsec task offloading is disabled.

In pfSense web gui System > Advanced > Networking,
Hardware checksum offload, hardware TCP segmentation offload, and hardware large receive offload are disabled.

WAN:
Cable modem directly connected to NIC1 on the server. 450 Mbit down / 20 Mbit up. The bandwidth has always been consistent. Host OS is not allowed to use this NIC. pfSense virtual NIC is the only thing connected to it.

LAN:
8 port dumb gigabit switch. Two wireless access points and one desktop PC are physically connected to the switch. NIC2 on the server is connected to this switch also.

Issues:
After upgrading from pfSense 2.5.2 to 2.6.0, all traffic going through the firewall seemed to slow down and my OpenVPN connection to my workplace started having issues. Internet throughput was all over the place, from < 1 Mbit up to about 16 Mbits. For some reason, YouTube was still able to pull down video at over 60 Mbits. It seemed like a physical issue, like a bad cable or NIC or maybe a duplex mismatch.

Troubleshooting:
Rebooted cable modem.
Tested WAN with laptop using existing cabling. Everything is OK.
Rebooted cable modem again.
Rebooted Hyper-V host.
Set up new pfSense VM 2.6.0 from scratch, keeping as many defaults as possible. Same slow results as before.
Upgraded server firmware to latest version (2.13.0).
Upgraded server iDRAC firmware to latest (2.83.83.83).
Upgraded server network drivers to latest (19.5.12).
Toggled the 3 NIC offload settings in pfSense one at a time in all the different combinations, rebooting the VM each time.
Toggled the SR-IOV and VMQ settings in Hyper-V for both virtual NICs, rebooting VM each time.

None of this made any significant difference.

Set up another 2.6.0 from scratch and set up new virtual switches using NIC3 and NIC4. Same as before.

I set up a new internal switch in Hyper-V and added it to pfSense. I also set up a test Win10 VM and assigned its vNIC to the new internal switch. It had the same slowness as when using an external Hyper-V switch. I noticed that it wasn't just internet traffic, but pretty much any traffic being routed by pfSense had weird latency and throughput issues. Using iperf3 from a laptop, I could upload > 900 Mbits to this test VM, but when downloading (-R option) I was getting between 4 kilobits and 70 kilobits, and sometimes it would just drop out completely.

One of my tests involved downloading a large file over FTP from a server at work over a VPN connection. Until I upgraded to 2.6.0, I used to get the full bandwidth of the ISP during this transfer. With 2.6.0, I had no issues connecting to the FTP server or browsing through folders, but the max download speed was less than 100 kilobits.

All the issues I had were fixed immediately by upgrading to 2.7.0 DEV, and transfers went back up to ISP limited speeds.

Fix:
Install pfSense 2.5.2 or earlier from scratch -or- Upgrade existing installation to 2.7.0 DEV release.

Ultimately I decided upgrading to 2.7.0 was easier than further troubleshooting, and I never looked into disabling RSC. I didn't want to have to deal with things like running scripts every time the VM was rebooted like others have reported.

Tried backing up the config from 2.6.0 and restoring to a new install of 2.5.2 (yes, I should have made a checkpoint before doing the upgrade...), but this bricked the VM and it kept complaining about an XML error. I feel like I've backed up and restored to an older version in the past, but maybe I was mistaken.

Conclusion:
Maybe there should be a 2.6.1 release? I mean, if I were just getting into pfSense and I just happened to be using Hyper-V, and I just happened to be having weird bandwidth issues... It's just not a great user experience to have to troubleshoot right out of the box. The RSC fix (or whatever it was) that was added in 2.7.0 DEV really needs to be addressed in an official release soon, even if it's not security related.

Cool_Corona

@benc Couldnt agree more.

murdof

@benc I was on the same boat as you.
I was wondering if I could downgrade from 2.6.0 to 2.5.2.

I just got the backup and restored and it worked.

Maybe you need to manually check your XML file and figure out what is causing the problem and remove it.

The reason I didn't stick with the nightly 2.7.0 was because of haproxy. It wasn't handling the certificates properly and was getting warning. Makes sense I guess since pfsense was nightly and haproxy was still 2.6.0...

stephenw10

If you're running a 2.7 snapshot it will pull the current HAProxy package. There shouldn't be any compatibility issues there.

murdof

@stephenw10 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

If you're running a 2.7 snapshot it will pull the current HAProxy package. There shouldn't be any compatibility issues there.

Hm... I should had raised a bug report then...

im.thatoneguy

Although this appears to have been resolved in an upcoming update for the sake of data collection thoroughness:

Hyper-V 2019 (pending updates) multiple VLANs on Connect-X 3: Slow
(VMQ enabled, SR-IOV enabled in Hyper-V)
Server 2022 (fully up to date) multiple VLANs on Broadcom 10gbe (Supermicro): Fast
(VMQ enabled (Maybe not supported), SR-IOV disabled in Hyper-V)

Disabling RSC on the HV2019 host's vSwitch matched performance between the two.

fsr

@hendryjl Thanks a lot, that worked perfectly!!
Microsoft has a curious way of developing optimizations....
In my case, the internet connection worked horribly slow. Downloading some less than 200 MB file took more than 1 hour when using the Hyper-V virtualized pfsense, and less than 20 seconds when downloading directly from the hardware router. After disabling RSC, it downloads as fast as the hardware router.

jba

I have had a problem updating the pfsesne version in virtual machines (2019 server) the download works slowly, I have found this solution to create the virtual machine in version 5.0
PowerShell
New-VM -Name "Pfsesne" -Version 5.0
speed works fine

Spanish
He tenido un problema al actualizar la version pfsesne en maquinas virtuales (2019 server) funciona lenta la descarga, he encontrado esta solucion crear la maquina virtual en version 5.0
PowerShell
New-VM -Name "Pfsesne" -Version 5.0
La velocidad funciona correctamente

deanfourie

@jba So ive been having the same issues!

My WAN speeds are super slow! If I do a speedtest on any LAN side client, its 2mbps down and 0.17 up. Totally dies!

If I do a speedtest on the HyperV Host itself, I am getting my full throughput of around 80mbps down and 7 up. Which is normal for me.

I have my WAN interface setup, and set to "Do not share this interface with the host" option ticked. Which still leaves me only 1 interface for WAN. In this interface properties on the host directly the only option I have ticked is "HyperV Extensible Virtual Switch", I have all other options including "Client for Microsoft Windows" and TCPIP v4 / Link Layer, etc etc disabled.

When this is the case, I am getting issues with my speed. However, If I enable IPv4 on the WAN interface and allow the interface to actually obtain DHCP, the issues goes away and all LAN clients get the full throughput. Enabling IPv4 on the WAN interface gives the HOST internet. (just to help you understand my setup a bit).

If I disable it again, it drops down to the miserable 2mbps download etc.

I usually have the HOST IPv4 on the WAN option disabled to take the HOST off the internet directly, thus pfSense is the only device connected to, in my case, a WAN upstream gateway.

deanfourie

@deanfourie here is an example of the behavior I am talking about.

https://youtu.be/VLq4vMvIGgE

jamroz

@rmh-0 said in After Upgrade inter (V)LAN communication is very slow (on Hyper-V).:

RSC

I am complete pfSense noob. I installed it for the first time about an hour ago. The first thing that I noticed was the terrible throughput. Instead of using Google I decided to try the Community here and your solution worked. I was expecting this to be one of those one you wrestle with for hours but you nailed. Thanks!

inmarket

Here is the cause, the outcomes, and how to solve it if you have problems...

The cause:
It appears that Hyper-V on Server 2019 has a bug in its RSC code that causes packets to be corrupted. There are reports that this problem is solved in Server 2022 but I have not confirmed this.
This bug causes problems for ALL virtual machines provided they try to implement RSC code in their ethernet drivers. This means that Linux, Windows and FreeBSD are all affected if the version of the operating system supports RSC. For Linux this is all recent kernels and for Windows this is Windows 10, 11, Server 2019, Server 2022 at least.
Talking about pfSense, it is based on FreeBSD. FreeBSD starting supporting RSC in the hn driver as of 12.3 and 13.1. This therefore affects pfSense from Version 2.6

As from FreeBSD kernel 13.X, RSC has been turned off by default (thus fixing the problem) but can optionally be turned on. Also, it appears that pfSense 2.7 turns off RSC thus also fixing the problem. Note: these fixes do not solve the problem for Linux or Windows Guest VM's.

The outcomes:
The problem exhibits in one of two ways, either the corrupted packets go up the IP stack and cause "interesting" affects, or the kernel drops the corrupted packets. How this is handled changes from Operating system to operating system and even kernel version to kernel version (in particular Linux). It is this behaviour that causes the slow network performance as packets need to be retried etc.

Solutions:

The best solution is to turn off Hyper-V RSC on each virtual switch. The powershell commands to do this are listed in previous posts. This solves it for pfSense as well as Linux and Windows guests.
An alternative is to turn RSC off on each virtual ethernet adapter on each virtual machine. This is a poor solution because it requires a lot of systems to be touched AND also doesn't stick across a Hyper-V reboot due to another bug in Hyper-V (even though the UI indicates that it has stuck).
Another alternative is to upgrade pfSense to the 2.7 stream. For FreeBSD upgrade to the very latest kernel which turns off RSC by default. For Linux and Windows, no current solution is known. Note: this solution does not solve it for all operating systems.
Another alternative is to upgrade Hyper-V to a version that doesn't have the problem. I have heard that Server 2022 Hyper-V doesn't have the problem but I have not personally confirmed that.

Conclusion
This is a right royal stuff-up by Microsoft (not that they will admit it). Unfortunately the problem has probably existed for a long time but it has only become noticeable as operating systems have started implementing RSC support in their ethernet drivers.
Fortunately, there is a simple workaround (#1 above). I suspect however that this problem has probably killed RSC as a reliable performance enhancing strategy especially now that kernels are turning it off by default.

deanfourie

@inmarket great summary, thanks!

I must be honest, I tried disabling RSC with pfSense 2.6 and it didn't work for me?

I have now however installed pfSense + 22.05 and the issue appears to be gone however I would still like it turned off for future VMs.

What's the correct command for this? I know it's posted above but like I say, it did not work for me? Maybe I'm missing something.

Cheers

gogglespisano

@deanfourie Same here for Win 10 + 2.6.

I don't have -RSCEnabled on the VMNetworkAdapters and -SoftwareRSCEnabled on the VMSwitches were already $false.

I need more clues.
Does it need to be toggled regardless of current state?
Does the VMSwitch need to be not in use when setting it?

inmarket

Just as a quick note for those who think the solutions above don't work...

After updating RSC at the virtual switch (and on every host machine if you are using a cluster), you will need to restart each virtual machine. You may also have to restart Hyper-V (or rather Server 2019) for it to show its affect.

There may be other configuration problems that also result in slow performance but that is not what this is post is about. This post is specifically about the performance issues caused by RSC in Hyper-V Server 2019 in pfSense 2.6 that weren't there in pfSense 2.5

murdof

@inmarket I’m running windows server 2022.
The problem is there and tried all the recommended “solutions”.
Only downgrading to 2.5 or upgrading to 2.7 fixed the performance issue.

emigu

@werter said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

Decision: do not use hyper-v as virtualization platform ))
Better try Proxmox VE (open source)

Netgate creating a buggy version is not the virtualization plattforms fault.

Bob.Dig

@emigu It was a change in FreeBSD and with that an incompatibility of pfSense with Hyper-V. Netgate is not selling any device with Hyper-V on it, so it is nobody's fault to begin with.

johnpoz

@bob-dig said in pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V:

It was a change in FreeBSD and with that an incompatibility of pfSense with Hyper-V.

Its not just freebsd - if you do a google for hyper-v and RSC you will find lots of issues where it can cause slow network, etc.

emigu

@bob-dig Something that works in 2.5.2 splendid, while boasting in the official documentation that it supports Hyper-V:
https://docs.netgate.com/pfsense/en/latest/virtualization/index.html - https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

Which then stops working in 2.6.0 is most certainly Netgate's fault, more specifically the QA department.

Clearly there were 0.0% tests done with Hyper-V (One of the largest hypervisors in the world).

If you think it's acceptable not testing and verifying (A very simple test that can be automated) a new major version on a hypervisor listed in the documentation that happens to be the 2nd or 3rd most used hypervisor in the world, AND the one that's powering one of the biggest public clouds in the world - I don't really know how to counter that other than let's agree to disagree.

I certainly hope this lack of quality doesn't extend to netgates paid versions.