Crippled download speed on Server 2022 Hyper-V using pfSense
-
I know that there are a billion topics both here and through a net search, on issues with Hyper-V, pfSense and slow speeds.
I have spent countless hours trying to figure out what's up with this, and have narrowed it down a bit.
The setup is a hosted server with just one Intel NIC.
It has two IP addresses managed by a virtual MAC address.I have Hyper-V running on a Windows Server 2022, with 4 virtual machines, where one being the router in form of pfSense.
I have configured an external switch, which shares the NIC with the host OS. The pfSense WAN NIC, connected to the external switch, has the additional virtual MAC address, and it receives the secondary IP from my host as it should.
I have configured a private switch, which is used for pfSense LAN and is the gateway for the remaning virtual machines.
I have tried with an internal switch, but it's the same and shouldn't matter at all.Speed test on the host OS is 1Gbit up/down.
Speed test (CLI) on pfSense directly is 1-5mbit down / 700-800mbit up.
Speed test on a virtual machine is 1-5mbit up/down.I had to disable the Large Send Offload for the vSwitch to bring up the upload speed, or it would be crippled at pfSense as well.
Now the weird part, if I shut down pfSense and re-configure a VM to use the external switch with the virtual MAC, it runs at 1gbit up/down. So it would seem as if pfSense is completely bottlenecking somehow.
In comparison, I have an ESXi server running essentially the same way, with the exact same pfSense version and setup, with zero bandwidth issues.
-
@ip this is exactly and I mean exactly the issue I have been chasing since June 29. I have an esxi on 2.6 that's great, on the other end hyperv 2022 on 22.05 now which about every 8 to 12 hours goes from gig download to 1 meg. The upload speed never changes. The RSC changes helped at first on 2.6 but after a Windows update it started this. I have since tried 22.01 and 22.05 in an effort to fix it. My server in a gen9 dl380. About 6 vms, almost 0 load.
-
@dougroutledge I'm glad I'm not the only one to have the issue, although it's of little comfort. I run 2.6 on both servers myself.
To me however, the slowdown is instant, whenever the traffic passes through pfSense. I have been going through all kinds of settings, which may have an effect on bottlenecking, but my efforts have been in vain so far. I'm considering trying a different piece of kit, although I prefer pfSense, as I have used it for a long time now. I just feel like it's the only thing left to try. It's quite maddening, the longer you stare at the issue, which makes no sense, emptying your mind of all possible solutions.
I really hope that some of the experts around here, may have an idea on what to try. -
@ip I really believe it's the nic drivers on freebsd 12.3 and hyperv. I bet if I restored the configuration to 2.5.2 everything would work like it did for a year.
-
Found this blog post:
https://blog.nuvotex.de/pfsense-2-6-on-hyper-v-with-low-throughput/ -
@dougroutledge An interesting thought. In my seemingly endless Google searches, I noticed several people hinting at 2.5.2 as the "better" version. Since I haven't done anything much in terms of configuration on the Hyper-V node in question, I could try to install 2.5.2 and see if things are different.
@mvikman Thank you for the link. Very interesting. It matches what @dougroutledge says. While this is frustrating, It's great with a possibility to narrow it down.
I will go ahead and make an alternate VM with pfSense 2.5.2 on it and see what results I get. I'll let you guys know.
-
Okay, back with some good results.
I did try creating a new "router VM" with pfSense 2.5.2 and it works out of the box without any speed issues.
Next I tried following the link by @mvikman. After reading that blog post and executing the below PowerShell commands, the download speed was fixed.
- netsh int tcp set global rsc=disabled
- Get-NetAdapterRsc | Disable-NetAdapterRsc
- Get-VmSwitch | Set-VMSwitch -EnableSoftwareRsc $false
Now I also had issues with the upload, but disabling LSO for the Virtual Adapter on the host OS and inside of each VM resolved this. I didn't have to change any settings overall on the physical network adapter, which in my case is an Intel I219-LM.
Following the case: https://reviews.freebsd.org/D34507 it will be interesting to see what happens in a future update. At least there are two ways to work around the issue for now.
-
@ip I went the other way, and spun up an opnsense on the same Hyper-v server, it's had none of the slowness. It's a little different to get around in, but most of my settings were easy to copy and paste, so it didn't take long.
-
@dougroutledge I did have a look at it and was considering giving it a try. I still might. Should be easy enough adapting I think. But good to know that it works as intended out of the box.
-
@ip I had the same issue with Windows Server 2022 Hyper-V.
I fixed it by disabling the following on the network adapters on the Hyper-V Host.- Disable "Large Send Offload v2 (IPv4)" and "Large Send Offload v2 (IPv6)"
- Get-VmSwitch | Set-VMSwitch -EnableSoftwareRsc $false
-
@blcktape That was already in place. This issues was different than the 100% slow down all the time those settings fix.
