[solved] pfSense (2.6.0 & 22.01 ) is very slow on Hyper-V

Bob.Dig

Today tried changing settings wildly in Windows but nothing changed. Again, went back.

2.5

hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	capabilities=48071b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,TSO6,LRO,LINKSTATE,TXCSUM_IPV6>
	ether 00:15:5d:8c:c0:1f
	inet6 fe80::215:5dff:fe8c:c01f%hn0 prefixlen 64 scopeid 0x5
	inet6 2003:da:a718:4300:215:5dff:fe8c:c01f prefixlen 128
	inet 172.25.0.2 netmask 0xffffff00 broadcast 172.25.0.255
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

hn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	capabilities=48071b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,TSO6,LRO,LINKSTATE,TXCSUM_IPV6>
	ether 00:15:5d:8a:c0:20
	inet6 fe80::215:5dff:fe8a:c020%hn1 prefixlen 64 scopeid 0x6
	inet6 fe80::1:1%hn1 prefixlen 64 scopeid 0x6
	inet6 2003:da:a718:43f0:215:5dff:fe8a:c020 prefixlen 64
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	groups: GroupNTP GroupDNS
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

2.6

hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	capabilities=48071b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,TSO6,LRO,LINKSTATE,TXCSUM_IPV6>
	ether 00:15:5d:8c:c0:1f
	inet6 fe80::215:5dff:fe8c:c01f%hn0 prefixlen 64 scopeid 0x5
	inet6 2003:da:a718:4300:215:5dff:fe8c:c01f prefixlen 64 autoconf
	inet 172.25.0.2 netmask 0xffffff00 broadcast 172.25.0.255
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

hn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	capabilities=48071b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,TSO6,LRO,LINKSTATE,TXCSUM_IPV6>
	ether 00:15:5d:8a:c0:20
	inet6 fe80::215:5dff:fe8a:c020%hn1 prefixlen 64 scopeid 0x6
	inet6 fe80::1:1%hn1 prefixlen 64 scopeid 0x6
	inet6 2003:da:a718:43f0:215:5dff:fe8a:c020 prefixlen 64
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	groups: GroupNTP GroupDNS
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	supported media:
		media autoselect
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Only thing I've seen so far is this post on reddit:
https://www.reddit.com/r/PFSENSE/comments/ssabdz/pfsense_plus_version_2201_and_pfsense_ce_version/hwznlap/

Which suggests it's an issue with the VMQ support that is now functioning in 2.6.
However it looks like several of you have tried running SR-IOV or were already with no improvement?

Steve

DonZalmrol

Hi @stephenw10 just checked on my Hyper-V PFSense guest SR-IOV has failed for my interfaces. I'm quite sure it worked without any issues when I set up my installation from scratch.

Is there a way to reactivate it?

FYI: Disabling SR-IOV, VMQ has no effect on the speed.

stephenw10

Not in pfSense. I have little to no experience in Hyper-V so I can't really help with that.

Steve

mxkied2

The nics I am using for pfsense (Intel i210) do not support VMQ or SR-IOV, not sure if that information helps at all but I thought I would mention it at least.

mxkied2

@stephenw10 I am not running vlans but I do have snort running so that might explain the promiscuous mode.

stephenw10

Yup, that probably would. There is clearly something else at work here though and it appears to be beyond just the NIC/driver.

RMH 0

I have same issue as covered here:

Win 2019 Server running Hyper-V

All Ok on 2.5.2 moved to 2.6 and network speed dropped from, 500mb to 2.5mb. I did all the changes suggested here with VMQ etc. Also swapped WAN interface from Intel Pro 1000 PT Dual NIC to onboard Marvell Yukon 88E8059 and no difference.

What did weirdly make a massive difference was using a VPN on my PC, I started NordVPN and my speed jumped back to 488mb. Strange thing was in pfsense it showed my WAN speed as 60mb. Dropping the VPN and speed dropped back bown to a couple of MB.

Essentially the connection via pfsesne 2.6 runs at normal speed if your client is conencting through it on a VPN. All be pfsense thinks it is running at nearly 10x a slower speed.

stephenw10

TCP vs UDP maybe? Some hardware off-loading happening?

Do you see error in the interfaces when it's slow?

Dominixise

@stephenw10

Hi Steve I can setup a zoom session with you on my server unless that requires paid tech support from netgate?

Dom

stephenw10

I'm not the guy for that, I know next to nothing about hyper-v.

Dominixise

packetcapture.zip @stephenw10

Could be a Microsoft error too, I remember just after Christmas there was an windows update that cause some Hyper-V errors and maybe some are still present. My Server is up to date.

But heres a packet capture of a speed test packetcapture.zip

Dominixise

@dominixise
Also pftop while doing a speed test dom pfsense pftop.jpg

stephenw10

No actual speedtesting shown there, was it filtered?

All small packets, some MTU issue?

Steve

Dominixise

@stephenw10
Sorry i am new to getting logs, try this onepacketcapture (1).zip

stephenw10

Nearly!

You probably want to filter by the client IP you are running the test on, if you're capturing on LAN. And set the capture to, say, 5000 packets.

Steve

Dominixise

@stephenw10

Okay thanks for the acknowledgment here is the new capture, I had to put the download on my webserver for download since its 5MB

https://zebrita.publicvm.com/files/packetcapture(2).cap

Dominixise

@dominixise
Here is another one with just my host ip

https://zebrita.publicvm.com/files/packetcapture(3).cap

RMH 0

A bit of digging and it looks like 2 issues to me.

One in Hyper-V which I have now got resolved, fix below (well for me anyhow)
One in pfsense that is missreporting throughput (I can live with that till a fix comes)

For Hyper-V I found this article on RSC https://www.doitfixit.com/blog/2020/01/15/slow-network-speed-with-hyper-v-virtual-machines-on-windows-server-server-2019/
Once I disabled RSC on all virtual switches my speed was back to normal. No restart needed, just go on to Hyper-V host, open powershell and input commands to disable RSC on each virtual switch.

These are commands I used

Get-VMSwitch -Name LAN | Select-Object RSC
Checks status, if true run next command LAN is my vswitch name

Set-VMSwitch -Name LAN -EnableSoftwareRsc $false
This disables RSC, re run first command to confirm it is disabled

If your vSwitch has a space in the name add "" around the name
Get-VMSwitch -Name "WAN #1" | Select-Object RSC

After applying speed is back to normal but pfsense seems to top out showing throughput at 60mb, even though I was getting over 500mb.

Anyhow, hope it helps thers on Hyper-V (this is a 2019 instance of Hyper-V)

DonZalmrol

@rmh-0 fantastic find!

I can confirm this has resolved it for me to, I'll leave it as is until a fix comes out.

Speed with RSC enabled:

Speed with RSC disabled: