pfSense SR-IOV support on Microsoft Hyper-V
-
I've got a couple of customers running pfSense on Hyper-V (Microsoft Hyper-V Server 2016) - the free version.
It works great and it was very simple to get pfSense 2.4.4-p3 working, next, next, next, next, done!
Thee customers have Intel 10 Gbe cards and SR-IOV running on Hyper-V today. SR-IOV is awesome and works well, reducing host CPU utilization and speeding up networking because SR-IOV bypasses the Hyper-V virtual switch communicates directly with the network card.
[https://docs.microsoft.com/en-us/windows-hardware/drivers/network/overview-of-single-root-i-o-virtualization--sr-iov-]A quick google search shows some people have had success with SR-IOV and FreeBSD so it must be actually possible.
Drives that I'd like to see are;
- Intel X710
- Intel X722
Are there any plans for pfSense to bake in SR-IOV support?
-
@nzkiwi68 I think it's Hyper-V that doesn't support SRV-IO on FreeBSD.
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-freebsd-virtual-machines-on-hyper-v#table-legendFor my own info, what version of Hyper-V server is free?
EDIT - Huh, how about that...
https://www.techrepublic.com/article/getting-started-with-microsoft-hyper-v-server-2016/ -
That’s a really old article from Microsoft, dated 30 Aug 2017 and I’d be quite surprised if FreeBSD doesn’t have SR-IOV support. A quick google search suggests FreeBSD does work with SR-IOV and hyper-v.
As for free hyper-v...
Hyper-v free edition server 2016 or 2019, both are easily available and totally free. -
@nzkiwi68 So I see now. Glad I read. Too bad my antique doesn't have SLAT.
-
From Intel, June '19. I think your gripe is with FreeBSD and not pfSense.
-
Well, that's not looking very promising!
Thanks though for your post, upvote for you!
-
What iperf speeds are you getting without SR-IOV? Can you use VMQ?
If your hardware supports it you could probably pass a NIC right through to the VM with DDA.
-
@bjd223 I couldn't pass the NIC through, as, I normally use 2 x 10 Gbe NIC setup with Microsoft SET (Switch Embedded Teaming) which since Server 2016 allows SR-IOV through to the NIC.
That way I am protected from a single NIC failure, and, because of SET we have double the VF (virtual function slots) for the VM to use with SR-IOV.
It's not just about speed, but, a lot lower CPU.
Consider 2 VMs on two different hosts talking to one another;
-
without SR-IOV
vm > virtual switch > physical NIC > across network switch > physical NIC > virtual switch > vm -
with SR-IOV
vm > physical NIC > across network switch > physical NIC > vm
So there's 2 lots of CPU savings to be made, because the virtual switch is all software/CPU cycles. Sure it's efficient, but it still adds overhead and takes CPU cycles.
SR-IOV makes a lot of sense.I notice FortiGate firewall VM have a requirement for SR-IOV for their implementation.
https://docs.fortinet.com/document/fortigate/6.0.0/fortigate-vm-on-kvm/553137/sr-iov
https://docs2.fortinet.com/document/fortigate/6.0.0/fortigate-vm-on-vmware-esxi/553137/sr-iov -
-
@nzkiwi68 Yes I understand the performance benefits on SR-IOV. However if it is not supported in the version of FreeBSD that pfsense is using then it will not work.
Your next best choice is VMQ and if you are lucky you will fall under tier 2 which is maximum offload for VMQ.
-
@bjd223 yep. Doesn't change the fact I'd still love to see SR-IOV support...
Thanks for your comments.
-
FWIW (and worth every penny paid), here's a H-V tuning article from Altaro.
https://www.altaro.com/hyper-v/hardware-tweaks-hyper-v-performance/ -
@nzkiwi68 If you look at the FreeBSD Wiki https://wiki.freebsd.org/HyperV which was last updated on 10/04/19, it implies that the article will be updated as new info is available, so I think that article reflects the current state of affairs.
I think the integration drivers are contributed to FreeBSD primarily by MS themselves. So I doubt the pfsense devs want to get involved upstream of the pfsense project (or can even do it to begin with since it is probably very complicated and may also need updates to Hyper-V itself).
I know that pfsense 2.5 is supposed to be based on FreeBSD 12, which if you are lucky will have updated integrations which include SR-IOV.
-
@bjd223 Thanks, it will be interesting to see what FreeBSD 12 brings.
-
Just a small note of caution for anyone looking at this again now that we have PFSense 2.4.5 stable - be extremely cautious using SR IOV with PFSense under Hyper-V
My experience this week is that if you make SR IOV available to it, it does appear to work - but causes big glitches that can pretty much bring down your hypervisor.
Obviously the particular driver/hardware in question will have an effect but with my X520 nic and SR IOV pfsense would hang on shutdown and fail to turn off, and nothing I did (killing processes on the hypervisor etc) would kill it off, or bring it back up.
I ended up in a real mess as when I rebooted the hypervisor it would automatically start the PFSense box again, with SR IOV enabled, and I'd be stuck - had to use powershell to turn off the automatic boot of the pfsense vm and reboot everything again.
-
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/supported-freebsd-virtual-machines-on-hyper-v
Updated 7 April 2020 and clearly FreeBSD 11.0, 11.1-11.13 and 12.-12.1 are supported for SR-IOV.
Feature Windows Server OS 12-12.1 11.1-11.3 11.0 SR-IOV 2019, 2016 ✔ ✔ ✔I expect then pfSense 2.4.5 based on FreeBSD 11.3 should work well with SR-IOV.
Does anyone have 2.4.5 running SR-IOV?
-
@nzkiwi68 With 2.4.5 utterly broken in multi-core virtualized environments, it's hard to say. I won't be able to upgrade my environment until this pfctl issue is fixed, which means waiting another 6-12 months for a 2.4.5-p1.
One step forward, one step back.
