CARP with SR-IOV enabled NIC under Hyper-V
-
Should CARP work with a SR-IOV enabled Intel Nic under Hyper-V 2016?
I have multiple interfaces - most work with carp as they should, but not the SR-IOV enabled external carp enabled IPs.
carp IP Status = MASTER, but no traffic reach the pfsense
2.4.4-RELEASE-p3 (amd64)
Hyper-V 2016
Intel I350 -
So I've just come up against this myself and was going to post my findings, figure I'll post them here instead of starting a new thread - it sounds like my issue is related.
I've just clean installed PFSense 2.5 on two of my hypervisors, intending to set them up for High Availability to replace an old cisco router that I don't want to spend the money to replace.
Both hypervisors are running Hyper-V 2019 with Intel X520 10gb nics - on one of them I'm running the Windows Server NIC teaming in an Active/Standby configuration - 10gb link active, 1gb standby - as you may be aware this configuration does not allow SR IOV to work.
On the other I'm using the newer 'Switch Embedded Teaming' mode which allows SR IOV, with 10gb/10gb active/active (switch capacity is the reason I'm only running this on one hypervisor at the moment, SET mode prevents an Active/Standby setup)
On both PFSense virtual machines I set the (virtual) NICS to enable MAC Spoofing, which as I understood from the documentation should be all that was required for such a configuration to work on Hyper-V.
In my testing, having set up the CARP virtual IP addresses, what I found was that on the SET teamed hypervisor, I couldn't contact it on the CARP ip address - it could ping itself on this address but nothing else could, which suggested some kind of connectivity issue rather than a setup issue (especially as it worked fine on the other hypervisor with the synced configuration when I failed over). The other hypervisor worked as expected.
Looking at the SET team documentation, there is mention here that the MAC address is overwritten when the virtual switch is in dynamic load balancing mode:
https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v-virtual-switch/rdma-and-switch-embedded-teaming#bkmk_macI'm not certain that I created the switch in Dynamic mode although it seems fairly likely - so I switched the mode across to Hyper V Port distribution, which specifically states there "No source MAC replacement is done" - but still no joy (I've not rebooted the hypervisor, it's possible that it's required to change the mode across but I would have thought unlikely).
Next thing I read that 'Receive Segment Coalescing' has caused some people problems in Hyper-V 2019, so I turned that off for the VSwitch, no joy there either (https://docs.microsoft.com/en-us/windows-server/networking/technologies/hpn/rsc-in-the-vswitch) - as an aside, this is still enabled on the other hypervisor and no issues with it there.
Now, one point to make is that enabling MAC spoofing prevents your virtual NICS from using SR IOV or VMQ acceleration, which is rather far from ideal in any case, as with heavy traffic usage this will at best increase CPU usage, and at worst limit the throughput.
So I'm not sure what is the best way forward now. I could just remove the switch embedded teaming from this hypervisor and drop back to the windows NIC teaming that I've been using previously, although in principle switch embedded teaming is a better option as it permits SR IOV and so should improve my network performance for the other virtual servers on this hypervisor - my broad plan had been to increase switch capacity and bring all my hypervisors across to SET.
-
So to relate this back to the original post, I took one of the Intel X520 ports out of the SET team to do some testing.
Creating a new virtual switch with SRIOV enabled using that NIC but no teaming, the symptoms persist - CARP on the virtual machine doesn't work.
I then destroyed that virtual switch, created another one with SRIOV disabled, and CARP works.
Now as it happens, when I created the virtual switch on the other hypervisor linked to the Windows NIC team, SRIOV was enabled but can't work due to the NIC teamed adapter not supporting it.
So fundamentally the 'Enable MAC Address Spoofing' feature is broken on a virtual switch with SR-IOV enabled.
-
Oh, and just for clarity, the links to the X520 network cards are trunk 802.1Q, with VLANS being assigned to PFSense via the "Enable virtual LAN identification" option on it's virtual NICs.
-
So perhaps to bring this to a close, further reading suggests that this might be a hardware limitation of most SR IOV capable network adapters (and a security consideration for hosted environments, where obviously they would not want clients to be able to packet sniff data from other clients).
In this blog post someone writes about their love for the Intel X710 and XL710 specifically because they actually support promiscuous mode with SR IOV where the great majority of network cards don't:
https://www.sealingtech.com/2018/07/30/sr-iov-and-promiscuous-mode/And I've not found any source that clearly states if, even with these cards, the spoofed mac addresses + sr iov scenario works on Hyper-V.
Would love to hear if anyone is using these cards in such a setup successfully!
I'll start a new post with my specific conundrum.
-
Many thanks for this useful information.
I didn't think that this could be a limitation of the network driver or the adapter. -
@hege Late reply on this topic, but relevant.
Hyper-V SR-IOV implementation does NOT support mac spoofing with SR-IOV
Technical;
-
Mac spoofing is required for CARP because the mac address is changed on outbound packets, that's part of CARP.
-
Hyper-V natively does not allow outbound packets through the virtual switch from a Hyper-V guest that does not have the exact same mac address as assigned to the virtual machine (unless you enable the "allow mac spoofing" checkbox.
-
SR-IOV technically can allow mac spoofing, this is all there in the IEEE specification for for this is to work, but, quite simply Microsoft Hyper-V doesn't implement it.
Therefore you need to enable "allow mac spoofing" and forego SR-IOV or VMQ network accelerate functions.
-
