Error with SR-IOV on pfSense using x710 NIC

zachj

Configuration Details:

Intel x710-T4 NIC running NVM firmware 9.4
NOTE: I only just recently upgraded to NVM 9.4; the same behavior happened on the old firmware too (whatever version is included with Dell v19.5.0)
vSphere 8.0 U2 running the matched i40en driver (2.7.2.0-1OEM.800.1.0.20613240) as per the VMware hardware compatibility list
Each of the 4 physical ports is configured with a single virtual function, on which I have disabled "Spoof Check" and I have enabled "Trusted" mode in ESXi using the following command:
esxcli intnet sriovnic vf set -n vmnic3 -v 0 -s false -t true
pfSense 2.7.2 CE clean install on a brand new VM
pfSense is using the 3.0.26-k in-box kernel driver for x710 SR-IOV virtual network devices
- NOTE: for what it's worth I think the in-box kernel driver is one revision behind Intel's latest and isn't actually the matching driver for NVM 9.4

The pfSense VM sees all 4 virtual functions without issue and automatically loads the in-box 3.0.26-k kernel driver

On boot and also whenever I make interface changes I get the error (where X is the interface number):

iavfX: iavf_vc_completion: AQ returned error VIRTCHNL_ERR_PARAM to our request CONFIG_RSS_KEY!

Does anyone got any brilliant ideas what's wrong or what to do to fix?

stephenw10

What NVM version did you upgrade from? Did that show the same error?

Does it actually fail to pass traffic?

zachj

@stephenw10 the previous firmware version was NVM 7.10.

No it doesn’t fail to pass traffic, although I’m having some weird issues where my VMs don’t seem to be consistently getting DHCP addresses.

I haven’t yet decided if that weirdness is related or is operator error.

I googled the error message and I can’t find anyone reporting the same or even a similar problem. All I could find was the source code that emits the error, but I didn’t spend much time trying to read that.

stephenw10

There's this but it's unhelpful!
https://forum.netgate.com/topic/172042/sr-iov-under-vmware-7-unable-to-config-rss-key

Other than showing this is not a new issue.

zachj

@stephenw10 Serves me right for searching using exact string in quotations...but agreed it's not helpful beyond confirming it's been a problem for a while and agreeing it doesn't seem to stop traffic from flowing.

Is it possible the reason it's happening is because ESXi keeps toggling "trusted" mode off and "Spoof Check" on during every VM boot and so I have to run the above esxcli command to reset them back to the desired values (Trusted=On & Spoof=Off) after the VM boots which foces the VFs to reset?

https://kb.vmware.com/s/article/74909

That KB was last updated in 2019 so it's not clear if this is still expected behavior, but as per the KB it was expected at some time in the past.

zachj

Regarding the fact that trusted/spoof check settings don't persist across VM reboots (https://kb.vmware.com/s/article/74909) I have opened a new discussion on the Intel Communities site in hopes that somebody from Intel support will see it and engage...

SR-IOV Virtual Function spoof check not persistent across VM reboots

zachj

After RTFM I was able to stop the Intel i40en driver from disabling trusted mode on the virtual functions on every reboot, but that didn't resolve the error message. The good news is now the firewall doesn't come up from a boot in a broken state every time, but this confirms the AQ returned error VIRTCHNL_ERR_PARAM to our request CONFIG_RSS_KEY! error is unrelated to the i40en driver's handling of trusted mode.

Anyone else have any brilliant ideas for what could be causing the error message?

Anyone have any guess as to whether it can safely be ignored? It doesn't stop pfSense from passing traffic on the NIC so it seems to "work," but God forbid it's creating some sort of security vulnerability!