TNSR with ESXI VMXNET3 -- not recognized

jantypas

I've loaded TNSR on ESXI 6.7 with the VMWare tools. The network drivers are all set for VMXNET3. When TNSR boots, I get my management interface and its address. I then try:

config terminal
dataplane dpdk dev <PCI address of card 2> network
dataplane dpdk dev <PCI address of card 3> network
service datplane restart
exit

When I do a show interfaces, nothing appears. What uio-driver should I be using?

MalnPr0

Try removing any settings you put in the dataplane dpdk section (devices, uio-driver, etc..), save your config and restart the dataplane. Then set the uio driver to vfio-pci (dataplane dpdk uio-driver vfio-pci). After doing so, add your DPDK network devices, commit your config, and restart the dataplane. If the interfaces still do not appear, verify the vfio-pci module is loaded by running this command from the shell: lsmod | grep vfio_pci

If it's not loaded, try loading the module from the shell: modprobe vfio_pci. If you receive an error, try modprobe vfio (you may need to load vfio_pci after this).

jantypas

Re: TNSR with ESXI VMXNET3 -- not recognized

Per your suggestion, I did that -- and I see the vfio_pci driver loading, but it complains it cannot probe the two VMXNET3 PCI devices (Error -22)

MalnPr0

I booted up a VM I have running on ESXI 6.7:
VMXNET 3
Status: Connect At Power On
DirectPath I/O: Enabled
Guest Managed VMware Tools: 10309 (openvm-tools v10.2.5)

The network interfaces allocated for TNSR are set to use the uio_pci_generic driver.

Can you try the following:

Verify the sysconfig network-script for the interfaces that you want to allocate for TNSR are marked with:

ONBOOT=no
NM_CONTROLLED=no

Example:
pci@0000:0b:00.0 | GigabitEthernetb/0/0 | ens192
sudo vi /etc/sysconfig/network-scripts/ifcfg-ens192
(append the ONBOOT and NM_CONTROLLED parameters to the bottom of the file if not present)

Set the uio-driver back to uio_pci_generic (save your config to startup after this)

Once this is done, reboot the system and see if the interfaces are allocated and accessible. If they are not, try adding this to /etc/vpp/startup followed by a restart of the VPP service from the shell:

no-tx-checksum-offload
socket-mem 128

If you still have issues, please provide the output from the following shell commands:

lshw -class network | grep -A 4 VM
sudo vppctl show hardware
sudo vppctl show pci

Dsploitar

Dsploitar

@MainPr0 Still not taking

talwell

@Dsploitar - Did you ever get this working? I have tested with ESXi and standard VPP & DPDK and it was a little tricky to get the VMXnet3 driver to load and bind but if you need help I would be glad to assist.

A Former User

This post is deleted!

Dsploitar

Thanks to all. It's been a while since I was working with this. We ended up using virtualized pfSense instead with pci passthrough for the NIC. Works great with Suricata, ServiceWatchdog, OpenVPN, and firewall rules.

We believed TNSR would be better for a WAN perimeter, but NIC drivers on TNSR were not present or available, and that came straight from Netgate engineers. We even tried to load them ourselves but found we would have to rewrite them to work with TNSR and we were not in the business of doing that as we had a time constraint and pfSense worked well. They were BCM **** NIC/drivers.

So the solution at one point was to sandwhich a PCIpassthrough pfSense on the WAN side of TNSR to scave off Intel csme + spectre/meltdown vulns since TNSR did not have NIC driver, and that is where this thread started at. It was a kludge and still dont know why that VMX3net would not load, but we implemented a more eloquent solution in the end.

We are still considering TNSR, mainly because it does seem more scalable than pfSense, and there is strong IPSec/StrongSwan documentation, which there is no way in hell we can get that to work with pfSense with all documentation, with Windows or Linux clients. Very odd. OpenVPN is better anyway overall, but some clients are hung on IPSec for some reason. Only reason I can fathom is that it comes stock on Windows which is a quite silly reason in my opinion.

Thanks again to all

Dsploitar

@leoan we tried that at one point and it loaded successfully, but as shown in my screenshot only the virtual NIC would load properly.

I initially named this thread wrong, not realizing the NIC naming convention came from the driver. Our VMX3 virtual side loaded just fine, it was the PCIpassthrough of a BCM driver that is not currently implemented or supported in TNSR.

Thanks for your help.