TNSR with ESXI VMXNET3 -- not recognized



  • I've loaded TNSR on ESXI 6.7 with the VMWare tools. The network drivers are all set for VMXNET3. When TNSR boots, I get my management interface and its address. I then try:

    config terminal
    dataplane dpdk dev <PCI address of card 2> network
    dataplane dpdk dev <PCI address of card 3> network
    service datplane restart
    exit
    

    When I do a show interfaces, nothing appears. What uio-driver should I be using?


  • Netgate Administrator

    Try removing any settings you put in the dataplane dpdk section (devices, uio-driver, etc..), save your config and restart the dataplane. Then set the uio driver to vfio-pci (dataplane dpdk uio-driver vfio-pci). After doing so, add your DPDK network devices, commit your config, and restart the dataplane. If the interfaces still do not appear, verify the vfio-pci module is loaded by running this command from the shell: lsmod | grep vfio_pci

    If it's not loaded, try loading the module from the shell: modprobe vfio_pci. If you receive an error, try modprobe vfio (you may need to load vfio_pci after this).



  • Re: TNSR with ESXI VMXNET3 -- not recognized

    Per your suggestion, I did that -- and I see the vfio_pci driver loading, but it complains it cannot probe the two VMXNET3 PCI devices (Error -22)


  • Netgate Administrator

    I booted up a VM I have running on ESXI 6.7:
    VMXNET 3
    Status: Connect At Power On
    DirectPath I/O: Enabled
    Guest Managed VMware Tools: 10309 (openvm-tools v10.2.5)

    The network interfaces allocated for TNSR are set to use the uio_pci_generic driver.

    Can you try the following:

    1. Verify the sysconfig network-script for the interfaces that you want to allocate for TNSR are marked with:
    ONBOOT=no
    NM_CONTROLLED=no
    

    Example:
    pci@0000:0b:00.0 | GigabitEthernetb/0/0 | ens192
    sudo vi /etc/sysconfig/network-scripts/ifcfg-ens192
    (append the ONBOOT and NM_CONTROLLED parameters to the bottom of the file if not present)

    1. Set the uio-driver back to uio_pci_generic (save your config to startup after this)

    Once this is done, reboot the system and see if the interfaces are allocated and accessible. If they are not, try adding this to /etc/vpp/startup followed by a restart of the VPP service from the shell:

    no-tx-checksum-offload
    socket-mem 128
    

    If you still have issues, please provide the output from the following shell commands:

    lshw -class network | grep -A 4 VM
    sudo vppctl show hardware
    sudo vppctl show pci
    


  • Screenshot from 2019-12-10 10-11-17.png Screenshot from 2019-12-10 10-10-46.png



  • @MainPr0 Still not taking



  • @Dsploitar - Did you ever get this working? I have tested with ESXi and standard VPP & DPDK and it was a little tricky to get the VMXnet3 driver to load and bind but if you need help I would be glad to assist.



  • If it's not loaded, try loading the module from the shell: modprobe vfio_pci. If you receive an error, try modprobe vfio (you may need to load vfio_pci after this). The network interfaces allocated for TNSR are set to use the uio_pci_generic driver.
    myschoolbucks



  • Thanks to all. It's been a while since I was working with this. We ended up using virtualized pfSense instead with pci passthrough for the NIC. Works great with Suricata, ServiceWatchdog, OpenVPN, and firewall rules.

    We believed TNSR would be better for a WAN perimeter, but NIC drivers on TNSR were not present or available, and that came straight from Netgate engineers. We even tried to load them ourselves but found we would have to rewrite them to work with TNSR and we were not in the business of doing that as we had a time constraint and pfSense worked well. They were BCM **** NIC/drivers.

    So the solution at one point was to sandwhich a PCIpassthrough pfSense on the WAN side of TNSR to scave off Intel csme + spectre/meltdown vulns since TNSR did not have NIC driver, and that is where this thread started at. It was a kludge and still dont know why that VMX3net would not load, but we implemented a more eloquent solution in the end.

    We are still considering TNSR, mainly because it does seem more scalable than pfSense, and there is strong IPSec/StrongSwan documentation, which there is no way in hell we can get that to work with pfSense with all documentation, with Windows or Linux clients. Very odd. OpenVPN is better anyway overall, but some clients are hung on IPSec for some reason. Only reason I can fathom is that it comes stock on Windows which is a quite silly reason in my opinion.

    Thanks again to all



  • @leoan we tried that at one point and it loaded successfully, but as shown in my screenshot only the virtual NIC would load properly.

    I initially named this thread wrong, not realizing the NIC naming convention came from the driver. Our VMX3 virtual side loaded just fine, it was the PCIpassthrough of a BCM driver that is not currently implemented or supported in TNSR.

    Thanks for your help.


Log in to reply