@dabernie said in TNSR Home+Lab ISO on KVM with Mellanox ConnectX-4 passthrough:

where did you get your qcow2 image ?

Although the OP has it on an "images" folder, that refers to the Virtual Machine disk. The actual image is an .iso.

@dabernie Maybe a little more information - what error exactly are you seeing? What bare metal system is it?

I got tnsr-v20.10.1-2 working within virtualization with Intel XXV710-DA2. This still faced some challenges:

upgrading firmware to 7.30 as suggested to match tnsr's DPDK version here wouldn't work from within virtualization. I had to put the card in a bare hardware machine or else ./nvmupdate64e found the card but showed "Access Error" on the RHS of the card table. After downloading Intel's firmware tool, a.k.a. "NVM", for old revision 7.30, the tool refused to touch the card at first, "no update available" while showing version "6.128(6.80)." The documented versions of intel i40e firmware seem to correspond to the "hex" version in parenthesis shown in nvmupdate64e, yet as is their typical style they needlessly show both. stfw showed there are a lot of OEM cards that Intel tries to force you to the OEM's payware service plans for updates, but closely reading Intel's docs 4.0, 'ethtool -i <device>' reveals the "EtrackID" as the second field of "firmware-revision", a hex number like 0x8000xxxx. Adding this number to the REPLACES: field of nvmupdate.cfg of a similar card (good luck!) will force the update to go through anyway. Intel's MAC is picky about SFP+ modules. A module with Cisco srom worked. A Dell module that works fine in ConnectX-3 En didn't work. Updated 7.30 firmware printed a dmesg warning about disliking the module on each insertion, but older 6.80 firmware silently showed link down, IIRC.

For me, my bias/impression that the Intel parts would be overcomplicated and buggy wrt Mellanox was confirmed. There could be something subtly wrong with my virtualization config, or something I can't even think of, blocking the ConnectX-3 and ConnectX-5 from working, but partially arguing against that at least I can confirm Intel XXV710-DA2 works with TNSR in a controlled situation where Mellanox parts don't.

@kiokoman

thanks a lot i reupload the iso and it's working now :

14193c8b-656e-4f23-bcf3-3437a49c377b-image.png

Sorry for the noise :/

@venix91 Hyper-V is not listed here:

https://docs.netgate.com/tnsr/en/latest/platforms/

That is the latest guidance.

@atoonk

kiokoman@nanto:/var/www/html/tnsr$ ls -a . .. BaseOS .discinfo EFI images isolinux ks.cfg media.repo TRANS.TBL .treeinfo

ks.cfg : <- there is probably unused / not needed stuff here, it's your work to figure out, i just copy/paste from other sources and adjusted the url

kiokoman@nanto:/var/www/html/tnsr$ cat ks.cfg firewall --disabled install url --url="http://tftp.kiokoman.home/tnsr" rootpw --iscrypted /hNTxhbZeFodHAO.D9uC. auth useshadow passalgo=sha512 text keyboard us lang en_US timezone America/Detroit bootloader clearpart --all --initlabel part swap --asprimary --fstype="swap" --size=1024 part /boot --fstype xfs --size=200 part pv.01 --size=1 --grow volgroup rootvg01 pv.01 logvol / --fstype xfs --name=lv01 --vgname=rootvg01 --size=1 --grow %packages @core %end %post %end

copy from isolinux directory ->

kiokoman@nanto:/tftp/tnsr$ ls initrd.img vmlinuz kiokoman@nanto:/tftp$ cat pxelinux.cfg/default default vesamenu.c32 prompt 0 timeout 300 ONTIMEOUT local menu title ########## PXE Boot Menu ########## other stuff other stuff other stuff .... label 6 menu label ^6) TNSR kernel tnsr/vmlinuz append initrd=tnsr/initrd.img inst.repo=http://nanto.kiokoman.home/tnsr/ ks=http://nanto.kiokoman.home/tnsr/ks.cfg

Immagine2.jpg

shell sudo dnf install libibverbs

That work?

the other network interface are hidden and you can't use it under centos if they are assigned to the dataplane
to manage the route of the dataplane you need to do it with clixon_cli ->
https://docs.netgate.com/tnsr/en/latest/routing/manage.html

example (i'm using random ip here just to show you):

sudo /usr/bin/clixon_cli config route ipv4 table ipv4-VRF:0 route 172.18.0.0/24 next-hop 0 via 192.168.150.2 exit exit show route

Immagine.jpg

How are they connected? They don't appear to be able to exchange traffic between each other. Nothing but transmit errors on WAN there.

I have felt some of the same pain with the ACL config being a little bulky. On the bright side, it's actually made me plan ACLs out a bit better and forced me into having many ACLs with fewer rules each. That being said, I miss being able to see a rule as a single line sometimes.

Hi,

So, that does work, it was even set automatically to dataplane with the default driver.

labtnsr1 tnsr# show configuration <dataplane-config xmlns="urn:netgate:xml:yang:netgate-dataplane"> <dpdk> <uio-driver>igb_uio</uio-driver> </dpdk> </dataplane-config> <nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm"> <enable-nacm>true</enable-nacm> <read-default>deny</read-default> <write-default>deny</write-default> <exec-default>deny</exec-default> <enable-external-groups>true</enable-external-groups> <groups> <group> <name>admin</name> <user-name>root</user-name> <user-name>tnsr</user-name> </group> </groups> <rule-list> <name>admin-rules</name> <group>admin</group> <rule> <name>permit-all</name> <module-name>*</module-name> <access-operations>*</access-operations> <action>permit</action> </rule> </rule-list> </nacm> labtnsr1 tnsr# show interface Interface: GigabitEthernetb/0/0 Admin status: down Link down, link-speed 10 Gbps, full duplex Link MTU: 9000 bytes MAC address: 00:50:56:ba:ed:31 IPv4 MTU: 0 bytes IPv4 Route Table: ipv4-VRF:0 IPv6 MTU: 0 bytes IPv6 Route Table: ipv6-VRF:0 VLAN tag rewrite: disable Rx-queues queue-id 0 : cpu-id 1 counters: received: 0 bytes, 0 packets, 0 errors transmitted: 0 bytes, 0 packets, 0 errors protocols: 0 IPv4, 0 IPv6 0 drops, 0 punts, 0 rx miss, 0 rx no buffer labtnsr1 tnsr#

Since there was a difference in the way I did the install, I retried with 6.7 compatibility again, and fails again.

Basically, this time I started the installation already with both network cards set in vmware, and this wasn't the case in the previous problematic installation.

But it's confirmed that was not this the issue, it failed again now with 6.7 and both cards available to the installer.

So, I don't know if this is a bug or not, but for centOS8 based OS, would make sense to support 6.7 already.

Thank you for your help, will continue the tests now.

@JeGr My edge device on the 10G is currently an XG-2758. Utilizing one of the igb ports for host management and the two ix ports for outside/inside.vlan.

That's at the OS level which is essentially CentOS and the Anaconda installer. We don't have any documented procedure to do that as far as I'm aware, but if you can find instructions for CentOS 8 the same type of procedure should apply.

Can you post exactly what you are seeing?

What is the exact NIC you are trying to use?

These are the Broadcom NICs on the compatibility list but they have received no in-house testing.

We would initially recommend Intel or Mellanox NICs as they have received the most testing in-house and are likely the path of least friction.