Multicore forwarding

[oudenos]

@derelict How can I check that?

[Derelict]

@oudenos Something like this might help you map your system:

apt-get update
apt-get install hwloc
lstopo --output-format png > ~tnsr/lstopo.png

Then scp that image off and view it with your preferred method.

[oudenos]

@derelict Thank you for your help and sorry for the delay.
As you correctly pointed out, the NIC is owned by the "wrong" CPU. However the same happens with the Intel one. Now I'm getting inconsistent results across reboots which lead me to think some sort of receive side scaling is involved. I'm working to set up a more realistic testbed with T-Rex as traffic generator and will also address NUMA pinning. I will get back to you as soon as I have results worth sharing with the community.

[LukeCage]

I'm following the thread

I'm looking forward to the results
good work

[oudenos]

@lukecage unfortunately, there is not much to share.
I noticed that changing the number of queues at runtime often requires to reboot both the VM (TNSR) and the hypervisor (Ubuntu 20.04 + KVM) to work properly, otherwise many packets will get lost. This happens with both the Mellanox and the Intel, though the first one seems to be "more affected". No idea why, I suspect the only way to correctly reinitialize the NIC is to power-cycle it, maybe I did something wrong with KVM and stuff. (*)
Also, I'm pretty sure both NICs use Receive Side Scaling (RSS) and I had to change my traffic generator to TRex in order to have more entropy. I tried this on an Intel E810 card @ 100 Gbps, but it doesn't use scale to more than one CPU core. Again, I believe some sort queue-ish thing went wrong, perhaps I should try with the latest versions of VPP, DPDK, driver and firmware, but it requires building a TNSR-like distro from scratch afaik.

(*) If someone is willing to give it a try, it may be worth starting with bare metal rather than KVM: DPDK uses hugepages and KVM by default only supports 2MB hugepages, not 1GB.

Eventually, I dediced to give up on this for the moment. It's a shame, but I don't have enough time do all the tests.

Thank you very much to the community and @Derelict for the help they provided.

[LukeCage]

@oudenos which cpu you using ?

my goal is 38m pps on gre tunel (i cant test more because my upstream dont allowed)

via interface 98m pps

and why do you need to run multiple cores, tnsr performance is fine

[oudenos]

@lukecage Intel Xeon Silver 4210R
With proper NUMA pinning I can achieve 5.5 Mpps IP forwarding per core. Splitting in multiples queues should enable multicore processing.

How did you reach those numbers without tuning??

[LukeCage]

@oudenos

I use it directly as bare metal
not via vmware or any virtualization

As a result, you are positioning a router, if you need high capacity, you should install it tnsr directly to the server.

my hardware specs;

i9-9900k
32gb ram
240gb ssd

uptime
14:51:44 up 69 days, 5:37, 2 users, load average: 1.00, 1.00, 1.00

I was using tnsr centos version before, as it is understood from uptime, I switched to ubuntu on that date and now I am using it in ubuntu without any problems.

[oudenos]

@lukecage Please run the following and post the result

dataplane shell sudo vppctl
show hardware-interfaces

[LukeCage]

@oudenos check private message