half network speed after upgrading

[stephenw10]

Right but if you test between the ports using iperf. Or between a client and pfSense running iperf itself that would show if it's the Chelsio.

[High_bounce]

ran iperf between my main desktop and pfsense and i get 450Mbps from pfsense but 950Mbps to pfsense. if there are other command you would like me to run please post them i haven't messed with iperf outside of speed tests

[stephenw10]

Ah, interesting. I'd try swapping the server and client roles so the states are opened the other way and see if that changes anything.
Make sure the Chelsio shows multiple Tx and Rx queues in the boot logs when the driver attaches. It could be using 1 CPU core in one direction for some reason. Though with that CPU i should still be OK.

[High_bounce]

the rx/tx cues are off i have turned off hyperthreading on the cpu. that's all i see that are wrong

[skogs]

@high_bounce perhaps this is old school and things don't work this way anymore but generally the second you actually SET a speed and duplex on one side...the link goes to hell. Because it is no longer auto-negotiating properly, the other side will drop to ultra slow and half duplex.
If you set anything at all manually...you need to be hyper vigilant at setting everything manually. ... and most consumer switches and modems aren't even capable of being set manually....so guaranteed failure.

[stephenw10]

Yeah 8 queues each way, no problem.

You could look at the off-load options that are enabled in: ifconfig -vvvm cxl0

[High_bounce]

[stephenw10]

Hmm, I'd try disabling TSO4 and LRO as a first step there. Those are both normally disabled by default in Sys > Adv > Networking.

[High_bounce]

both are already checked in the Adv settings. should i try unchecking then rechecking in the UI?

[stephenw10]

Yes, try that. Though that may require you to reboot.

Otherwise you should be able to manually disable them using ifconfig like:

ifconfig cxl0 -tso4

Unfortunately the syntax there can be confusing and I don't have anything setup right now to check.

[High_bounce]

no changes with the radio buttons or ifconfig. ifconfig gets reset after reboot so maybe needs to be set in a config file

[stephenw10]

Disabling those options using ifconfig doesn't work? Or it doesn't make any difference to throughput?

[High_bounce]

it changes the options for the interface but no change in throughput. also i changed boot environment back to 22.05 to check for differences i found a few i have no idea if they make a difference:

boot logs
23.01 8 txq/rxq, 22 MSI-X, 46eq, 21iq
22.05 4 txq/rxq, 14 MSI-X, 30eq, 13iq

ifconfig looks all the same but 23.01 has an option called "NOMAP" i am not sure what it means and cant find anything online about it

[stephenw10]

NOMAP is to allow unmapped mbufs. I don't expect it to make any difference but I can't find any way to disable it! It looks like you should be able to ifconfig cxl0 -nomap but that doesn't work for me.

You can almost certainly tune the queues back to 4x in the sysctls/tunables but it's hard to imagine that would help. 22 MSI-X interrupts.... I couldn't say but again it's hard to see how more would be worse.

If you run top -HaSP during a test do you see the loading spread across the cores correctly?

[High_bounce]

no change in cpu utilisation im still at a lost as to cause are you able to replicate on your end?

[stephenw10]

I don't have a Chelsio 10GBase-T NIC to test with unfortunately. But I haven't seen anything like that on any other NICs.

[High_bounce]

no worries i just switch to an old intel x540-t2 should be fine until i upgrade the rest of my network to multi-gig thank you for all your help. i hope it gets fixed at some point i have always herd that Chelsio nics were the best with bsd maybe that is changing now