half network speed after upgrading

High_bounce

intel i3-12100 on an gigabyte motherboard. intel 225i onboard and chelsio t5-520bt in pci-e slot

stephenw10

So plenty powerful enough then!

Link states are all as expected?

Try testing locally using iperf. Or use speedtest-cli from pfSense itself. Try to narrow down which NIC(s) are throttling the connection.

Steve

High_bounce

link states are as expected i even tried manually setting them to values. the i225 is running at 2.5g and the chelsio is 1g so because the speed is limited to exactly half of 1g i am going to guess its something with the chelsio card. i don't have a second card to drop in and test though. pfblocker seems to be giving me some trouble as well after update. for some reason its taking 1-2 full cores for some tmp log. will have to report that to them

stephenw10

@high_bounce said in half network speed after upgrading:

chelsio 520bt.

That's a dual port card? Try testing between those ports if they are both internal subnets.

High_bounce

yes it is a dual port. tried changing to other port same problem

stephenw10

Right but if you test between the ports using iperf. Or between a client and pfSense running iperf itself that would show if it's the Chelsio.

High_bounce

ran iperf between my main desktop and pfsense and i get 450Mbps from pfsense but 950Mbps to pfsense. if there are other command you would like me to run please post them i haven't messed with iperf outside of speed tests

stephenw10

Ah, interesting. I'd try swapping the server and client roles so the states are opened the other way and see if that changes anything.
Make sure the Chelsio shows multiple Tx and Rx queues in the boot logs when the driver attaches. It could be using 1 CPU core in one direction for some reason. Though with that CPU i should still be OK.

High_bounce

boot log.png

the rx/tx cues are off i have turned off hyperthreading on the cpu. that's all i see that are wrong

skogs

@high_bounce perhaps this is old school and things don't work this way anymore but generally the second you actually SET a speed and duplex on one side...the link goes to hell. Because it is no longer auto-negotiating properly, the other side will drop to ultra slow and half duplex.
If you set anything at all manually...you need to be hyper vigilant at setting everything manually. ... and most consumer switches and modems aren't even capable of being set manually....so guaranteed failure.

stephenw10

Yeah 8 queues each way, no problem.

You could look at the off-load options that are enabled in: ifconfig -vvvm cxl0

High_bounce

stephenw10

Hmm, I'd try disabling TSO4 and LRO as a first step there. Those are both normally disabled by default in Sys > Adv > Networking.

High_bounce

both are already checked in the Adv settings. should i try unchecking then rechecking in the UI?

stephenw10

Yes, try that. Though that may require you to reboot.

Otherwise you should be able to manually disable them using ifconfig like:

ifconfig cxl0 -tso4

Unfortunately the syntax there can be confusing and I don't have anything setup right now to check.

High_bounce

no changes with the radio buttons or ifconfig. ifconfig gets reset after reboot so maybe needs to be set in a config file

stephenw10

Disabling those options using ifconfig doesn't work? Or it doesn't make any difference to throughput?

High_bounce

it changes the options for the interface but no change in throughput. also i changed boot environment back to 22.05 to check for differences i found a few i have no idea if they make a difference:

boot logs
23.01 8 txq/rxq, 22 MSI-X, 46eq, 21iq
22.05 4 txq/rxq, 14 MSI-X, 30eq, 13iq

ifconfig looks all the same but 23.01 has an option called "NOMAP" i am not sure what it means and cant find anything online about it

stephenw10

NOMAP is to allow unmapped mbufs. I don't expect it to make any difference but I can't find any way to disable it! It looks like you should be able to ifconfig cxl0 -nomap but that doesn't work for me.

You can almost certainly tune the queues back to 4x in the sysctls/tunables but it's hard to imagine that would help. 22 MSI-X interrupts.... I couldn't say but again it's hard to see how more would be worse.

If you run top -HaSP during a test do you see the loading spread across the cores correctly?

High_bounce

no change in cpu utilisation im still at a lost as to cause are you able to replicate on your end?