Upgrade to 23.01 WAN speed halved

battlebot32

@tman222 said in Upgrade to 23.01 WAN speed halved:

I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?

In my last round of testing I wasn't using any tunable outside of what is baked into Pfsense. As stated earlier I even went as far as doing a factory reset in case I missed anything with the same result. Maybe an issue that only affects the Chelsio T520-CR cards then?

battlebot32

So... The Saga continues. I had to put the Chelsio card back in. The server that I put it into (where I got the intel card) didn't like it. I also tried with a Chelsio t420 card that
had lying around (hoping it would work better on 23.0)1 and it was also not getting past 470mbps. I will submit a bug report as suggested earlier and see if this can get figured out.

JensErling

Can confirm that this bug is related to the NIC on version 23.01. Since I have 4 completely different pfSense systems running where the only common thing is the same NIC, and all the systems are capping out at about 470 Mbit after the upgrade from pfSense 2.6 CE to 23.01 Plus.
I also did some tests on a fresh install with both Iperf (locally) and speedtest and the throughput is always maxing out at 470 Mbit after doing the upgrade. It does not seem to matter if I use the regular 1G ethernet port or the 10G sfp + with link speed set to 1G. (I have not tested with 10G link speed)

I’m using the Chelsio T422-CR NIC

battlebot32

@jenserling I did put in a bug report with the PFsense developers (hopefully correctly) and included this post in the report. I am hoping that they will see it and work on figuring out why the older Chelsio line (t4/t5 which use the same driver I believe) is having this issue.

guygg

I finally got around to messing with the speed problem I was having with my x550-T2, mentioned a while back in this thread. I tried all manner of auto/manual link speed settings on both sides of the connections and just couldn't get past half the speed I should be getting (usually just barely over 2gb of my 5gb fiber). I did a config back and did a fresh 2.6 install and tested and was then getting full speed again before doing anything else with the install. I then did an upgrade to 23.01 and tested again, and again I continued to get full speed. I restored the config back from before the re-install and surprisingly still got full speed. So, the issue seems resolved. I guess something went sideways during the 22.05 to 23.01 upgrade process? Whatever the reason, things seem good now.

JensErling

@guygg I did a clean install yesterday and the error was still present, so no, this is not fixed.

debo4479

@guygg although your issue may have been similar, everyone else is using Chelsio T4/T5 variants and no matter what the config is speeds don't go past 470mb/s.

Hopefully the Bug report gets some traction.

guygg

@debo4479 Yeah, definitely seems to be the case. Just figured I'd follow up to explain the end result of my issue, as long ago in the thread when I first chimed in, it wasn't as clearly a Chelsio thing, and I had a "speed halved" symptom of getting about half my 5gb fiber speed. So yeah, seems unrelated to what the thread became about. Just wanted to toss in a note to say what solved it for my case in case anybody came across this in a search. Good luck to those with the Chelsio issue...

stephenw10

Anyone seeing this with a T520-SO NIC?

That's the only one I have on hand to test and, so far, I'm not seeing the same limitations. I can pass whatever the CPU will support, which isn't much in this crappy test box but it's well over 1G.

JensErling

Any updates on this issue?

Since I upgraded from 2.6.0 CE to 23.01 Plus it does not seem that I can revert back to the 22 version or choose a “Development” option on the update page. How can I revert back or where can I download the 22 plus version?

stephenw10

If you're running on Netgate hardware you can open a ticket and request an older image.

If you are running on your own hardware and upgraded from 2.6 then it's not possible to reach Plus 22.0X.

A good test here would be to try a 2.7 snapshot if you can. That has the most recent drivers. If you still hit this issue there then it will be in 23.05 too.

Steve

JensErling

@stephenw10 Did some testing now and the same problem applies to the latest development version "Current Base System2.7.0.a.20230417.0600"

stephenw10

Ok, that's good info. And that's using the T422-CR?

JensErling

@stephenw10 That's correct

Jimbohello

Same Scenario with this chelsio

download is max at 400-450 (50%)
while uploads perform 100%

em0@pci0:1:0:0:	class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
t5iov0@pci0:2:0:0:	class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov1@pci0:2:0:1:	class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov2@pci0:2:0:2:	class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5iov3@pci0:2:0:3:	class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet
t5nex0@pci0:2:0:4:	class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    device     = 'T520-CR Unified Wire Ethernet Controller'
    class      = network
    subclass   = ethernet

tman222

No speed issues to report here - using a Chelsio T540-SO-CR on the LAN side and Intel i210 on the WAN side.

Below is the set tunables I'm currently using for the Chelsio card:

hw.cxgbe.cong_drop="1"
hw.cxgbe.pause_settings="0"
hw.cxgbe.nrxq="8"
hw.cxgbe.ntxq="8"
hw.cxgbe.qsize_txq="2048"
hw.cxgbe.qsize_rxq="2048"
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"

Hope this helps.

debo4479

@tman222

Thanks for the info but it is only impacting the T4/5 CR variants without the SO.

tman222

@debo4479 said in Upgrade to 23.01 WAN speed halved:

@tman222

Thanks for the info but it is only impacting the T4/5 CR variants without the SO.

Hmmm, that's interesting. As far as I know, the only difference are the offload capabilities of the regular CR vs. SO-CR cards (SO = Server Offload). If those offloading capabilities are enabled by default by the Chelsio driver in newer version(s) of pfSense has anyone tried disabling them explicitly to see if that helps the performance?

Redmine issue for reference:
https://redmine.pfsense.org/issues/9091

stephenw10

The TCP off-load stuff should have no impact on connections through the firewall, rather than terminating on it, even if it is loaded. But it's not loaded by default anyway.
It does seem like the -SO variants are not affected though.
Is anyone seeing this issue on an -SO card?

RobbieTT

Anyone looked at duplex testing in detail, given the halving at 5 GbE and 1 GbE?

️