Upgrade to 23.01 WAN speed halved
-
@steveits Yes. I've been running stable on 22.05 with an average of about 3.5 Gbps throughput before 23.01.
Before rolling back to 22.05, i did verify that the Chelsio fine tuning settings where correct and retained. It just won't go past 480 ~500 Mbps with a Chelsio Card on 23.01.
2.6 works fine. "update to 22.05, still functions as expected. Apply 23.01. Stuck at 500 Mbps.
Our network monitoring system indicated that the LAN port on the Chelsio card had over a million transmit discards on the LAN port while on 23.01
-
I also have had the exact symptoms. Running Intel I210 adapter for the WAN interface and a Chelsio T520-CR for the LAN 10gb backend. Was fine on 22.05 running ~940 down and up. After the upgrade to 23.01 I was getting ~470 download and ~940 upload. I do have an internal speedtest server and did a test from a computer (10gb nic) -> vlan hop through pfsense -> speedtest server (10gb nic) I was able to do 9GB up and down fine. But when I was doing a download from my file server on a separate computer (10gb server -> vlan hop through pfsenes -> 1gb nic computer this time) and was getting the 470Mbps speeds internally. That conversion from 10gb to 1gb on the Chelsio card doesn't seem to be working correctly. I re-installed 2.6 from scratch, pulled the backup (up to 940mbps again up and down), 22.01, then finally to 23.01. As soon as I hit 23.01 it again dropped to 470 down, 940 up. I did test by putting the WAN interface on the Chelsio card with the same behaviors (I didn't test upload speed at this point because it was getting late.) I kept all the same settings and I was also using the tuning options suggested by Pfsense for the Chelsio cards and those were retained across the upgrades. I did get a message about a kernel panic the second time I upgraded to 23.01 and tried to download the debug log which failed at the time and couldn't find the log before I had to re-image again to get it working late that night. I'll see if I can reproduce the panic this weekend so that I can get it posted.
-
So. I did install fresh (2.6) upgraded to 22.01, then 22.05 (pulled the config back during the process). Full gig speeds at this point. Upgraded to 23.01 and speeds was 470 down, 940 up again. Looked at making sure the card was on the latest firmware (1.27.0.0) which it was. Did a factory reset from the GUI to wipe any configurations that I had and take the config to stock to see if there was anything that I had configured that was causing the issue. Had to jump on the box for a minute to change the LAN port to the Chelsio card (WAN was still on Intel copper port) after the reset and once again did a speed test. 470 down, 940 up. I did see some interesting options on the ifconfig for the chelsio interface that I hadn't seen before. I turned off TXRTLMT and HWRXTSTMP with no change. I also tried to disable NOMAP but there was unable to disable it through ifconfig. This appears to be a driver issue and I have not been able to make a single change in the configuration to even affect the speeds. TSO and LRO were confirmed off by ifconfig. I didn't have any of the tunning options in the loader.conf file that I had before all of these changes were made. I did notice another post in the pfsense forums where this was happening during the development of 23.01 (https://forum.netgate.com/topic/177308/half-network-speed-after-upgrading) but apparently wasn't addressed before release. I'm hoping that this is something the developers are aware of/working on and would be willing to work with them on this.
-
Great detail shared there, thanks for the effort! FWIW, I think this is related to the T520 card specifically. I ordered/installed a T6225-CR (to replace my T520's) and I do NOT have performance degradation with that chipset.
Not sure if that's just by chance or if the drivers have been better optimized compared to the T520 cards? Perhaps the T520 cards are just getting a little long in the tooth? Total assumptions, but this is definitely not an isolated problem.
-
hoping that this is something the developers are aware of/working on
I would suggest looking for and/or opening an issue at Redmine.pfSense.org.
-
If it is an issue with the Chelsio NIC specifically it's probably a change in the driver. In that case though I'd expect to see more reports in FreeBSD or even a bug report there. Not seeing anything relevant though:
https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=chelsio -
I agree with Stephenw10 that this is probably something to do with the driver for the Chelsio t520 (possibly this specific line of cards). I pulled the card out this morning and put in a Intel X520 82599ES (pulled from a supermicro server) and installed that on the router and did the upgrade and was able to get full speeds through the intel card after the upgrade. (Also no errors reported on the interfaces. I was getting those before on the Chelsio card after the upgrade, but not consistently.) So not an issue with Pfsense as much as the driver in FreeBSD 14.0 (coming from 12-stable). I'll see if I can pick up a Chelsio T6 card in the future and test with that.
-
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
-
One other thing I also wanted to mention - there have been some changes to FreeBSD networking (routing) since FreeBSD 12. It looks like a lot changed in FreeBSD 13 so I assume all this is incorporated in FreeBSD 14 as well which pfSense 23.01 is based on:
https://papers.freebsd.org/2021/eurobsdcon/chernikov-routing_freebsd_13/
-
@tman222 said in Upgrade to 23.01 WAN speed halved:
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
In my last round of testing I wasn't using any tunable outside of what is baked into Pfsense. As stated earlier I even went as far as doing a factory reset in case I missed anything with the same result. Maybe an issue that only affects the Chelsio T520-CR cards then?
-
So... The Saga continues. I had to put the Chelsio card back in. The server that I put it into (where I got the intel card) didn't like it. I also tried with a Chelsio t420 card that
had lying around (hoping it would work better on 23.0)1 and it was also not getting past 470mbps. I will submit a bug report as suggested earlier and see if this can get figured out. -
Can confirm that this bug is related to the NIC on version 23.01. Since I have 4 completely different pfSense systems running where the only common thing is the same NIC, and all the systems are capping out at about 470 Mbit after the upgrade from pfSense 2.6 CE to 23.01 Plus.
I also did some tests on a fresh install with both Iperf (locally) and speedtest and the throughput is always maxing out at 470 Mbit after doing the upgrade. It does not seem to matter if I use the regular 1G ethernet port or the 10G sfp + with link speed set to 1G. (I have not tested with 10G link speed)Iām using the Chelsio T422-CR NIC
-
@jenserling I did put in a bug report with the PFsense developers (hopefully correctly) and included this post in the report. I am hoping that they will see it and work on figuring out why the older Chelsio line (t4/t5 which use the same driver I believe) is having this issue.
-
I finally got around to messing with the speed problem I was having with my x550-T2, mentioned a while back in this thread. I tried all manner of auto/manual link speed settings on both sides of the connections and just couldn't get past half the speed I should be getting (usually just barely over 2gb of my 5gb fiber). I did a config back and did a fresh 2.6 install and tested and was then getting full speed again before doing anything else with the install. I then did an upgrade to 23.01 and tested again, and again I continued to get full speed. I restored the config back from before the re-install and surprisingly still got full speed. So, the issue seems resolved. I guess something went sideways during the 22.05 to 23.01 upgrade process? Whatever the reason, things seem good now.
-
@guygg I did a clean install yesterday and the error was still present, so no, this is not fixed.
-
@guygg although your issue may have been similar, everyone else is using Chelsio T4/T5 variants and no matter what the config is speeds don't go past 470mb/s.
Hopefully the Bug report gets some traction.
-
@debo4479 Yeah, definitely seems to be the case. Just figured I'd follow up to explain the end result of my issue, as long ago in the thread when I first chimed in, it wasn't as clearly a Chelsio thing, and I had a "speed halved" symptom of getting about half my 5gb fiber speed. So yeah, seems unrelated to what the thread became about. Just wanted to toss in a note to say what solved it for my case in case anybody came across this in a search. Good luck to those with the Chelsio issue...
-
Anyone seeing this with a T520-SO NIC?
That's the only one I have on hand to test and, so far, I'm not seeing the same limitations. I can pass whatever the CPU will support, which isn't much in this crappy test box but it's well over 1G.
-
Any updates on this issue?
Since I upgraded from 2.6.0 CE to 23.01 Plus it does not seem that I can revert back to the 22 version or choose a āDevelopmentā option on the update page. How can I revert back or where can I download the 22 plus version?
-
If you're running on Netgate hardware you can open a ticket and request an older image.
If you are running on your own hardware and upgraded from 2.6 then it's not possible to reach Plus 22.0X.
A good test here would be to try a 2.7 snapshot if you can. That has the most recent drivers. If you still hit this issue there then it will be in 23.05 too.
Steve
