Upgrade to 23.01 WAN speed halved
-
Great detail shared there, thanks for the effort! FWIW, I think this is related to the T520 card specifically. I ordered/installed a T6225-CR (to replace my T520's) and I do NOT have performance degradation with that chipset.
Not sure if that's just by chance or if the drivers have been better optimized compared to the T520 cards? Perhaps the T520 cards are just getting a little long in the tooth? Total assumptions, but this is definitely not an isolated problem.
-
hoping that this is something the developers are aware of/working on
I would suggest looking for and/or opening an issue at Redmine.pfSense.org.
-
If it is an issue with the Chelsio NIC specifically it's probably a change in the driver. In that case though I'd expect to see more reports in FreeBSD or even a bug report there. Not seeing anything relevant though:
https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=chelsio -
I agree with Stephenw10 that this is probably something to do with the driver for the Chelsio t520 (possibly this specific line of cards). I pulled the card out this morning and put in a Intel X520 82599ES (pulled from a supermicro server) and installed that on the router and did the upgrade and was able to get full speeds through the intel card after the upgrade. (Also no errors reported on the interfaces. I was getting those before on the Chelsio card after the upgrade, but not consistently.) So not an issue with Pfsense as much as the driver in FreeBSD 14.0 (coming from 12-stable). I'll see if I can pick up a Chelsio T6 card in the future and test with that.
-
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
-
One other thing I also wanted to mention - there have been some changes to FreeBSD networking (routing) since FreeBSD 12. It looks like a lot changed in FreeBSD 13 so I assume all this is incorporated in FreeBSD 14 as well which pfSense 23.01 is based on:
https://papers.freebsd.org/2021/eurobsdcon/chernikov-routing_freebsd_13/
-
@tman222 said in Upgrade to 23.01 WAN speed halved:
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
In my last round of testing I wasn't using any tunable outside of what is baked into Pfsense. As stated earlier I even went as far as doing a factory reset in case I missed anything with the same result. Maybe an issue that only affects the Chelsio T520-CR cards then?
-
So... The Saga continues. I had to put the Chelsio card back in. The server that I put it into (where I got the intel card) didn't like it. I also tried with a Chelsio t420 card that
had lying around (hoping it would work better on 23.0)1 and it was also not getting past 470mbps. I will submit a bug report as suggested earlier and see if this can get figured out. -
Can confirm that this bug is related to the NIC on version 23.01. Since I have 4 completely different pfSense systems running where the only common thing is the same NIC, and all the systems are capping out at about 470 Mbit after the upgrade from pfSense 2.6 CE to 23.01 Plus.
I also did some tests on a fresh install with both Iperf (locally) and speedtest and the throughput is always maxing out at 470 Mbit after doing the upgrade. It does not seem to matter if I use the regular 1G ethernet port or the 10G sfp + with link speed set to 1G. (I have not tested with 10G link speed)Iām using the Chelsio T422-CR NIC
-
@jenserling I did put in a bug report with the PFsense developers (hopefully correctly) and included this post in the report. I am hoping that they will see it and work on figuring out why the older Chelsio line (t4/t5 which use the same driver I believe) is having this issue.
-
I finally got around to messing with the speed problem I was having with my x550-T2, mentioned a while back in this thread. I tried all manner of auto/manual link speed settings on both sides of the connections and just couldn't get past half the speed I should be getting (usually just barely over 2gb of my 5gb fiber). I did a config back and did a fresh 2.6 install and tested and was then getting full speed again before doing anything else with the install. I then did an upgrade to 23.01 and tested again, and again I continued to get full speed. I restored the config back from before the re-install and surprisingly still got full speed. So, the issue seems resolved. I guess something went sideways during the 22.05 to 23.01 upgrade process? Whatever the reason, things seem good now.
-
@guygg I did a clean install yesterday and the error was still present, so no, this is not fixed.
-
@guygg although your issue may have been similar, everyone else is using Chelsio T4/T5 variants and no matter what the config is speeds don't go past 470mb/s.
Hopefully the Bug report gets some traction.
-
@debo4479 Yeah, definitely seems to be the case. Just figured I'd follow up to explain the end result of my issue, as long ago in the thread when I first chimed in, it wasn't as clearly a Chelsio thing, and I had a "speed halved" symptom of getting about half my 5gb fiber speed. So yeah, seems unrelated to what the thread became about. Just wanted to toss in a note to say what solved it for my case in case anybody came across this in a search. Good luck to those with the Chelsio issue...
-
Anyone seeing this with a T520-SO NIC?
That's the only one I have on hand to test and, so far, I'm not seeing the same limitations. I can pass whatever the CPU will support, which isn't much in this crappy test box but it's well over 1G.
-
Any updates on this issue?
Since I upgraded from 2.6.0 CE to 23.01 Plus it does not seem that I can revert back to the 22 version or choose a āDevelopmentā option on the update page. How can I revert back or where can I download the 22 plus version?
-
If you're running on Netgate hardware you can open a ticket and request an older image.
If you are running on your own hardware and upgraded from 2.6 then it's not possible to reach Plus 22.0X.
A good test here would be to try a 2.7 snapshot if you can. That has the most recent drivers. If you still hit this issue there then it will be in 23.05 too.
Steve
-
@stephenw10 Did some testing now and the same problem applies to the latest development version "Current Base System2.7.0.a.20230417.0600"
-
Ok, that's good info. And that's using the T422-CR?
-
@stephenw10 That's correct
-
S SteveITS referenced this topic on
-
G Gertjan referenced this topic on
-
G Gertjan referenced this topic on
