Upgrade to 23.01 WAN speed halved
-
One other thing I also wanted to mention - there have been some changes to FreeBSD networking (routing) since FreeBSD 12. It looks like a lot changed in FreeBSD 13 so I assume all this is incorporated in FreeBSD 14 as well which pfSense 23.01 is based on:
https://papers.freebsd.org/2021/eurobsdcon/chernikov-routing_freebsd_13/
-
@tman222 said in Upgrade to 23.01 WAN speed halved:
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
In my last round of testing I wasn't using any tunable outside of what is baked into Pfsense. As stated earlier I even went as far as doing a factory reset in case I missed anything with the same result. Maybe an issue that only affects the Chelsio T520-CR cards then?
-
So... The Saga continues. I had to put the Chelsio card back in. The server that I put it into (where I got the intel card) didn't like it. I also tried with a Chelsio t420 card that
had lying around (hoping it would work better on 23.0)1 and it was also not getting past 470mbps. I will submit a bug report as suggested earlier and see if this can get figured out. -
Can confirm that this bug is related to the NIC on version 23.01. Since I have 4 completely different pfSense systems running where the only common thing is the same NIC, and all the systems are capping out at about 470 Mbit after the upgrade from pfSense 2.6 CE to 23.01 Plus.
I also did some tests on a fresh install with both Iperf (locally) and speedtest and the throughput is always maxing out at 470 Mbit after doing the upgrade. It does not seem to matter if I use the regular 1G ethernet port or the 10G sfp + with link speed set to 1G. (I have not tested with 10G link speed)I’m using the Chelsio T422-CR NIC
-
@jenserling I did put in a bug report with the PFsense developers (hopefully correctly) and included this post in the report. I am hoping that they will see it and work on figuring out why the older Chelsio line (t4/t5 which use the same driver I believe) is having this issue.
-
I finally got around to messing with the speed problem I was having with my x550-T2, mentioned a while back in this thread. I tried all manner of auto/manual link speed settings on both sides of the connections and just couldn't get past half the speed I should be getting (usually just barely over 2gb of my 5gb fiber). I did a config back and did a fresh 2.6 install and tested and was then getting full speed again before doing anything else with the install. I then did an upgrade to 23.01 and tested again, and again I continued to get full speed. I restored the config back from before the re-install and surprisingly still got full speed. So, the issue seems resolved. I guess something went sideways during the 22.05 to 23.01 upgrade process? Whatever the reason, things seem good now.
-
@guygg I did a clean install yesterday and the error was still present, so no, this is not fixed.
-
@guygg although your issue may have been similar, everyone else is using Chelsio T4/T5 variants and no matter what the config is speeds don't go past 470mb/s.
Hopefully the Bug report gets some traction.
-
@debo4479 Yeah, definitely seems to be the case. Just figured I'd follow up to explain the end result of my issue, as long ago in the thread when I first chimed in, it wasn't as clearly a Chelsio thing, and I had a "speed halved" symptom of getting about half my 5gb fiber speed. So yeah, seems unrelated to what the thread became about. Just wanted to toss in a note to say what solved it for my case in case anybody came across this in a search. Good luck to those with the Chelsio issue...
-
Anyone seeing this with a T520-SO NIC?
That's the only one I have on hand to test and, so far, I'm not seeing the same limitations. I can pass whatever the CPU will support, which isn't much in this crappy test box but it's well over 1G.
-
Any updates on this issue?
Since I upgraded from 2.6.0 CE to 23.01 Plus it does not seem that I can revert back to the 22 version or choose a “Development” option on the update page. How can I revert back or where can I download the 22 plus version?
-
If you're running on Netgate hardware you can open a ticket and request an older image.
If you are running on your own hardware and upgraded from 2.6 then it's not possible to reach Plus 22.0X.
A good test here would be to try a 2.7 snapshot if you can. That has the most recent drivers. If you still hit this issue there then it will be in 23.05 too.
Steve
-
@stephenw10 Did some testing now and the same problem applies to the latest development version "Current Base System2.7.0.a.20230417.0600"
-
Ok, that's good info. And that's using the T422-CR?
-
@stephenw10 That's correct
-
S SteveITS referenced this topic on
-
G Gertjan referenced this topic on
-
G Gertjan referenced this topic on
-
Same Scenario with this chelsio
download is max at 400-450 (50%)
while uploads perform 100%em0@pci0:1:0:0: class=0x020000 card=0xa01f8086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet t5iov0@pci0:2:0:0: class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T520-CR Unified Wire Ethernet Controller' class = network subclass = ethernet t5iov1@pci0:2:0:1: class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T520-CR Unified Wire Ethernet Controller' class = network subclass = ethernet t5iov2@pci0:2:0:2: class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T520-CR Unified Wire Ethernet Controller' class = network subclass = ethernet t5iov3@pci0:2:0:3: class=0x020000 card=0x00001425 chip=0x50011425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T520-CR Unified Wire Ethernet Controller' class = network subclass = ethernet t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 hdr=0x00 vendor = 'Chelsio Communications Inc' device = 'T520-CR Unified Wire Ethernet Controller' class = network subclass = ethernet -
No speed issues to report here - using a Chelsio T540-SO-CR on the LAN side and Intel i210 on the WAN side.
Below is the set tunables I'm currently using for the Chelsio card:
hw.cxgbe.cong_drop="1" hw.cxgbe.pause_settings="0" hw.cxgbe.nrxq="8" hw.cxgbe.ntxq="8" hw.cxgbe.qsize_txq="2048" hw.cxgbe.qsize_rxq="2048" hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0"Hope this helps.
-
Thanks for the info but it is only impacting the T4/5 CR variants without the SO.
-
@debo4479 said in Upgrade to 23.01 WAN speed halved:
Thanks for the info but it is only impacting the T4/5 CR variants without the SO.
Hmmm, that's interesting. As far as I know, the only difference are the offload capabilities of the regular CR vs. SO-CR cards (SO = Server Offload). If those offloading capabilities are enabled by default by the Chelsio driver in newer version(s) of pfSense has anyone tried disabling them explicitly to see if that helps the performance?
Redmine issue for reference:
https://redmine.pfsense.org/issues/9091 -
The TCP off-load stuff should have no impact on connections through the firewall, rather than terminating on it, even if it is loaded. But it's not loaded by default anyway.
It does seem like the -SO variants are not affected though.
Is anyone seeing this issue on an -SO card?
