Upgrade to 23.01 WAN speed halved
-
Hey folks,
Upgraded from 22.05 on my Lenovo ST50 server running a 6C Xeon 2176G, 16GB ECC memory, and 2 Chelsio T520 cards (CR and BT), with the CR (LAN) connected via 10Gig twinax to my Juniper Ex2300 and the BT (WAN) connected to my Nokia GPON ONT @ 1Gig. System has been rock-steady for quite some time (last 300 days 'ish).
Completed upgrade via console and everything went really smooth. Rebooted the machine several times after the upgrade but as of now, this is what's new:
- WAN performance is halved @ 470 Mbps and was always a stout @ 940Mbps on any wired gigabit device on the network via Speedtest
- CPU usage is stuck at around 9-10% now and has never gone above 2-3% unless under heavy load previously
-CPU interrupt is now showing @ 9-10% and has ALWAYS been at 0%.
Checked all my interfaces and they retained previous settings (Full duplex, Tx, Rx) and Juniper ex2300 xe SFP+ port still showing 10G full duplex on LAN.
No pfsense packages currently installed from base install.
Any ideas what could've changed here to cause the WAN speed to halve? Are these Chelsio cards no longer any good and perhaps drivers being deprecated somehow?
Thanks for any input.
Partial snapshot of vmstat -i. Not sure what's on IRQ16 but it looks to be a potential culprit..
[23.01-RELEASE][root@pfSense]/root: vmstat -i
interrupt total rate
irq4: uart0 650 0
irq16: ig4iic0+ 84027864 36922 --> Believe this is IRQ reserved for the Intel chipset on mobo
cpu0:timer 133628 59
cpu1:timer 123888 54
cpu2:timer 2547516 1119
cpu3:timer 115151 51
cpu4:timer 94963 42
cpu5:timer 117391 52EDIT UPDATE: Decided to unplug my interfaces from the Lenovo ST50 last night and into my up and running Netgate 8200 Plus (still running 22.05.1) and configured same as the Lenovo. As expected, everything came right up and WAN performance again pegged across all devices @ 950Mbps.
Also, I kept the Lenovo server running overnight albeit disconnected from all interfaces. Not sure if this was CRON or some other cleanup happening behind the scences but CPU interrupts are gone completely and CPU usage is back down to normal.
However, swapped all connections back into the Lenovo this AM and while system is seemingly humming along beautifully, WAN interfaces is still stuck at half previous performance (from 950Mbps to 480Mbps). Really at a loss on the WAN speed drop...
EDIT 2: Since reconnected interfaces back into Lenovo, CPU usage back to 10% and CPU interrupt back at 9%. ??!
EDIT 3: Selected 22.05 boot environment and reloaded server. Came right back up, CPU usage low to nonexistent, CPU interrupts .2%, WAN speed to 950Mbps up/down. (Me...facepalming myself)
-
Hmm, not seen that but it looks likely to be the NIC /driver. Are you able to try a different NIC?
-
Thanks Stephen.
Yes, I can plug back in an Intel X550-T2 (in place of the Chelsio card) which I would use for the WAN interface. Can I do that without having to reinstall from CE-->plus? Can I halt the system, swap cards, and reboot and config?
-
It will change the NDI but won't prevent it booting so you can test.
You might also try speed test through just one NIC. So iperf to pfSense itself from the LAN. Or run speedtest (or iperf) from the pfSense CLI to test only the WAN.
-
I too seem to have had my speed roughly cut in half for my x550-T2 (down from 5gb to the low 2+ gb range after going from 22 to 23. Haven't done much experimenting yet. I'll try stepping back to 22 at some point over the weekend. On 22, it never said what my speed was negotiated at (simply saying it was unknown), but looking at the 5gb port on the AT&T fiber hardware on the other side of the link showed it at the proper 5gb port speed, plus I could speedtest up to 5gb (4.75 or so), no problem. Now, when set to auto, it will only auto-negotiate to 1gb. If I set it to 5gb, that's when it won't go any higher than 2.3 or so. So yeah, now it can report link speeds other than 1gb or 10gb, but performance has taken a major hit.
-
If you set to 5Gbps specifically the other side must also be set to that otherwise you will end up with a negotiation failure at one end. It's probably defaulting to 2.5G. Are you seeing errors or collisions on the interface?
This seems unrelated to what the OP here is seeing though.Steve
-
OK, did some experiments. I set the interface on the AT&T side from auto to 5gb. Even restarted both pfsense and the AT&T equipment and still getting no more than half the speed I used to before the upgrade to 23. Setting both to auto (which they were both at on 22 and working fine up to 5gb speeds) will only get me a 1gb negotiated link every time. And with both manually set to 5gb, I get no better than half the speed I used to. I'm seeing a ton of interrupts on both interfaces on the x550-T2 in pfsense. Don't recall what that was like in v22, as I didn't really look at that when things were working great. Think I'm gonna fall back to v22 (gonna have to poke around about doing that, as I've not needed to undo an update in pfsense before), which despite not actually indicating link speed, worked great for performance. Here are a couple screenshots of both the interface on the AT&T equipment and the pfsense x550-T2.
-
Hmm, check the boot logs to see if the NIC is coming up with the expected number of queues.
There is also a sysctl you can use to set the advertised speeds available for link negotiation. That might allow it to negotiate at 5G correctly.
-
I have not had the chance to update the firmware on one of X550-T2's yet, but will do so and test on my end. For the time being, I kept both my Chelsio cards installed, did a fresh install from 2.6CE to 22.05 (and installed the 23.01 security patches) and the system is just running super snappy, WAN speeds full symmetrical 950Mbps.
Will grab another nvme SSD, update the X550 firmware, reinstall --> 23.01 and retest WAN speeds sometime this week to verify if it's my Nokia GPON ONT that is the issue with the updated drivers in 23.01 for the Chelsio (and potentially X550) and report back.
-
Are any of you using traffic shaping? I have a post in the traffic shaping forum about my speed being cut in half with traffic shaping. I turned it off and that solved my problem. Unfortunately, due to another (known) bug, I can’t turn it back on to do more experimentation.
-
Thanks for sharing. Unless traffic shaping is enabled by default on a fresh install (which I would assume No) then I am currently not.
-
@brachy33 Correct. It is not.
-
Can also confirm I am experiencing the same issue. I am running pfsense as VM on unRaid and passing through Ethernet controller: Chelsio Communications Inc T520-CR. I get half of my fios 1gb speed on Wan. Rolled back to 22.05 for now.
-
@stephenw10 Not the best expert at what it should be reporting, but here are the interface lines from the boot log:
ix0: <Intel(R) X550-T2> mem 0xd1000000-0xd11fffff,0xd1400000-0xd1403fff at device 0.0 on pci1
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 4 RX queues 4 TX queues
ix0: Using MSI-X interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: ********************
ix0: PCI Express Bus: Speed 8.0GT/s Width x4
ix0: eTrack 0x80000492 PHY FW V523
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) X550-T2> mem 0xd1200000-0xd13fffff,0xd1404000-0xd1407fff at device 0.1 on pci1
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 4 RX queues 4 TX queues
ix1: Using MSI-X interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: ******************
ix1: PCI Express Bus: Speed 8.0GT/s Width x4
ix1: eTrack 0x80000492 PHY FW V523
ix1: netmap queues/slots: TX 4/2048, RX 4/2048 -
That's fine, 4 Tx/Rx queues on each. More than enough to handle 5Gbps unless your CPU is extremely low power.
What CPU is it? Does it report the expected clock speed? -
@stephenw10 It's an i5-6500 with 8gb RAM running on a 256 NVMe. Under full load, the CPU doesn't break a sweat. Interrupts on the interfaces in the status screen continue to pile up. Any other things to check/tweak? May roll back to v22 soon so that I can get proper network performance back.
https://imgur.com/a/6hBSAn1
Thanks
-
If the CPU speed reported correctly by:
sysctl dev.cpu | grep freq?I've seen one report of another 6th gen CPU stuck at 800MHz.
Seeing interrupts on the interfaces like that is expected. If it's moving a lot of traffic that rate isn't that high either. It would be interesting to see how that value compares in 22.05 but I suspect it will be similar.
Steve
-
@stephenw10 Hmmm, CPU freq seems OK to me:
dev.cpu.3.freq_levels: 3200/-1
dev.cpu.3.freq: 1152
dev.cpu.2.freq_levels: 3200/-1
dev.cpu.2.freq: 1194
dev.cpu.1.freq_levels: 3200/-1
dev.cpu.1.freq: 1149
dev.cpu.0.freq_levels: 3200/-1
dev.cpu.0.freq: 1190And I wasn't aware of what those interrupt value should look like. Not something I ever really paid attention to when things were working well. :)
-
Yup looks like what I expect with speed shift support, which is new in 23.01.
-
Quick update. Installed a new X550-T2 card and flashed to latest firmware, installed and configured from scratch 2.60 CE-->22.05 with security patches-->23.01. WAN is connected to same Metronet GPON Nokia ONT and...full WAN performance is back. 950Mbps/950Mbps symmetrical.
Of course now...I'm seeing small errors IN forming on the WAN interface (I had this happening before with this NIC as WAN interface) but it's working at correct speeds. The errors IN on this card is why I went to the Chelsio T520-BT in the first place, and that card plays nice with my ONT, but the speeds were halved on 23.01 when I first upgraded.
Next step is to double check firmware on the Chelsio card, update Lenovo server BIOS to latest, then go through entire install process again and see if I can get the speeds on the Chelsio card to remain full-speed on 23.01.
UPDATE: With the system using the x550-T2 for WAN on 23.01... after a restart no more errors IN on WAN from GPON ONT! Almost 1TB of data transferred since that reboot and no errors on either WAN or LAN. Beauty!! Looks like 23.01 is gravvy.
-
Update on my side. I had a spare Intel nic i211 that I passed through to use for my WAN.
Did a fresh install, dropped my config on it and brought it up to 22.05. I was getting full gigabit speeds download/upload.
Upgraded to 23.01 and again WAN download speeds are cut in half. I am seeing half WAN downloads speed on both Chelsio and Intel NIC. The Intel nic is at least giving me full upload speed. Could it be something in my config? I don't have anything exotic. And it works as expected on 22.05.
-
Do you have any sort of traffic shaping in play? That's about the only thing that might affect it like that.
-
@stephenw10 no traffic shaping configured.
-
Just a thought here and no idea if this is valid. But when I did my reinstall from 2.6-->23.01 I did not use a config backup. I manually reconfigured all my settings vs using the backup config...just to test (also my network config is pretty simple).
Could there be something in your old configuration backup that's just not carrying over to 23.01 and causing issues?
-
I have a backup pfsense VM that I have on a spare poweredge server. It has the same config as my main server. I updated that to 23.01 and I get full 1gb upload/download. Only difference with main server is cpu and Broadcom nics.
My next step is to setup a new vanilla pfsense install on my main server to test my Chelsio/i211 nics.
-
I am also experiencing this issue on all interfaces of my Chelsio T422 after the upgrade to 23.01 iperf from pfSense to my Unraid boxes is capped around 460-480mbps same with testing from one Unraid box to another through different interfaces. Doesn't seem to matter if its a SFP+ or Gig copper port, I can't get over 480mbps.
I will probably have to roll back to 22.05 until this is addressed. Not sure the best way to do so as I didn't notice the issue until after I made some changes to my configuration while on 23.01
-
same problem here. Chelsio 10Gbp card can only get about 460 Mps through 23.1.
I've wasted the better part of a day getting back to 22.05
Any update on this netgate"??
-
@superbree I don't have a Chelsio card but did a bit of searching and found
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#chelsio-cxgbe-4-cards
which has I think the same settings as
https://forums.freebsd.org/threads/intel-vs-chelsio-network-card-performance.70202/
...did you try those? (and restart)
-
@steveits I have had those settings enabled for quite some time on my rig, I checked that they carried over to 23.01 and they did. Doesn’t seem to matter.
I ended up rolling back to 22.05 via a boot environment after saving my config from 23.01 then I changed the repo version over to 22.05 and imported only the sections of the new config I had changes in (dhcp, rules, aliases).
Seems stable for now, I hope they fix this Chelsio issue and the Unbound issue I was having on 23.01 where it would drop dns upon first visit to a web page then 5 seconds later it would pull dns and display the page.
-
@steveits Yes. I've been running stable on 22.05 with an average of about 3.5 Gbps throughput before 23.01.
Before rolling back to 22.05, i did verify that the Chelsio fine tuning settings where correct and retained. It just won't go past 480 ~500 Mbps with a Chelsio Card on 23.01.
2.6 works fine. "update to 22.05, still functions as expected. Apply 23.01. Stuck at 500 Mbps.
Our network monitoring system indicated that the LAN port on the Chelsio card had over a million transmit discards on the LAN port while on 23.01
-
I also have had the exact symptoms. Running Intel I210 adapter for the WAN interface and a Chelsio T520-CR for the LAN 10gb backend. Was fine on 22.05 running ~940 down and up. After the upgrade to 23.01 I was getting ~470 download and ~940 upload. I do have an internal speedtest server and did a test from a computer (10gb nic) -> vlan hop through pfsense -> speedtest server (10gb nic) I was able to do 9GB up and down fine. But when I was doing a download from my file server on a separate computer (10gb server -> vlan hop through pfsenes -> 1gb nic computer this time) and was getting the 470Mbps speeds internally. That conversion from 10gb to 1gb on the Chelsio card doesn't seem to be working correctly. I re-installed 2.6 from scratch, pulled the backup (up to 940mbps again up and down), 22.01, then finally to 23.01. As soon as I hit 23.01 it again dropped to 470 down, 940 up. I did test by putting the WAN interface on the Chelsio card with the same behaviors (I didn't test upload speed at this point because it was getting late.) I kept all the same settings and I was also using the tuning options suggested by Pfsense for the Chelsio cards and those were retained across the upgrades. I did get a message about a kernel panic the second time I upgraded to 23.01 and tried to download the debug log which failed at the time and couldn't find the log before I had to re-image again to get it working late that night. I'll see if I can reproduce the panic this weekend so that I can get it posted.
-
So. I did install fresh (2.6) upgraded to 22.01, then 22.05 (pulled the config back during the process). Full gig speeds at this point. Upgraded to 23.01 and speeds was 470 down, 940 up again. Looked at making sure the card was on the latest firmware (1.27.0.0) which it was. Did a factory reset from the GUI to wipe any configurations that I had and take the config to stock to see if there was anything that I had configured that was causing the issue. Had to jump on the box for a minute to change the LAN port to the Chelsio card (WAN was still on Intel copper port) after the reset and once again did a speed test. 470 down, 940 up. I did see some interesting options on the ifconfig for the chelsio interface that I hadn't seen before. I turned off TXRTLMT and HWRXTSTMP with no change. I also tried to disable NOMAP but there was unable to disable it through ifconfig. This appears to be a driver issue and I have not been able to make a single change in the configuration to even affect the speeds. TSO and LRO were confirmed off by ifconfig. I didn't have any of the tunning options in the loader.conf file that I had before all of these changes were made. I did notice another post in the pfsense forums where this was happening during the development of 23.01 (https://forum.netgate.com/topic/177308/half-network-speed-after-upgrading) but apparently wasn't addressed before release. I'm hoping that this is something the developers are aware of/working on and would be willing to work with them on this.
-
Great detail shared there, thanks for the effort! FWIW, I think this is related to the T520 card specifically. I ordered/installed a T6225-CR (to replace my T520's) and I do NOT have performance degradation with that chipset.
Not sure if that's just by chance or if the drivers have been better optimized compared to the T520 cards? Perhaps the T520 cards are just getting a little long in the tooth? Total assumptions, but this is definitely not an isolated problem.
-
hoping that this is something the developers are aware of/working on
I would suggest looking for and/or opening an issue at Redmine.pfSense.org.
-
If it is an issue with the Chelsio NIC specifically it's probably a change in the driver. In that case though I'd expect to see more reports in FreeBSD or even a bug report there. Not seeing anything relevant though:
https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=chelsio -
I agree with Stephenw10 that this is probably something to do with the driver for the Chelsio t520 (possibly this specific line of cards). I pulled the card out this morning and put in a Intel X520 82599ES (pulled from a supermicro server) and installed that on the router and did the upgrade and was able to get full speeds through the intel card after the upgrade. (Also no errors reported on the interfaces. I was getting those before on the Chelsio card after the upgrade, but not consistently.) So not an issue with Pfsense as much as the driver in FreeBSD 14.0 (coming from 12-stable). I'll see if I can pick up a Chelsio T6 card in the future and test with that.
-
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
-
One other thing I also wanted to mention - there have been some changes to FreeBSD networking (routing) since FreeBSD 12. It looks like a lot changed in FreeBSD 13 so I assume all this is incorporated in FreeBSD 14 as well which pfSense 23.01 is based on:
https://papers.freebsd.org/2021/eurobsdcon/chernikov-routing_freebsd_13/
-
@tman222 said in Upgrade to 23.01 WAN speed halved:
I have been using a Chelsio T540-SO-CR for a few years now with pfSense and did not see any reduction in speed after upgrading to 23.01 - in fact, I saw a modest performance increase in certain cases. WAN connection here is symmetric 1 Gbit/s fiber, but the Chelsio is on the LAN side (WAN interface is Intel i210 I believe). Maybe that is the difference? What tunables do you guys have set for the Chelsio card?
In my last round of testing I wasn't using any tunable outside of what is baked into Pfsense. As stated earlier I even went as far as doing a factory reset in case I missed anything with the same result. Maybe an issue that only affects the Chelsio T520-CR cards then?
-
So... The Saga continues. I had to put the Chelsio card back in. The server that I put it into (where I got the intel card) didn't like it. I also tried with a Chelsio t420 card that
had lying around (hoping it would work better on 23.0)1 and it was also not getting past 470mbps. I will submit a bug report as suggested earlier and see if this can get figured out.
