Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Upgrade to 23.01 WAN speed halved

    Scheduled Pinned Locked Moved
    General pfSense Questions
    19
    106
    24.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      brachy33 @guygg
      last edited by

      @guygg

      I have not had the chance to update the firmware on one of X550-T2's yet, but will do so and test on my end. For the time being, I kept both my Chelsio cards installed, did a fresh install from 2.6CE to 22.05 (and installed the 23.01 security patches) and the system is just running super snappy, WAN speeds full symmetrical 950Mbps.

      Will grab another nvme SSD, update the X550 firmware, reinstall --> 23.01 and retest WAN speeds sometime this week to verify if it's my Nokia GPON ONT that is the issue with the updated drivers in 23.01 for the Chelsio (and potentially X550) and report back.

      1 Reply Last reply Reply Quote 1
      • D
        drewsaur
        last edited by

        Are any of you using traffic shaping? I have a post in the traffic shaping forum about my speed being cut in half with traffic shaping. I turned it off and that solved my problem. Unfortunately, due to another (known) bug, I can’t turn it back on to do more experimentation.

        B 1 Reply Last reply Reply Quote 0
        • B
          brachy33 @drewsaur
          last edited by

          @drewsaur

          Thanks for sharing. Unless traffic shaping is enabled by default on a fresh install (which I would assume No) then I am currently not.

          D 1 Reply Last reply Reply Quote 0
          • D
            drewsaur @brachy33
            last edited by

            @brachy33 Correct. It is not.

            1 Reply Last reply Reply Quote 0
            • D
              debo4479
              last edited by

              Can also confirm I am experiencing the same issue. I am running pfsense as VM on unRaid and passing through Ethernet controller: Chelsio Communications Inc T520-CR. I get half of my fios 1gb speed on Wan. Rolled back to 22.05 for now.

              1 Reply Last reply Reply Quote 0
              • G
                guygg
                last edited by

                @stephenw10 Not the best expert at what it should be reporting, but here are the interface lines from the boot log:

                ix0: <Intel(R) X550-T2> mem 0xd1000000-0xd11fffff,0xd1400000-0xd1403fff at device 0.0 on pci1
                ix0: Using 2048 TX descriptors and 2048 RX descriptors
                ix0: Using 4 RX queues 4 TX queues
                ix0: Using MSI-X interrupts with 5 vectors
                ix0: allocated for 4 queues
                ix0: allocated for 4 rx queues
                ix0: Ethernet address: ********************
                ix0: PCI Express Bus: Speed 8.0GT/s Width x4
                ix0: eTrack 0x80000492 PHY FW V523
                ix0: netmap queues/slots: TX 4/2048, RX 4/2048
                ix1: <Intel(R) X550-T2> mem 0xd1200000-0xd13fffff,0xd1404000-0xd1407fff at device 0.1 on pci1
                ix1: Using 2048 TX descriptors and 2048 RX descriptors
                ix1: Using 4 RX queues 4 TX queues
                ix1: Using MSI-X interrupts with 5 vectors
                ix1: allocated for 4 queues
                ix1: allocated for 4 rx queues
                ix1: Ethernet address: ******************
                ix1: PCI Express Bus: Speed 8.0GT/s Width x4
                ix1: eTrack 0x80000492 PHY FW V523
                ix1: netmap queues/slots: TX 4/2048, RX 4/2048

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  That's fine, 4 Tx/Rx queues on each. More than enough to handle 5Gbps unless your CPU is extremely low power.
                  What CPU is it? Does it report the expected clock speed?

                  1 Reply Last reply Reply Quote 0
                  • G
                    guygg
                    last edited by

                    @stephenw10 It's an i5-6500 with 8gb RAM running on a 256 NVMe. Under full load, the CPU doesn't break a sweat. Interrupts on the interfaces in the status screen continue to pile up. Any other things to check/tweak? May roll back to v22 soon so that I can get proper network performance back.

                    https://imgur.com/a/6hBSAn1

                    Thanks

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      If the CPU speed reported correctly by: sysctl dev.cpu | grep freq ?

                      I've seen one report of another 6th gen CPU stuck at 800MHz.

                      Seeing interrupts on the interfaces like that is expected. If it's moving a lot of traffic that rate isn't that high either. It would be interesting to see how that value compares in 22.05 but I suspect it will be similar.

                      Steve

                      G 1 Reply Last reply Reply Quote 0
                      • G
                        guygg @stephenw10
                        last edited by

                        @stephenw10 Hmmm, CPU freq seems OK to me:

                        dev.cpu.3.freq_levels: 3200/-1
                        dev.cpu.3.freq: 1152
                        dev.cpu.2.freq_levels: 3200/-1
                        dev.cpu.2.freq: 1194
                        dev.cpu.1.freq_levels: 3200/-1
                        dev.cpu.1.freq: 1149
                        dev.cpu.0.freq_levels: 3200/-1
                        dev.cpu.0.freq: 1190

                        And I wasn't aware of what those interrupt value should look like. Not something I ever really paid attention to when things were working well. :)

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Yup looks like what I expect with speed shift support, which is new in 23.01.

                          1 Reply Last reply Reply Quote 0
                          • B
                            brachy33
                            last edited by brachy33

                            Quick update. Installed a new X550-T2 card and flashed to latest firmware, installed and configured from scratch 2.60 CE-->22.05 with security patches-->23.01. WAN is connected to same Metronet GPON Nokia ONT and...full WAN performance is back. 950Mbps/950Mbps symmetrical.

                            Of course now...I'm seeing small errors IN forming on the WAN interface (I had this happening before with this NIC as WAN interface) but it's working at correct speeds. The errors IN on this card is why I went to the Chelsio T520-BT in the first place, and that card plays nice with my ONT, but the speeds were halved on 23.01 when I first upgraded.

                            Next step is to double check firmware on the Chelsio card, update Lenovo server BIOS to latest, then go through entire install process again and see if I can get the speeds on the Chelsio card to remain full-speed on 23.01.

                            UPDATE: With the system using the x550-T2 for WAN on 23.01... after a restart no more errors IN on WAN from GPON ONT! Almost 1TB of data transferred since that reboot and no errors on either WAN or LAN. Beauty!! Looks like 23.01 is gravvy.

                            1 Reply Last reply Reply Quote 1
                            • D
                              debo4479
                              last edited by debo4479

                              Update on my side. I had a spare Intel nic i211 that I passed through to use for my WAN.

                              Did a fresh install, dropped my config on it and brought it up to 22.05. I was getting full gigabit speeds download/upload.

                              Upgraded to 23.01 and again WAN download speeds are cut in half. I am seeing half WAN downloads speed on both Chelsio and Intel NIC. The Intel nic is at least giving me full upload speed. Could it be something in my config? I don't have anything exotic. And it works as expected on 22.05.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Do you have any sort of traffic shaping in play? That's about the only thing that might affect it like that.

                                D 1 Reply Last reply Reply Quote 0
                                • D
                                  debo4479 @stephenw10
                                  last edited by

                                  @stephenw10 no traffic shaping configured.

                                  B 1 Reply Last reply Reply Quote 0
                                  • B
                                    brachy33 @debo4479
                                    last edited by

                                    @debo4479

                                    Just a thought here and no idea if this is valid. But when I did my reinstall from 2.6-->23.01 I did not use a config backup. I manually reconfigured all my settings vs using the backup config...just to test (also my network config is pretty simple).

                                    Could there be something in your old configuration backup that's just not carrying over to 23.01 and causing issues?

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      debo4479
                                      last edited by

                                      I have a backup pfsense VM that I have on a spare poweredge server. It has the same config as my main server. I updated that to 23.01 and I get full 1gb upload/download. Only difference with main server is cpu and Broadcom nics.

                                      My next step is to setup a new vanilla pfsense install on my main server to test my Chelsio/i211 nics.

                                      1 Reply Last reply Reply Quote 1
                                      • K
                                        kanser-russ @brachy33
                                        last edited by

                                        I am also experiencing this issue on all interfaces of my Chelsio T422 after the upgrade to 23.01 iperf from pfSense to my Unraid boxes is capped around 460-480mbps same with testing from one Unraid box to another through different interfaces. Doesn't seem to matter if its a SFP+ or Gig copper port, I can't get over 480mbps.

                                        I will probably have to roll back to 22.05 until this is addressed. Not sure the best way to do so as I didn't notice the issue until after I made some changes to my configuration while on 23.01

                                        1 Reply Last reply Reply Quote 1
                                        • S
                                          superbree
                                          last edited by

                                          same problem here. Chelsio 10Gbp card can only get about 460 Mps through 23.1.

                                          I've wasted the better part of a day getting back to 22.05

                                          Any update on this netgate"??

                                          S 1 Reply Last reply Reply Quote 0
                                          • S
                                            SteveITS Rebel Alliance @superbree
                                            last edited by

                                            @superbree I don't have a Chelsio card but did a bit of searching and found

                                            https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#chelsio-cxgbe-4-cards

                                            which has I think the same settings as

                                            https://forums.freebsd.org/threads/intel-vs-chelsio-network-card-performance.70202/

                                            ...did you try those? (and restart)

                                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                            Upvote 👍 helpful posts!

                                            K S 2 Replies Last reply Reply Quote 1
                                            • First post
                                              Last post