Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Abysmal Performance after pfSense hardware upgrade

    Scheduled Pinned Locked Moved General pfSense Questions
    69 Posts 4 Posters 11.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 8
      8ayM @8ayM
      last edited by

      I'm just going to grab another m.2 and do a clean install of 2.7.2

      Only config I'll be making would be assigning a MAC address to the Wan interface as the ISP locks the router MAC to their OTN handoff.

      I'll be in the dark again while I get this setup.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Hmm, should be identical then. Could be a bad PHY/port?

        1 Reply Last reply Reply Quote 0
        • G
          Gblenn @stephenw10
          last edited by

          @stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

          You will see loading there from either the ix or igc NICs.

          What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.

          Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

          8 S 2 Replies Last reply Reply Quote 0
          • 8
            8ayM @Gblenn
            last edited by

            @Gblenn said in Abysmal Performance after pfSense hardware upgrade:

            @stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

            You will see loading there from either the ix or igc NICs.

            What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.

            Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

            This is the hardware offloading options I have selected
            4ac5f5eb-2c5b-47ec-8a8f-fd21f523fe28-image.png

            G 1 Reply Last reply Reply Quote 0
            • G
              Gblenn @8ayM
              last edited by Gblenn

              @8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
              Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated.

              8 1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                The altq setting only affects hn NICs.

                What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit q while it's running it leaves the output on the console so you can copy/paste it out.

                G 1 Reply Last reply Reply Quote 0
                • 8
                  8ayM @Gblenn
                  last edited by

                  64a9abc0-b04c-4485-9496-8aeca8e18349-image.png

                  Clean install

                  @Gblenn said in Abysmal Performance after pfSense hardware upgrade:

                  @8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
                  Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated.

                  I'll give this a shot and see if there is a difference

                  8 1 Reply Last reply Reply Quote 0
                  • S
                    SteveITS Galactic Empire @Gblenn
                    last edited by

                    @Gblenn said in Abysmal Performance after pfSense hardware upgrade:

                    Does HW offloading play a part in this, which I have activated as I'm running Suricata

                    https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
                    "11.2.3. Offloading

                    Network cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.

                    LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.

                    Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."

                    On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.

                    Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                    When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                    Upvote 👍 helpful posts!

                    8 1 Reply Last reply Reply Quote 0
                    • 8
                      8ayM @8ayM
                      last edited by 8ayM

                      Current Settings after reboot to apply
                      3d7b103f-f30e-4e87-ad8e-7ca42daf7d53-image.png

                      And the speed test results
                      a8eebcbb-d5eb-41ed-8bf1-e803f6295900-image.png
                      2c5ed710-1981-4180-89a2-70bc3fb9e40f-image.png

                      Enabled SSH
                      b5eb7ad5-cd59-439b-914d-3fad54997771-image.png

                      Ran top -HaSP
                      https://streamable.com/tjs0r1

                      1 Reply Last reply Reply Quote 0
                      • 8
                        8ayM @SteveITS
                        last edited by 8ayM

                        @Gblenn said in Abysmal Performance after pfSense hardware upgrade:

                        @stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

                        Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

                        My previous system had been on a dual sfp+ x520. I have another dual in my Desktop I've been testing with and FreeNAS box


                        @SteveITS said in Abysmal Performance after pfSense hardware upgrade:

                        @Gblenn said in Abysmal Performance after pfSense hardware upgrade:

                        Does HW offloading play a part in this, which I have activated as I'm running Suricata

                        https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
                        "11.2.3. Offloading

                        Network cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.

                        LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.

                        Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."

                        On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.

                        So is your thought/suggestion to check/disable these feature


                        The clean install's web UI is substantially more responsive than what I've grown accustomed to.

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          SteveITS Galactic Empire @8ayM
                          last edited by

                          @8ayM said in Abysmal Performance after pfSense hardware upgrade:

                          So is your thought/suggestion to check/disable these feature

                          Yes we check the three "offloading" checkboxes. Those need a restart.

                          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                          Upvote 👍 helpful posts!

                          8 1 Reply Last reply Reply Quote 0
                          • 8
                            8ayM @SteveITS
                            last edited by

                            Well I've gone through my stack of SFP+ Dac cables short of a 1' as I really don't have room to cable the unit, and be within reach for the short data cable.

                            So I broke out 2 x Gtek 10G SFP+ Modules and attached a OM4 fibre cable between the two
                            95453278-5c61-4453-bdcd-9775f7d9840b-image.png

                            I'm at a loss as the DAC's have worked up until I did this firewall upgrade.

                            @SteveITS said in Abysmal Performance after pfSense hardware upgrade:

                            @8ayM said in Abysmal Performance after pfSense hardware upgrade:

                            So is your thought/suggestion to check/disable these feature

                            Yes we check the three "offloading" checkboxes. Those need a restart.

                            I'll revert these changes in a moment and see how that makes a difference

                            Is this the state that your sugesting?
                            c09e28c2-a2ea-4e47-9de3-10cf82dcb77e-image.png

                            After that I'll slap in my original mirrored m.2's as originally intended and see how things fare.

                            You almost can't be a normal person with out spare parts to trouble shoot a lot of this. 99% of the time I'm just a packrat until events like this occur.

                            Although I would like to get back to DAC's for the lower power draw and heat.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              The ix NICs in the C3K chipset a missing the lines that allow reading the link status from SFP modules. That's why you see RJ-45-copper modules are not supported.

                              DAC cables normally work but you often see the link status shown as 'unknown'.

                              The X520 does not have that issue and I suspect that's what you're seeing here.

                              8 1 Reply Last reply Reply Quote 0
                              • 8
                                8ayM @stephenw10
                                last edited by

                                @stephenw10

                                I wasn't aware I couldn't utilize a SFP+ to RJ45 module

                                Making the changes as suggested by @SteveITS yielded similar results as before speed wise.
                                5dcfc343-094c-4d84-a58e-f12bc05030ba-image.png
                                I'm going to try swapping back to my mirrored m.2 and see how that goes. with the sfp+ modules and fiber back to the switching infrastructure.

                                1 Reply Last reply Reply Quote 1
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  For reference:
                                  https://downloadmirror.intel.com/732258/readme.txt

                                  In addition, SFP+ devices based on the Intel(R) Ethernet Connection X552 and
                                  Intel(R) Ethernet Connection X553 do not support the following features:
                                  * Speed and duplex auto-negotiation.
                                  * Wake on LAN
                                  * 1000BASE-T SFP Modules
                                  

                                  Though in reality we have seen some modules will work.

                                  8 1 Reply Last reply Reply Quote 0
                                  • 8
                                    8ayM @stephenw10
                                    last edited by

                                    @stephenw10
                                    Explains the Module not supported message I'm getting at the console for using a 10Gtek Ubiquity 10G SFP+ module.

                                    Might look at some new DAC's

                                    Anyway restored old config, made the hardware offloading changes as mentioned above, and things are looking better. Also remove traffic shaping as I'm struggling to image hitting that limit short of benchmarking.
                                    12e9dd2c-f7ef-44ee-bc1d-b501c6213233-image.png

                                    G 1 Reply Last reply Reply Quote 1
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      A 10G DAC cable will usually work in my experience. A DAC connected to 1G at the other end will almost always fail and doesn't allow setting 1G manually.

                                      8 1 Reply Last reply Reply Quote 0
                                      • G
                                        Gblenn @8ayM
                                        last edited by

                                        @8ayM said in Abysmal Performance after pfSense hardware upgrade:

                                        @stephenw10
                                        Explains the Module not supported message I'm getting at the console for using a 10Gtek Ubiquity 10G SFP+ module.

                                        Might look at some new DAC's

                                        Anyway restored old config, made the hardware offloading changes as mentioned above, and things are looking better. Also remove traffic shaping as I'm struggling to image hitting that limit short of benchmarking.
                                        12e9dd2c-f7ef-44ee-bc1d-b501c6213233-image.png

                                        So it seems the one thing that made the difference was that you turned off HW checksum offload (the first item in the list)??

                                        @SteveITS said in Abysmal Performance after pfSense hardware upgrade:

                                        @8ayM said in Abysmal Performance after pfSense hardware upgrade:

                                        So is your thought/suggestion to check/disable these feature

                                        Yes we check the three "offloading" checkboxes. Those need a restart.

                                        Is there no benefit at all having any of the HW offloading active, even with e.g. X520 NIC? I think I have always had all three turned on, on both my sites (other site has i211 NIC's),

                                        S 8 2 Replies Last reply Reply Quote 0
                                        • 8
                                          8ayM @stephenw10
                                          last edited by

                                          @stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

                                          A 10G DAC cable will usually work in my experience. A DAC connected to 1G at the other end will almost always fail and doesn't allow setting 1G manually.

                                          Threw on one of my 10g DAC's again just for giggles
                                          49111720-2bb2-49e7-845b-6732f002a441-image.png

                                          I'm going back to the modules after testing the 1' DAC. When not working on it I can get by with that. Otherwise I'll deal with the coil of fiber for now.

                                          1 Reply Last reply Reply Quote 0
                                          • G
                                            Gblenn @stephenw10
                                            last edited by

                                            @stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

                                            The altq setting only affects hn NICs.

                                            What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit q while it's running it leaves the output on the console so you can copy/paste it out.

                                            Yes absolutely, Suricata comes out at the top when I run speedtest...

                                            34748d43-74cd-4f7c-87d2-f28b7e4baf0d-image.png

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.