Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

    Scheduled Pinned Locked Moved Hardware
    166 Posts 6 Posters 18.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • keyserK
      keyser Rebel Alliance @ngr2001
      last edited by

      @ngr2001 ohh, that thread was started by you as well…

      The SSH command for flow control is in that thread, and there is som inspiration for changing buffers on a switch

      Love the no fuss of using the official appliances :-)

      N 1 Reply Last reply Reply Quote 0
      • N
        ngr2001 @keyser
        last edited by

        @keyser

        Lol totally forgot that, must have hit my head. Yes had the same issue with my Cisco switch it seems. I moved to a Brocade to get more 10Gb ports. Now I need to reproduce the same success.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Mmm, I would also check for MTU/MSS issues. They can present exactly like that.

          I'd be amazed if the ICX7250 had a problem with that. Though it has many config options, it could misconfigured to do it!

          N 1 Reply Last reply Reply Quote 0
          • N
            ngr2001 @stephenw10
            last edited by

            @stephenw10

            So I forgot you solved this issue once for me when I had a Cisco 3650.

            Seems like the Brocade ICX-7250 has the same issue but I find its CLI way more confusing and not as well documented as Cisco.

            The Cisco fix was:
            qos queue-softmax-multiplier 1200

            Brocade does not seem to have an equivalent that I can find, thus far I have tried.

            Enabling Flow Control on all the Brocade Ports - Result no difference

            Enabling "buffer-sharing-full" - Result no difference

            Perhaps Brocades QOS "ingress-buffer-profile" or "egress-buffer-profile" would do the trick but the documentation and google searching is not leading me anywhere with something I can try.

            If I cant get this working I may seriously consider getting a Cisco 3850, however I would like to get something that has 8Mb+ port buffers so I don't have to play this tuning game.

            My ICX 7250 Config:

            SSH@romulus>show run
            Current configuration:
            !
            ver 08.0.95pT213
            !
            stack unit 1
            module 1 icx7250-48-port-management-module
            module 2 icx7250-sfp-plus-8port-80g-module
            stack-port 1/2/1
            stack-port 1/2/3
            !
            vlan 1 name DEFAULT-VLAN by port
            router-interface ve 1
            !
            !
            symmetrical-flow-control enable
            !
            !
            optical-monitor
            optical-monitor non-ruckus-optic-enable
            aaa authentication web-server default local
            aaa authentication login default local
            enable aaa console
            hostname romulus
            ip dhcp-client disable
            ip dns server-address 10.0.0.1
            ip route 0.0.0.0/0 10.0.0.1
            !
            no telnet server

            !
            clock timezone us Eastern
            !
            !
            ntp
            disable serve
            server time.cloudflare.com
            !
            !
            no web-management http
            !
            manager disable
            !
            !
            manager port-list 987
            !

            !
            interface ethernet 1/1/4
            flow-control neg-on
            !
            interface ethernet 1/1/48
            flow-control neg-on
            !
            interface ethernet 1/2/1
            flow-control neg-on
            !
            interface ethernet 1/2/8
            flow-control neg-on
            !
            interface ve 1
            ip address 10.0.0.3 255.255.255.0
            !
            !
            end

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              I would also try specifically disabling flow-control on all interfaces in the path. We have seen cases where flow-control itself was the problem. I really wouldn't expect flow-control to be an issue here when there are 1G links both up and down stream limiting the flow already.

              N 1 Reply Last reply Reply Quote 0
              • N
                ngr2001 @stephenw10
                last edited by

                @stephenw10

                Last time I tried it made no difference but I'll try again. To me its clearly a Brocade issue, much like the Cisco issue I had but was able to fix with your help, I just cant find a comparable setting.

                I should have just bought a Cisco 3850 with the 12x multigig ports. I am seeing them on ebay for $99. At that price I may just buy one and give up on the brocade.

                With the 3850 I could have my WAN, LAN, and Win 11 clients all at 2.5Gb with a few remaining Win 11 clients at 1Gb. With the larger buffer and known QOS tweaks it would likely go a lot smoother for me.

                keyserK 1 Reply Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator @ngr2001
                  last edited by

                  @ngr2001 Ok so we are sure we are on the same page..

                  In this config where its 1ge from your switch to pfsense, a single client is able to get 900ish Mbps..

                  1ge.jpg

                  But in this config.. Where pfsense has 10ge to your switch.. A single client is only able to get 600Mbps?

                  10ge.jpg

                  Is there any way you can test this config?

                  client.jpg

                  Where the client has a connection that can do your wan 2Ge isp connection? ie 2.5 or 5 or 10 directly connected to just a single client?

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                  N 1 Reply Last reply Reply Quote 0
                  • keyserK
                    keyser Rebel Alliance @ngr2001
                    last edited by

                    @ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

                    @stephenw10

                    Last time I tried it made no difference but I'll try again. To me its clearly a Brocade issue, much like the Cisco issue I had but was able to fix with your help, I just cant find a comparable setting.

                    I should have just bought a Cisco 3850 with the 12x multigig ports. I am seeing them on ebay for $99. At that price I may just buy one and give up on the brocade.

                    With the 3850 I could have my WAN, LAN, and Win 11 clients all at 2.5Gb with a few remaining Win 11 clients at 1Gb. With the larger buffer and known QOS tweaks it would likely go a lot smoother for me.

                    It just borderline insane how cheap Cisco switches are used in the US..... You really must have a lot of shops that just rotates all the equipment on a schedule instead of actually looking at the value and lifetime the products offer.

                    Love the no fuss of using the official appliances :-)

                    johnpozJ 1 Reply Last reply Reply Quote 1
                    • johnpozJ
                      johnpoz LAYER 8 Global Moderator @keyser
                      last edited by

                      @keyser said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

                      With the 3850

                      hahah - yeah you would also have what sounds like a jet taking off where ever you put it ;)

                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                      If you get confused: Listen to the Music Play
                      Please don't Chat/PM me for help, unless mod related
                      SG-4860 24.11 | Lab VMs 2.8, 24.11

                      1 Reply Last reply Reply Quote 0
                      • N
                        ngr2001 @johnpoz
                        last edited by

                        @johnpoz

                        In this setup all clients perform great, 900+Mbps sustained speedtests, no issues.
                        808f834c-1b54-4557-9ebc-cadf305d823b-image.png

                        In this setup those 1Gb clients have issues, speedtest starts off strong and quickly drops to 500-600Mbps.
                        75a97e3b-1772-46d0-9509-d674630e971d-image.png

                        In this test both PFSense LAN NIC and the Client is at 10Gb, is this scenario the clients performance is also perfect, hitting 1.9Gbps sustained speedtest scores. The WAN is at 2.5Gb.
                        6e59a061-c6d1-43d1-a30e-1f502afa8c03-image.png

                        johnpozJ 1 Reply Last reply Reply Quote 0
                        • johnpozJ
                          johnpoz LAYER 8 Global Moderator @ngr2001
                          last edited by

                          @ngr2001then yeah that sure seems like its switch related to me.

                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                          If you get confused: Listen to the Music Play
                          Please don't Chat/PM me for help, unless mod related
                          SG-4860 24.11 | Lab VMs 2.8, 24.11

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            So do you see dropped packets in the switch? On which ports?

                            N 1 Reply Last reply Reply Quote 0
                            • N
                              ngr2001 @stephenw10
                              last edited by

                              @stephenw10

                              Oddly, on the new Brocade switch I am not seeing any dropped packets, even when the performance drops.

                              If you recall, the Cisco 3650 would record dropped packets like crazy until we modified the config.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Are you running the L2 or L3 firmware? L2 doesn't appear to offer the qos values but I'm not sure if that implies it doesn't use it..... 🤔

                                N 1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  If you haven't already you might as well try bumping the ip-qos-session value.

                                  1 Reply Last reply Reply Quote 0
                                  • N
                                    ngr2001 @stephenw10
                                    last edited by ngr2001

                                    @stephenw10

                                    On the Brocade ICX 7250 I have the Full Layer 3 Firmware / License.

                                    Just the pure lack of community support and viable example documentation for the Brocade has me about to click buy-it now on a Cisco WS-C3850-12X48U-S.

                                    Since I messed up once already, can anyone think of a better multigig switch than the WS-C3850-12X48U-S, by better I mean larger buffers to handle 10Gb traffic and mixed client speeds.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      I must say I find it almost impossible to believe that the 7250 can't handle this. I have an older 6450 here and have never seen problems like that with it. I have a 7250 also I just haven't found time to install it. Yet.

                                      N L 2 Replies Last reply Reply Quote 0
                                      • N
                                        ngr2001 @stephenw10
                                        last edited by

                                        @stephenw10

                                        I agree, I 100% feel its fixable, there is just not much info or example configs floating around to go off of.

                                        Even their VLAN setup feels confusing compared to Cisco.

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Ha, well you have to hand it to Cisco, they have successfully convinced the world that their UI is the only and best UI. Including the Cisco terms for things that already had perfectly good names. 😉

                                          But a 10G trunk link between a router and switch with 1G downstream devices is pretty common setup. I'd expect to see numerous threads complaining about this yet....

                                          N 1 Reply Last reply Reply Quote 0
                                          • N
                                            ngr2001 @stephenw10
                                            last edited by

                                            @stephenw10

                                            I wonder though, how many people go the length to properly test or verify their performance, even in my case I think the average person may have missed it.

                                            I found this useful chart that attempts to document various switches and their buffer sizes, could come in handy for someone.
                                            https://people.ucsc.edu/~warner/buffer.html

                                            I noted that the ICX7250 is only 2MB and the Cisco 3650 and 3850 are 6MB. I got the brocade only to get additional 10Gb ports but with the numbers showing it has 1/3 the buffer size, its probably no wonder the issues I am seeing,. Granted, may be fixable with some type of QOS or buffer tuning but there is not a whole lot of memory to go around.

                                            https://people.ucsc.edu/~warner/buffer.html

                                            The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.

                                            L 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.