• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

Scheduled Pinned Locked Moved Hardware
166 Posts 6 Posters 35.9k Views 5 Watching
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J Offline
    johnpoz LAYER 8 Global Moderator @keyser
    last edited by Feb 10, 2025, 3:17 PM

    @keyser said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

    With the 3850

    hahah - yeah you would also have what sounds like a jet taking off where ever you put it ;)

    An intelligent man is sometimes forced to be drunk to spend time with his fools
    If you get confused: Listen to the Music Play
    Please don't Chat/PM me for help, unless mod related
    SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1

    1 Reply Last reply Reply Quote 0
    • N Offline
      ngr2001 @johnpoz
      last edited by Feb 10, 2025, 3:28 PM

      @johnpoz

      In this setup all clients perform great, 900+Mbps sustained speedtests, no issues.
      808f834c-1b54-4557-9ebc-cadf305d823b-image.png

      In this setup those 1Gb clients have issues, speedtest starts off strong and quickly drops to 500-600Mbps.
      75a97e3b-1772-46d0-9509-d674630e971d-image.png

      In this test both PFSense LAN NIC and the Client is at 10Gb, is this scenario the clients performance is also perfect, hitting 1.9Gbps sustained speedtest scores. The WAN is at 2.5Gb.
      6e59a061-c6d1-43d1-a30e-1f502afa8c03-image.png

      J 1 Reply Last reply Feb 10, 2025, 3:38 PM Reply Quote 0
      • J Offline
        johnpoz LAYER 8 Global Moderator @ngr2001
        last edited by Feb 10, 2025, 3:38 PM

        @ngr2001then yeah that sure seems like its switch related to me.

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 25.07.1 | Lab VMs 2.8.1, 25.07.1

        1 Reply Last reply Reply Quote 0
        • S Offline
          stephenw10 Netgate Administrator
          last edited by Feb 10, 2025, 3:45 PM

          So do you see dropped packets in the switch? On which ports?

          N 1 Reply Last reply Feb 10, 2025, 4:12 PM Reply Quote 0
          • N Offline
            ngr2001 @stephenw10
            last edited by Feb 10, 2025, 4:12 PM

            @stephenw10

            Oddly, on the new Brocade switch I am not seeing any dropped packets, even when the performance drops.

            If you recall, the Cisco 3650 would record dropped packets like crazy until we modified the config.

            1 Reply Last reply Reply Quote 0
            • S Offline
              stephenw10 Netgate Administrator
              last edited by Feb 10, 2025, 4:12 PM

              Are you running the L2 or L3 firmware? L2 doesn't appear to offer the qos values but I'm not sure if that implies it doesn't use it..... 🤔

              N 1 Reply Last reply Feb 10, 2025, 4:13 PM Reply Quote 0
              • S Offline
                stephenw10 Netgate Administrator
                last edited by Feb 10, 2025, 4:13 PM

                If you haven't already you might as well try bumping the ip-qos-session value.

                1 Reply Last reply Reply Quote 0
                • N Offline
                  ngr2001 @stephenw10
                  last edited by ngr2001 Feb 10, 2025, 4:16 PM Feb 10, 2025, 4:13 PM

                  @stephenw10

                  On the Brocade ICX 7250 I have the Full Layer 3 Firmware / License.

                  Just the pure lack of community support and viable example documentation for the Brocade has me about to click buy-it now on a Cisco WS-C3850-12X48U-S.

                  Since I messed up once already, can anyone think of a better multigig switch than the WS-C3850-12X48U-S, by better I mean larger buffers to handle 10Gb traffic and mixed client speeds.

                  1 Reply Last reply Reply Quote 0
                  • S Offline
                    stephenw10 Netgate Administrator
                    last edited by Feb 10, 2025, 4:20 PM

                    I must say I find it almost impossible to believe that the 7250 can't handle this. I have an older 6450 here and have never seen problems like that with it. I have a 7250 also I just haven't found time to install it. Yet.

                    N L 2 Replies Last reply Feb 10, 2025, 4:33 PM Reply Quote 0
                    • N Offline
                      ngr2001 @stephenw10
                      last edited by Feb 10, 2025, 4:33 PM

                      @stephenw10

                      I agree, I 100% feel its fixable, there is just not much info or example configs floating around to go off of.

                      Even their VLAN setup feels confusing compared to Cisco.

                      1 Reply Last reply Reply Quote 0
                      • S Offline
                        stephenw10 Netgate Administrator
                        last edited by Feb 10, 2025, 5:03 PM

                        Ha, well you have to hand it to Cisco, they have successfully convinced the world that their UI is the only and best UI. Including the Cisco terms for things that already had perfectly good names. 😉

                        But a 10G trunk link between a router and switch with 1G downstream devices is pretty common setup. I'd expect to see numerous threads complaining about this yet....

                        N 1 Reply Last reply Feb 10, 2025, 6:06 PM Reply Quote 0
                        • N Offline
                          ngr2001 @stephenw10
                          last edited by Feb 10, 2025, 6:06 PM

                          @stephenw10

                          I wonder though, how many people go the length to properly test or verify their performance, even in my case I think the average person may have missed it.

                          I found this useful chart that attempts to document various switches and their buffer sizes, could come in handy for someone.
                          https://people.ucsc.edu/~warner/buffer.html

                          I noted that the ICX7250 is only 2MB and the Cisco 3650 and 3850 are 6MB. I got the brocade only to get additional 10Gb ports but with the numbers showing it has 1/3 the buffer size, its probably no wonder the issues I am seeing,. Granted, may be fixable with some type of QOS or buffer tuning but there is not a whole lot of memory to go around.

                          https://people.ucsc.edu/~warner/buffer.html

                          The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.

                          L 1 Reply Last reply Feb 11, 2025, 2:44 PM Reply Quote 1
                          • S Offline
                            stephenw10 Netgate Administrator
                            last edited by Feb 10, 2025, 7:53 PM

                            Mmm, just no real idea of how much buffer space might be needed...

                            I'll try and get mine setup for a test.

                            Do you still see it in a local test or only to the remote speedtest server?

                            N 1 Reply Last reply Feb 10, 2025, 7:58 PM Reply Quote 0
                            • N Offline
                              ngr2001 @stephenw10
                              last edited by Feb 10, 2025, 7:58 PM

                              @stephenw10

                              Just on remote speed test thus far

                              Try fast.com and speedtest.net

                              as dumb as fast.com is, its seems to highlight the issue immediately.

                              1 Reply Last reply Reply Quote 0
                              • S Offline
                                stephenw10 Netgate Administrator
                                last edited by Feb 10, 2025, 8:49 PM

                                Mmm, I hit that exact symptom recently and it turned out to be an MTU/MSS issue. The test at fast.com takes ages to start doing anything, then gives some crappy download figure and errors out on the upload.

                                Try setting an MSS value on the interface in pfSense as a test. I'd just use, say, 1460 to be sure.

                                N 1 Reply Last reply Feb 10, 2025, 8:58 PM Reply Quote 0
                                • N Offline
                                  ngr2001 @stephenw10
                                  last edited by Feb 10, 2025, 8:58 PM

                                  @stephenw10

                                  Well that's very interesting.

                                  All my MTU's are 1500 though, (PF WAN & LAN) all my switchports etc. Are you suggesting I change that everywhere to 1460, that seems like a pain for sure.

                                  1 Reply Last reply Reply Quote 0
                                  • S Offline
                                    stephenw10 Netgate Administrator
                                    last edited by stephenw10 Feb 10, 2025, 9:06 PM Feb 10, 2025, 9:04 PM

                                    Nope it only needs to be set in one place. pf will force it to that when it passes the traffic as long as pf-scrub is enabled. So I would add it on the pfSense internal interface.

                                    N 1 Reply Last reply Feb 10, 2025, 10:59 PM Reply Quote 0
                                    • N Offline
                                      ngr2001 @stephenw10
                                      last edited by Feb 10, 2025, 10:59 PM

                                      @stephenw10

                                      Wouldn't that cause packet fragmentation though, all the clients are going to be hitting the PF Lan Nic at a MTU of 1500.

                                      1 Reply Last reply Reply Quote 0
                                      • S Offline
                                        stephenw10 Netgate Administrator
                                        last edited by Feb 11, 2025, 12:50 AM

                                        No it should cause TCP to just send smaller packets. Where I hit it doing that completely resolved the issue even though it shouldn't have done anything as far as I could see. But something had broken PMTU. Took waaay too long to find it

                                        Hard to see how the 1G switch interface could do that but the symptoms you're seeing are so similar it's worth trying. It's trivial to test too.

                                        N 1 Reply Last reply Feb 11, 2025, 2:06 PM Reply Quote 0
                                        • N Offline
                                          ngr2001 @stephenw10
                                          last edited by Feb 11, 2025, 2:06 PM

                                          @stephenw10

                                          Ill try it, but what is "pf-scrub" how do I enable.

                                          L 1 Reply Last reply Feb 11, 2025, 2:19 PM Reply Quote 0
                                          40 out of 166
                                          • First post
                                            40/166
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received