Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

    Scheduled Pinned Locked Moved Hardware
    166 Posts 6 Posters 28.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L
      lnguyen @ngr2001
      last edited by

      @ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

      The issue with my old 3650 was that it only had 2x 10Gb ports, with the 3850 having 12x I am thinking this may be my best path forward.

      That is why I recommended that to you in the first place. The larger buffers don't completely resolve the issue, but makes it a lot better:

      sudo ethtool enp110s0 | grep Speed
      	Speed: 1000Mb/s
      
      speedtest -s 1783
      
         Speedtest by Ookla
      
            Server: Comcast - San Francisco, CA (id: 1783)
               ISP: Comcast Cable
      Idle Latency:    13.45 ms   (jitter: 1.66ms, low: 10.66ms, high: 14.05ms)
          Download:   827.00 Mbps (data used: 743.6 MB)                                                   
                       16.85 ms   (jitter: 10.94ms, low: 8.75ms, high: 273.13ms)
            Upload:   353.04 Mbps (data used: 384.2 MB)                                                   
                       16.75 ms   (jitter: 1.10ms, low: 12.49ms, high: 35.54ms)
       Packet Loss: Not available.
      
      1 Reply Last reply Reply Quote 0
      • L
        lnguyen
        last edited by

        An interesting datapoint that makes me point the finger to DOCSIS is that I have a secondary WAN connection through Sail Internet. If I force the traffic for this 1GbE client through my WAN2 with a rule--it reaches 940/940Mbps

        speedtest
        
           Speedtest by Ookla
        
              Server: Sail Internet - Santa Clara, CA (id: 56367)
                 ISP: Sail Internet
        Idle Latency:     1.34 ms   (jitter: 0.06ms, low: 1.14ms, high: 1.38ms)
            Download:   937.62 Mbps (data used: 423.4 MB)                                                   
                         34.14 ms   (jitter: 7.94ms, low: 0.81ms, high: 289.02ms)
              Upload:   938.43 Mbps (data used: 422.4 MB)                                                   
                        142.21 ms   (jitter: 75.57ms, low: 0.90ms, high: 825.07ms)
         Packet Loss:     0.0%
        
        1 Reply Last reply Reply Quote 0
        • L
          lnguyen @ngr2001
          last edited by lnguyen

          @ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

          I am going to try this MTU thing here in a sec, curious to what happens.

          Doesn't do anything. I already tried everything on the pfSense that I could think of:

          • Disable Hardware Checksum Offloading
          • Changing the State Type from Keep to Loose
          • Setting MSS & MTUs to 1420
          • Disabling pf Scrub

          The only solution that I don't like (which is not supported on the Cisco) is Ethernet Flow Control on both the WAN and LAN interfaces of the pfSense. Again a very ugly blunt solution that pauses the entire interface impacting other clients on your network.

          N 1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator @lnguyen
            last edited by

            @lnguyen said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

            @stephenw10 Do you actually have Cable Internet? Or lucky enough to have standard AT&T Fiber ethernet?

            I don't have either. I'm not in the US. 😉

            That is interesting that adding the switch would make any difference to a docsis issue though. Hard to see how that could happen. 🤔

            L N 2 Replies Last reply Reply Quote 0
            • N
              ngr2001 @lnguyen
              last edited by

              @lnguyen

              I just noticed something odd, when I moved the PF LAN nic from 1Gb to 10Gb the GUI is no longer showing Flow Control being enabled. I would expect to see rxpause & txpause.

              9d4ae57d-788e-401f-9037-339ca98776bf-image.png

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                What NIC is it? Not all drivers report it so it can be displayed there.

                L N 2 Replies Last reply Reply Quote 0
                • L
                  lnguyen @stephenw10
                  last edited by

                  @stephenw10 Using a switch with larger buffers only mask the symptoms of broken TCP Flow Control. This isn't a pfSense issue. It's a DOCSIS issue when the WAN speed is greater than a LAN client, ergo 2.35Gbps WAN to 940Mbps LAN

                  N 1 Reply Last reply Reply Quote 0
                  • N
                    ngr2001 @stephenw10
                    last edited by

                    @stephenw10

                    My LAN MTU is now 1460 in PF, so far no change is certain speedtests like fast.com when PF lan @ 10Gb. Speeds still slowly tapering down to 500Mbps. However with MTU at 1460 and PF Lan at 1Gb performance is still solid.

                    df59a454-a701-4812-8717-29b819e240c5-image.png

                    stephenw10S 1 Reply Last reply Reply Quote 0
                    • L
                      lnguyen @stephenw10
                      last edited by

                      @ngr2001 Yes what NIC are you using on the pfSense?

                      1 Reply Last reply Reply Quote 0
                      • N
                        ngr2001 @stephenw10
                        last edited by

                        @stephenw10

                        Intel X550-T2

                        When that same LAN NIC is moved to 1Gb is reports (rxpause & txpause) when I move the patch cable over to 10Gb these values disappear in the GUI.

                        L 1 Reply Last reply Reply Quote 1
                        • L
                          lnguyen @ngr2001
                          last edited by

                          @ngr2001 Can you do a show int 10GigabitEthernet1/2/8?

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Ah, that seems like a pretty big clue....

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              The 7250 doesn't have any 10GbaseT ports though so I assume that's a different Intel NIC?

                              L 1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator @ngr2001
                                last edited by

                                @ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

                                My LAN MTU is now 1460 in PF, so far no change is certain speedtests

                                Did you set MSS like I suggested though? MTU does nothing to pf.

                                N 1 Reply Last reply Reply Quote 0
                                • L
                                  lnguyen @stephenw10
                                  last edited by

                                  @stephenw10 Copper 10G-BaseT SFP+ modules most likely being used

                                  N 1 Reply Last reply Reply Quote 1
                                  • N
                                    ngr2001 @lnguyen
                                    last edited by

                                    @lnguyen

                                    Yes I have SPF+ RJ45 adapters in the 7250, thus I can easily move the CAT8 from one jack to the over for this test.

                                    L 1 Reply Last reply Reply Quote 0
                                    • L
                                      lnguyen @ngr2001
                                      last edited by

                                      @ngr2001 Can you do a show int 10GigabitEthernet1/2/8?

                                      N 1 Reply Last reply Reply Quote 0
                                      • N
                                        ngr2001 @stephenw10
                                        last edited by ngr2001

                                        @stephenw10

                                        I did not, but I just corrected that. I reset the states, waited a few mins and so far there may be a small boost in performance. I seem to be averaging now 700Mpbs instead of 500Mbps on fast.com, however if I switch ports we get the full 900+. So better but not perfect.

                                        64251468-6ed7-46ed-8090-ba5a1bd08939-image.png

                                        1 Reply Last reply Reply Quote 0
                                        • N
                                          ngr2001 @lnguyen
                                          last edited by

                                          @lnguyen

                                          1/2/8 - My 10Gb uplink to PF LAN Nic.
                                          689ff83c-9756-4b10-8dcf-72af121c1c0e-image.png

                                          I just noticed that 2 of my main clients currently at 1Gb on the switch have a lot of dropped packets. However right now I am running speedtests and I am not seeing the dropped packet counter increasing, so who know under which setup all these packets were dropped.

                                          1Gb Win 11 Client
                                          f5181091-f64a-483e-9e0a-69214260fd35-image.png

                                          1Gb Win 11 Client
                                          2ffd20e3-6932-412c-9007-0460e4980f0d-image.png

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            ngr2001 @lnguyen
                                            last edited by

                                            @lnguyen

                                            Should I disable flow control on my WAN NIC as a long term production setup being I am on Cable Internet ?

                                            L 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.