• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question

Hardware
6
166
13.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L
    lnguyen @stephenw10
    last edited by Feb 12, 2025, 11:05 PM

    @stephenw10 Yes, if the 10GbE connection can negotiate 802.3x FC it will work around the issue with the caveat that all traffic will be paused in and out of the pfSense LAN interface everytime it receives a PAUSE frame.

    1 Reply Last reply Reply Quote 0
    • C
      ChuckLasher
      last edited by Feb 12, 2025, 11:37 PM

      I was slightly err'd in my post.

      For clarity, this is my configuration (as reported by pfSense) with Ethernet flow control active.
      The XB8 is connected at 2.5 G to ixl0 WAN 10G port,
      The 10G switch is connected to ilx1 LAN 10G port.

      login-to-view

      In this configuration, with ethernet flow control enabled, isn't it true that all my 10G switch's traffic is paused when any 1G port it serves needs it. This seems to be the case in my testing

      Therefore, I would like to know where the problem is; in the XB8 or the ixl0 port ?
      Is it possible to determine this?

      S 1 Reply Last reply Feb 12, 2025, 11:45 PM Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Feb 12, 2025, 11:41 PM

        Mmm, so if the upstream DOCSIS flow/congestion control is causing so much of an issue that TCP CC is shutting the window down you should be able to see that in a packet capture quite easily.

        Though I'm not sure it is since here we only see an issue with the 10G to 1G 'bridge' happening in the switch. 🤔

        Might be able to limit the burst rate from the pfSense LAN maybe....

        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator @ChuckLasher
          last edited by Feb 12, 2025, 11:45 PM

          @ChuckLasher said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

          Therefore, I would like to know where the problem is; in the XB8 or the ixl0 port ?
          Is it possible to determine this?

          Those interfaces only show rxpause not txpause. So if that's helping it can only be because the switch is telling the NIC to pause.

          But you should also be able to tell be looking at any error counters. They will show errors in on whichever device is exhausting it's buffer.

          1 Reply Last reply Reply Quote 0
          • C
            ChuckLasher
            last edited by ChuckLasher Feb 12, 2025, 11:58 PM Feb 12, 2025, 11:46 PM

            How do I enable txpause ? ( Not an internals expert )

            I'll grab whatever info you need.
            My switch doesn't show any buffer error type info.
            The best I can get is CRC errors

            In the netgear, I have this configured for ethernet flow control testing.

            login-to-view

            port number | name | port status | port speed | linked speed | flow control | max mtu

            L 1 Reply Last reply Feb 12, 2025, 11:58 PM Reply Quote 0
            • L
              lnguyen @ChuckLasher
              last edited by Feb 12, 2025, 11:58 PM

              @ChuckLasher

              Two ways to enable both 802.3x RX and TX

              • sysctl dev.ixl.1.fc=3

              • Create a System Tunable Entry with the variable being dev.ixl.1.fc and a value of 3

              N 1 Reply Last reply Feb 13, 2025, 12:19 AM Reply Quote 0
              • N
                ngr2001 @lnguyen
                last edited by Feb 13, 2025, 12:19 AM

                @lnguyen

                Isn't 3 the default value ?

                S 1 Reply Last reply Feb 13, 2025, 1:44 AM Reply Quote 0
                • N
                  ngr2001 @lnguyen
                  last edited by Feb 13, 2025, 12:26 AM

                  @lnguyen

                  Here are the results, no speed issues when connected directly to the modem, I did expect this though, being I have no issues when PFSense LAN and client is also at 1Gb regardless of the WAN NIC at 2.5 or 1Gb.

                  In this test a Win11 Laptop with a 1Gb NIC was plugged into my Netgear Modems Yellow 2.5Gb WAN Port. The Laptop of course negotiated a 1Gb Link.

                  Also Waveform is always stable, its speedtest.net and fast.com that really highlight the issue.

                  login-to-view

                  login-to-view

                  login-to-view

                  login-to-view

                  1 Reply Last reply Reply Quote 0
                  • C
                    ChuckLasher
                    last edited by Feb 13, 2025, 12:43 AM

                    @ngr2001

                    Can you check if flow control is enabled on your windows machine?

                    Google / AI states:

                    To check Ethernet flow control settings on Windows 11 using the command line, open a command prompt as administrator and type "Get-NetAdapterQos"; this PowerShell command will display the Quality of Service (QoS) capabilities and configurations of your network adapter, including flow control settings if supported by your hardware.

                    I don't have a way to check.
                    Might you be able to please ?

                    N 2 Replies Last reply Feb 13, 2025, 1:16 AM Reply Quote 0
                    • N
                      ngr2001 @ChuckLasher
                      last edited by Feb 13, 2025, 1:16 AM

                      @ChuckLasher

                      Get-NetAdapterQos -Name "*" | Where-Object -FilterScript { $_.Enabled }

                      Get-NetAdapterQos -Name "*" | Where-Object -FilterScript { $_.Enabled -Eq "False" }

                      Neither command is returning any results so far, still figuring this one out.

                      1 Reply Last reply Reply Quote 0
                      • N
                        ngr2001 @ChuckLasher
                        last edited by Feb 13, 2025, 1:26 AM

                        @ChuckLasher

                        So the Intel Driver on the laptop I used shows both Rx & Tx Flow Control Enabled. Whether or not its actually working I guess is a different story. But It seems like Windows 11 with most Intel and Realtek NICs enable Flow Control by default. Now whether or not the Netgear modem and Xfinity ISP properly support it I guess is another story.

                        Side note, my PFSense WAN NIC also shows Rx & Tx FC established when connected to the same modem.

                        FC working or not 1Gb to 1Gb speeds are fine and have always been fine, I wouldn't think FC is needed in this situation and even if enabled I would not expect it to kick in, so I am kind of wondering how much this test would help.

                        login-to-view

                        L 1 Reply Last reply Feb 13, 2025, 2:23 AM Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator @ngr2001
                          last edited by Feb 13, 2025, 1:44 AM

                          @ngr2001 said in Mixing different NIC Speeds (1Gb & 10Gb) Performance Problem Question:

                          Isn't 3 the default value

                          It is but that could have been changed. Also @ChuckLasher is using ixl NICs not ix so there are also driver differences.

                          1 Reply Last reply Reply Quote 0
                          • L
                            lnguyen @ngr2001
                            last edited by Feb 13, 2025, 2:23 AM

                            @ngr2001 If you can turn 802.3x FC off, we will know whether or not TCP Flow control works with your Windows PC directly connected to a DOCSIS network where the inbound from the coaxial is 2.35Gbps.

                            N 2 Replies Last reply Feb 13, 2025, 2:36 AM Reply Quote 1
                            • N
                              ngr2001 @lnguyen
                              last edited by Feb 13, 2025, 2:36 AM

                              @lnguyen

                              I'll give it a try in the am.

                              But 1 ? The modems port supports 2.5Gb and I pay for 2Gb service but when you plug a laptop with a 1Gb port Into that modem it will negotiate simply at 1Gb and give you a live ext IP. I guess you are thinking that if FC is disabled in windows and the speed test is poor then the isp supports and needs FC. However what If the speediest is fine, I guess what do the results mean in either case. I guess I'm hung up that there is no NAT on my end and we are live tapped in.

                              L 1 Reply Last reply Feb 13, 2025, 2:53 AM Reply Quote 0
                              • L
                                lnguyen @ngr2001
                                last edited by Feb 13, 2025, 2:53 AM

                                @ngr2001 If 802.3x FC is disabled and you still get 940/940, that means TCP Flow Control worked and it was able to tell the server to slow down so that the client doesn't end up with buffer overflow but still achieve its maximum potential. Which then points us back in the direction of pfSense. If it doesn't achieve 940/940, then we know TCP Flow Control doesn't work over DOCSIS and must rely on Layer 2 802.3x which is blah.....

                                1 Reply Last reply Reply Quote 0
                                • S
                                  stephenw10 Netgate Administrator
                                  last edited by Feb 13, 2025, 4:44 AM

                                  TCP flow/congestion control always 'works', the fact that it just slows down rather than fails is evidence that it is. The issue here is that other flow control methods at lower layers adversely affect the TCP CC to drop the speed more than is required as I understand it.
                                  In this situation though it appears that can only be the switch to pfSense link when it's overloaded by burst traffic. I can't see anything else that would explain it. It does feel like you might be able to some shaping in pfSense to mitigate it though. 🤔

                                  1 Reply Last reply Reply Quote 0
                                  • N
                                    ngr2001 @lnguyen
                                    last edited by Feb 13, 2025, 7:52 PM

                                    @lnguyen

                                    Just ran a new test with Flow Control Hard Disabled in Windows.

                                    The result was no difference in speed, but again I expect this. The modems port although being 2.5Gb ready can auto negotiate either 2.5Gb or 1Gb. In my test its negotiating 1Gb to the laptop thus I would expect this to be a very clean 1:1 connection with no speed mismatches to cause an any issues.

                                    Flow Control Off:
                                    login-to-view
                                    Flow Control Off:
                                    login-to-view
                                    Flow Control Off:
                                    login-to-view

                                    L 1 Reply Last reply Feb 13, 2025, 8:11 PM Reply Quote 0
                                    • L
                                      lnguyen @ngr2001
                                      last edited by Feb 13, 2025, 8:11 PM

                                      @ngr2001 Thanks. This was very helpful to me to eliminate DOCSIS being the cause of any TCP Flow Control failure. This points us back to the pfSense and how its interacting with switches at 1GbE and 10GbE. I have been analyzing packet captures at the client. Next will be analyzing packet capture of the pfSense LAN interface

                                      N 1 Reply Last reply Feb 16, 2025, 2:59 AM Reply Quote 0
                                      • N
                                        ngr2001 @lnguyen
                                        last edited by Feb 16, 2025, 2:59 AM

                                        @lnguyen

                                        I'd love to test the problematic setup but with a different firewall in place like a Ubiquiti Dream Machine Pro Max. Have you or anyone else tried that, this would at least help pinpoint whether or not there is a potential problem on the PFSense side, worth a shot ?

                                        L 1 Reply Last reply Feb 16, 2025, 6:03 AM Reply Quote 0
                                        • L
                                          lnguyen @ngr2001
                                          last edited by Feb 16, 2025, 6:03 AM

                                          @ngr2001 It's worth a try.

                                          1 Reply Last reply Reply Quote 0
                                          119 out of 166
                                          • First post
                                            119/166
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.