Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.5Gbps NICs only getting 1.5Gbps

    Scheduled Pinned Locked Moved General pfSense Questions
    26 Posts 4 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RobbieTTR
      RobbieTT @Stewart
      last edited by

      @Stewart

      There are a number of BSD and pfSense tuning guides for higher bandwidth. Having waded through a number of them most seem to have been applied to pfSense in subsequent releases.

      Other I have yet to test include:

      For PPPoE WANs with Multi-Queue NICs (single core issue) an entry for net.isr.dispatch=deferred
      can lead to performance gains on affected hardware.
      pfSense default in sysctl = net.isr.dispatch=direct
      net.isr.dispatch="deferred"
      
      
      To disable flow control on all ix interfaces. For LAN use the default of “3” is appropriate.
      pfSense default in sysctl = hw.ix.flow_control: 3
      hw.ix.flow_control="0"
      
      
      To disable flow control on individual igc interface (eg igc.3 when used as WAN).
      pfSense default in sysctl = dev.igc.3.fc: 3
      dev.igc.3.fc="0"
      
      
      Increasing hw.igc.max_interrupt_rate of up to 20000 provides gains in some scenarios.
      pfSense default in sysctl = hw.igc.max_interrupt_rate: 8000
      hw.igc.max_interrupt_rate=“16000"
      
      
      Some FreeBSD tuning guides suggest removing the igc rx_process_limit for higher performance.
      Maximum number of received packets to process at a time. Default is 100 packets. A value of -1 means
      unlimited.
      pfSense default in sysctl = hw.igc.rx_process_limit=100
      hw.igc.rx_process_limit="-1"
      
      
      Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)to a value larger than
      the default value of 1500.  Best throughput results are seen with a large MTU; use 9706 if possible (ref Intel) or 
      use 9216 (ref iflib stated maximum). Use the ifconfig command to confirm or increase the MTU size.
      To confirm the MTU used between two specific devices, use: route get <destination_IP_address>
      To set MTU on an ix interface for LAN use, enter the following where 1 is the interface number:
      ifconfig ix1 mtu 9216
      
      
      Doubling (or more) IP Input Queue (intr_queue) length on highly loaded systems may be required, if 
      queue drops are above zero. To check for queue drops (ideally adjust to a zero value) via CLI use command:
      sysctl net.inet.ip.intr_queue_drops
      net.inet.ip.intr_queue_drops: 0
      pfSense default in sysctl = net.inet.ip.intr_queue_maxlen: 1000
      net.inet.ip.intr_queue_maxlen="2000"
      
      
      For RJ45 interface stability energy efficient ethernet (eee) may need to be disabled.
      The default pfSense setting of 1 means disabled (which is somewhat counterintuitive) and 0 means enabled.
      pfSense (eg igc.3) default in sysctl = dev.igc.3.eee_control=1
      dev.igc.3.eee_control=“1”
      

      I attach no weight to the above; more akin to my musing than anything else - your issue is probably elsewhere but you never know.

      ☕️

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Yes, pretty much the only difference there is NAT vs no NAT.

        Try increasing the interrupt rate. That is a known limitation in igc.

        S 1 Reply Last reply Reply Quote 0
        • S
          Stewart @stephenw10
          last edited by

          @stephenw10 @RobbieTT
          I added hw.igc.max_interrupt_rate with a value of 16000 in the Tunables page but it didn't make a difference. Is there a tuning guide for IGC NICs? Most everything I'm finding is for IGB.

          Also, the slowdown isn't related to LAN vs WAN as I previously inidicated. I'm creating a Matrix of ports and iperf and it kind of appears random. Most of the time it is the 1.6Gbps but sometimes I get the ~2.4Gbps. While I thought it was reproducible before I can no longer reproduce it consistently.

          RobbieTTR 1 Reply Last reply Reply Quote 0
          • RobbieTTR
            RobbieTT @Stewart
            last edited by

            @Stewart
            My examples in the scroll box are only ix or igc, no igb here!

            ☕️

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              hw.igc.max_interrupt_rate is a loader value, it needs to be added in /boot/loader.conf.local. Create that file if required.

              S 1 Reply Last reply Reply Quote 0
              • S
                Stewart @stephenw10
                last edited by Stewart

                @stephenw10 I had a fleeting thought that might be the case. How can I tell the difference if something goes in the tuneables or if it goes into the loader.conf,local file?

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by stephenw10

                  Check the man page: https://man.freebsd.org/cgi/man.cgi?query=igc

                  In general though hw.X values are loader variables. dev.X are sysctls. But not always! 😉 And you can usually read the loader values as sysctls after boot but not set them.

                  S 1 Reply Last reply Reply Quote 1
                  • S
                    Stewart @stephenw10
                    last edited by Stewart

                    @stephenw10 Clear as mud! Thanks! I need to go re-rack and add a switch. Hopefully I can work on this later today.

                    Edit: When I put that in I get this message on boot:
                    Setting up extended sysctls...sysctl: oid 'hw.igc.max_interrupt_rate is a read only tunable
                    sysctl: Tunablevalues are set in /boot/loader.conf
                    sysctl: oid 'hw.igc.max_interrupt_rate' is a read only tunable
                    sysctl: Tunable values are set in /boot/loader.conf
                    done.

                    Adding the setting didn't change the speeds but it may be because I've done it wrong if theses errors indicate anything. I put it into the file /boot/loader.conf.local.

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      You're seeing that error because it's still set in System Tunables (sysctls). If it's set correctly in loader.conf.local it will show as that value when read via sysctl after boot.

                      1 Reply Last reply Reply Quote 0
                      • S
                        Stewart
                        last edited by

                        This doesn't appear to be a pfSense thing. I've switched over to Windows and am getting the same speed results. So far the advice I've received from HUNSN is to enable Turbo, but Turbo is enabled by default so it isn't that. I'll update this thread if I get it solved or if anyone has experience with these units or has any idea what BIOS modifications to try, please let me know.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Something in the way it's connected? PCIe bus using enough channels?

                          S 1 Reply Last reply Reply Quote 0
                          • S
                            Stewart @stephenw10
                            last edited by

                            @stephenw10 The connections are about as simple as it comes. Right now it's just a PC with a 10Gbe nic plugged directly into the HUNSN box with a CAT6 cable. Link speed is 2.5Gbps. Still just getting the 1.59Gbps speeds in iperf. Using pfSense it was 2 PCs connected to 2 ports on the HUNSN running iperf on each of the boxes. Still just 1.59Gbps.

                            With the 2 PCs connected directly to each other they link up at 10Gbps and I get iperf around 8Gbps.

                            w0wW 1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Sorry I meant how they're connected internally. Looks like the n100 doesn't have any NIC on the SoC so I assume they are PCIe connected. Though even 1 channel of PCIev1 should pass more than 1.5Gbps....

                              1 Reply Last reply Reply Quote 0
                              • w0wW
                                w0w @Stewart
                                last edited by

                                @Stewart

                                Please provide output of those commands

                                pciconf -lcv igc0
                                pciconf -lcv igc1
                                pciconf -lcv igc2
                                pciconf -lcv igc3
                                pciconf -lcv igc4
                                
                                
                                S 1 Reply Last reply Reply Quote 1
                                • S
                                  Stewart @w0w
                                  last edited by Stewart

                                  @w0w I sure wish these had console ports. I'll try to type them out.

                                  igc0@pci0:1:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x08086 subdevice=0x0000
                                  vendor = 'Intel Corporation'
                                  device = 'Ethernet Controller'
                                  class = network
                                  subclass = ethernet
                                  cap 01[40] = powerspec 3 supports D0 D3 current D0
                                  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
                                  cap 11[70] = MSI-X supports 5 messages, enabled. Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
                                  cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                                  Max Read 512
                                  link x1(x1) speed 2.5(5.0) ASPM disabled (L1)
                                  ecap 0001[100] = AER 2 0 fatal 1 non-fatal 1 corrected
                                  ecap 0003[140] = Serial 1 00e259ffff005232
                                  ecap 0018[1c0] = LTR 1
                                  ecap 001f[1f0] = Precision Time Measurement 1
                                  ecap 001e[1e0] = L1 PM Substates 1

                                  igc1@pci0:2:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x08086 subdevice=0x0000
                                  vendor = 'Intel Corporation'
                                  device = 'Ethernet Controller'
                                  class = network
                                  subclass = ethernet
                                  cap 01[40] = powerspec 3 supports D0 D3 current D0
                                  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
                                  cap 11[70] = MSI-X supports 5 messages, enabled. Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
                                  cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                                  Max Read 512
                                  link x1(x1) speed 2.5(5.0) ASPM disabled (L1)
                                  ecap 0001[100] = AER 2 0 fatal 1 non-fatal 1 corrected
                                  ecap 0003[140] = Serial 1 00e259ffff005233
                                  ecap 0018[1c0] = LTR 1
                                  ecap 001f[1f0] = Precision Time Measurement 1
                                  ecap 001e[1e0] = L1 PM Substates 1

                                  igc2@pci0:3:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x08086 subdevice=0x0000
                                  vendor = 'Intel Corporation'
                                  device = 'Ethernet Controller'
                                  class = network
                                  subclass = ethernet
                                  cap 01[40] = powerspec 3 supports D0 D3 current D0
                                  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
                                  cap 11[70] = MSI-X supports 5 messages, enabled. Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
                                  cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                                  Max Read 512
                                  link x1(x1) speed 5.0(5.0) ASPM disabled (L1)
                                  ecap 0001[100] = AER 2 0 fatal 1 non-fatal 1 corrected
                                  ecap 0003[140] = Serial 1 00e259ffff005234
                                  ecap 0018[1c0] = LTR 1
                                  ecap 001f[1f0] = Precision Time Measurement 1
                                  ecap 001e[1e0] = L1 PM Substates 1

                                  igc3@pci0:1:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x08086 subdevice=0x0000
                                  vendor = 'Intel Corporation'
                                  device = 'Ethernet Controller'
                                  class = network
                                  subclass = ethernet
                                  cap 01[40] = powerspec 3 supports D0 D3 current D0
                                  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
                                  cap 11[70] = MSI-X supports 5 messages, enabled. Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
                                  cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                                  Max Read 512
                                  link x1(x1) speed 5.0(5.0) ASPM disabled (L1)
                                  ecap 0001[100] = AER 2 0 fatal 1 non-fatal 1 corrected
                                  ecap 0003[140] = Serial 1 00e259ffff005235
                                  ecap 0018[1c0] = LTR 1
                                  ecap 001f[1f0] = Precision Time Measurement 1
                                  ecap 001e[1e0] = L1 PM Substates 1

                                  igc4@pci0:5:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x08086 subdevice=0x0000
                                  vendor = 'Intel Corporation'
                                  device = 'Ethernet Controller'
                                  class = network
                                  subclass = ethernet
                                  cap 01[40] = powerspec 3 supports D0 D3 current D0
                                  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
                                  cap 11[70] = MSI-X supports 5 messages, enabled. Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
                                  cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                                  Max Read 512
                                  link x1(x1) speed 5.0(5.0) ASPM disabled (L1)
                                  ecap 0001[100] = AER 2 0 fatal 1 non-fatal 1 corrected
                                  ecap 0003[140] = Serial 1 00e259ffff005236
                                  ecap 0018[1c0] = LTR 1
                                  ecap 001f[1f0] = Precision Time Measurement 1
                                  ecap 001e[1e0] = L1 PM Substates 1

                                  w0wW 1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    You can copy/paste that out of an SSH session.

                                    S 1 Reply Last reply Reply Quote 0
                                    • w0wW
                                      w0w @Stewart
                                      last edited by

                                      @Stewart
                                      Three ports are full speed pcie 5Gbit/s and two are only 2.5Gbit/s.
                                      You need to find out what ports are 5Gbit and re-test only using those ports.
                                      For me, it looks like some hardware limitation. I can only suppose, that those two ports with 2.5 speed are using some pcie hub, not CPU one, but some external.

                                      1 Reply Last reply Reply Quote 2
                                      • S
                                        Stewart @stephenw10
                                        last edited by

                                        @stephenw10 I suppose I could have. Our tech benches are currently full and I'm setting this up in the conference room to give me space. I don't really have all the same amenities over there.

                                        @w0w I think you've hit the nail on the head. And since I was flipping the unit over and back to swap between Windows and pfSense I must have been confused which port was which when I was programming them. All the going back and forth is what made go from consistent to inconsistent. I guess I just didn't track well enough.

                                        igc0 - link x1(x1) speed 2.5(5.0) ASPM disabled (L1)
                                        igc1 - link x1(x1) speed 2.5(5.0) ASPM disabled (L1)
                                        igc2 - link x1(x1) speed 5.0(5.0) ASPM disabled (L1)
                                        igc3 – link x1(x1) speed 5.0(5.0) ASPM disabled (L1)
                                        igc4 – link x1(x1) speed 5.0(5.0) ASPM disabled (L1)

                                        S 1 Reply Last reply Reply Quote 0
                                        • S
                                          Stewart @Stewart
                                          last edited by

                                          @Stewart I've been in communication with HUNSN and they have sent me a BIOS update. I'll report back when I get a chance to install it.

                                          S 1 Reply Last reply Reply Quote 2
                                          • S
                                            Stewart @Stewart
                                            last edited by

                                            @Stewart The new BIOS fixed the issue so now all 5 ports run at 2.5Gbe. It also sets much higher default PL1, PL2, and PL4 settings. The temps still stay in normal parameters but isn't the most efficient. I've run 17 different combinations of PL1 and PL2 with the default PL4 of 33W to find the best speed and then started adjusting PL4 to fine tune. Maybe not the best way of doing it but that was my process. Overall I found the best combination of speed and power for iperf over OpenVPN is:
                                            PL1=9
                                            PL2=10
                                            PL4=30

                                            Dropping PL1 to 8 impacts performance about 60Mbps but doesn't reduce heat or power.
                                            Dropping PL2 to 8 also reduces performance but doesn't reduce heat or power.
                                            PL4 default is now 33. Lowering it to 30 reduces temps from 69C to 61-62C and lowers speed from 575Mbps to 550Mbps. Lowering it to 29 reduces speed to 490 and keeps temp at 61C so no real change.

                                            Temp was determined by running iperf -P4 -t 300 and seeing what the temp was just before the end of the run. Everything seemed to idle the same no matter what the settings were at around 45C. Skin of the unit is always warm to the touch. I have a thermometer that I've set on top that generally reads 33C-34C (around 95F). It also doesn't seem to change much whether it is idle or under heavy load. The unit idles around 11W no matter the settings and, depending on PL settings, only goes up to 15-18W (most all settings showed 15W as the load limit for the iperf tests). For reference it does spike into the mid-20's in Windows.

                                            Speed was determined by the average of the last 10 seconds of the 5 minute test. I felt that I had to do this as the tests generally started out very high, in the 1.2Gbps-1.4Gbps range, and then fell over the course of the first minute to settle around the 5 minute average. Sometimes it would hold that for 5-10 seconds, some times for over 40 seconds. No idea unless it's some kind of TAU setting allowing the assigned core to spike for varying amounts of times.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.