Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Poor speeds with Chelsio T420-CR 10gb NIC

    Scheduled Pinned Locked Moved Hardware
    17 Posts 4 Posters 15.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      trumee
      last edited by

      Formatted the disk and re-installed with no customization i.e. 1 WAN and 1 LAN.

      Test1
      With pf running

      
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   116 MBytes   970 Mbits/sec    0    249 MBytes       
[  4]   1.00-2.00   sec   116 MBytes   970 Mbits/sec    0    249 MBytes       
[  4]   2.00-3.00   sec   115 MBytes   969 Mbits/sec    0    249 MBytes       
[  4]   3.00-4.00   sec   116 MBytes   969 Mbits/sec    0    249 MBytes       
[  4]   4.00-5.00   sec   116 MBytes   970 Mbits/sec    0    249 MBytes       
[  4]   5.00-6.00   sec   116 MBytes   970 Mbits/sec    0    249 MBytes       
[  4]   6.00-7.01   sec   117 MBytes   977 Mbits/sec    0    249 MBytes       
[  4]   7.01-8.00   sec   115 MBytes   976 Mbits/sec    0    249 MBytes       
[  4]   8.00-9.01   sec   117 MBytes   977 Mbits/sec    0    249 MBytes       
[  4]   9.01-10.00  sec   116 MBytes   976 Mbits/sec    0    249 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.13 GBytes   972 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.13 GBytes   972 Mbits/sec                  receiver

      Test2
      pf disabled, pfctl -d

      
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   189 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   1.00-2.00   sec   188 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   2.00-3.00   sec   188 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   4.00-5.00   sec   189 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   5.00-6.00   sec   188 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   6.00-7.00   sec   189 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   7.00-8.00   sec   188 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   8.00-9.00   sec   188 MBytes  1.58 Gbits/sec    0    247 MBytes       
[  4]   9.00-10.00  sec   189 MBytes  1.58 Gbits/sec    0    247 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.84 GBytes  1.58 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.84 GBytes  1.58 Gbits/sec                  receiver

      10gb is very much broken in pfsense on Supermicro C2758 platform

      1 Reply Last reply Reply Quote 0
      • KOMK
        KOM
        last edited by

        I'll probably never see one of these cards in my lifetime, but here are some links I've seen others post in relation to 10G issues:

        https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

        https://calomel.org/freebsd_network_tuning.html

        https://calomel.org/network_performance.html

        https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting

        1 Reply Last reply Reply Quote 0
        • T
          trumee
          last edited by

          Can somebody who has official C2758 based from pfsense post their system tunables and /boot/loader.conf. Clearly out of the box the router performs very poorly with 10gbe hardware.

          1 Reply Last reply Reply Quote 0
          • Q
            q54e3w
            last edited by

            I have two systems, one with Chelsio T520-CR-SO and another with Intel x520, neither has had any perf issues I've observed.

            Here's the A1SRM-2758f system (although not an official negate box) with T520-CR-SO tuneable which is the only one I can get to easily.

            
[2.3.2-RELEASE][admin@pfSense.local.lan]/root: cat  /boot/loader.conf
autoboot_delay="3"
comconsole_speed="115200"
hw.usb.no_pf="1"

            pf-10g.png
            pf-10g.png_thumb

            1 Reply Last reply Reply Quote 0
            • T
              trumee
              last edited by

              I compared the tunables with my system and there is no difference. Is there anything in /boot/loader.conf.local? And  what do you have in  System>Advanced>Networking>Network Interfaces?

              I am guessing you are getting 10gb speeds.

              Here is another test where i booted into vanilla FreeBSD-11 on the pfsense router
              Chelsio T420-CR (pfsense router running FreeBSD 11) and Mellanox ConnectX-2 EN (linux desktop)

              
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.03 MBytes
[  4]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

              So, the card works well in native FreeBSD-11. I then tested in pfSense-2.4 snapshot image which is based on FreeBSD-11. Again i got dismal speeds.

              1 Reply Last reply Reply Quote 0
              • Q
                q54e3w
                last edited by

                Nothing interesting…

                
[2.3.2-RELEASE][admin@pfSense.local.lan]/root: cat  /boot/loader.conf.local
kern.cam.boot_delay=10000

                pf10g2.png
                pf10g2.png_thumb

                1 Reply Last reply Reply Quote 0
                • Q
                  q54e3w
                  last edited by

                  Ill grab some benchmark points for you over the weekend when I get a free moment.

                  1 Reply Last reply Reply Quote 0
                  • T
                    trumee
                    last edited by

                    Again, this is what i have.

                    @jimp @jwt any ideas?

                    1 Reply Last reply Reply Quote 0
                    • Q
                      q54e3w
                      last edited by

                      @trumee:

                      I compared the tunables with my system and there is no difference. Is there anything in /boot/loader.conf.local? And  what do you have in  System>Advanced>Networking>Network Interfaces?

                      I am guessing you are getting 10gb speeds.

                      Here is another test where i booted into vanilla FreeBSD-11 on the pfsense router
                      Chelsio T420-CR (pfsense router running FreeBSD 11) and Mellanox ConnectX-2 EN (linux desktop)

                      
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.03 MBytes
[  4]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
[  4]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   1.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

                      So, the card works well in native FreeBSD-11. I then tested in pfSense-2.4 snapshot image which is based on FreeBSD-11. Again i got dismal speeds.

                      sorry for a stream of replies… take a look at your network stack variables and see if theres anything obvious. Check buffer/window sizes/scaling etc. The articles KOM listed above are good starting points to troubleshoot.

                      1 Reply Last reply Reply Quote 0
                      • G
                        gcu_greyarea
                        last edited by

                        I could be completely off the mark …. from memory (when working with storage) the Chelsio 10GB cards stuggled under heavy load with Flow Control enabled. I.e. Too many pause frames.
                        Try to disable flow control on all devices and see how things go.

                        I appolgize in advance if this is leading you down the wrong path....

                        1 Reply Last reply Reply Quote 0
                        • T
                          trumee
                          last edited by

                          @gcu_greyarea:

                          I could be completely off the mark …. from memory (when working with storage) the Chelsio 10GB cards stuggled under heavy load with Flow Control enabled. I.e. Too many pause frames.
                          Try to disable flow control on all devices and see how things go.

                          I appolgize in advance if this is leading you down the wrong path....

                          I am clutching at the straws myself so appreciate any idea.

                          I can't find a flowcontrol sysctl for cxgb card. The only ones listed are for the igb interface

                          
$sysctl -a | grep .fc:
dev.igb.3.fc: 3
dev.igb.2.fc: 3
dev.igb.1.fc: 3
dev.igb.0.fc: 3

                          However, i did find the dev.cxgbe.0.pause_settings sysctl. man page for cxgbe interface suggests

                          
     hw.cxgbe.pause_settings
             PAUSE frame settings.  Bit 0 is rx_pause, bit 1 is tx_pause.
             rx_pause = 1 instructs the hardware to heed incoming PAUSE
             frames, 0 instructs it to ignore them.  tx_pause = 1 allows the
             hardware to emit PAUSE frames when its receive FIFO reaches a
             high threshold, 0 prohibits the hardware from emitting PAUSE
             frames.  The default is 3 (both rx_pause and tx_pause = 1).  This
             tunable establishes the default PAUSE settings for all ports.
             Settings can be displayed and controlled on a per-port basis via
             the dev.cxgbe.X.pause_settings (dev.cxl.X.pause_settings for T5
             cards) sysctl.

                          The default value was set to

                          
$sysctl dev.cxgbe.0.pause_settings
dev.cxgbe.0.pause_settings: 3 <pause_rx,pause_tx></pause_rx,pause_tx>

                          I changed it to 0

                          
$sysctl dev.cxgbe.0.pause_settings=0
dev.cxgbe.0.pause_settings: 3 <pause_rx,pause_tx>-> 3 <pause_rx,pause_tx>$sysctl dev.cxgbe.0.pause_settings
dev.cxgbe.0.pause_settings: 0</pause_rx,pause_tx></pause_rx,pause_tx>

                          iperf3 results were the same as before

                          
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   110 MBytes   919 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   930 Mbits/sec                  
[  5]   2.00-3.00   sec   111 MBytes   929 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   929 Mbits/sec                  
[  5]   4.00-5.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   929 Mbits/sec                  
[  5]   7.00-8.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   931 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   930 Mbits/sec                  
[  5]  10.00-10.01  sec   660 KBytes   925 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.01  sec  1.08 GBytes   929 Mbits/sec                  receiver
                          1 Reply Last reply Reply Quote 0
                          • T
                            trumee
                            last edited by

                            I enabled tso on the cxgbe interface

                            
ifconfig cxgbe0 tso

                            and the speed improved

                            
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec    0    251 MBytes       
[  4]   1.00-2.00   sec   185 MBytes  1.55 Gbits/sec    0    251 MBytes       
[  4]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec    0    251 MBytes       
[  4]   3.00-4.00   sec   176 MBytes  1.48 Gbits/sec    0    251 MBytes       
[  4]   4.00-5.00   sec   183 MBytes  1.54 Gbits/sec    0    251 MBytes       
[  4]   5.00-6.00   sec   184 MBytes  1.55 Gbits/sec    0    251 MBytes       
[  4]   6.00-7.00   sec   180 MBytes  1.51 Gbits/sec    0    251 MBytes       
[  4]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec    0    251 MBytes       
[  4]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec    0    251 MBytes       
[  4]   9.00-10.00  sec   192 MBytes  1.61 Gbits/sec    0    251 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec                  receiver

                            and after disabling pf

                            
#pfctl -d

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.06 GBytes  9.12 Gbits/sec    0    247 MBytes       
[  4]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec    0    247 MBytes       
[  4]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    0    247 MBytes       
[  4]   3.00-4.00   sec  1.09 GBytes  9.38 Gbits/sec    0    247 MBytes       
[  4]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec    0    247 MBytes       
[  4]   5.00-6.00   sec  1.09 GBytes  9.38 Gbits/sec    0    247 MBytes       
[  4]   6.00-7.00   sec   733 MBytes  6.15 Gbits/sec   22   -1812469760.00 Bytes       
[  4]   7.00-8.00   sec   660 MBytes  5.54 Gbits/sec    0   -1602799360.00 Bytes       
[  4]   8.00-9.00   sec   610 MBytes  5.12 Gbits/sec    0   -1405709184.00 Bytes       
[  4]   9.00-10.00  sec   908 MBytes  7.61 Gbits/sec    0   -1261036608.00 Bytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  9.36 GBytes  8.04 Gbits/sec   22             sender
[  4]   0.00-10.00  sec  9.36 GBytes  8.04 Gbits/sec                  receiver
                            1 Reply Last reply Reply Quote 0
                            • T
                              trumee
                              last edited by

                              I changed from an MTU of 1500 to an MTU of 9000

                              With pf enabled

                              
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   808 MBytes  6.78 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   1.00-2.00   sec   864 MBytes  7.25 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   2.00-3.00   sec   898 MBytes  7.54 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   3.00-4.00   sec   873 MBytes  7.32 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   4.00-5.00   sec   869 MBytes  7.28 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   5.00-6.00   sec   885 MBytes  7.42 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   6.00-7.00   sec   843 MBytes  7.08 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   7.00-8.00   sec   879 MBytes  7.37 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   8.00-9.00   sec   844 MBytes  7.08 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   9.00-10.00  sec   871 MBytes  7.31 Gbits/sec    0   -1880960188.00 Bytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.43 GBytes  7.24 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.43 GBytes  7.24 Gbits/sec                  receiver

                              With pf disabled

                              
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.10 GBytes  9.45 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   1.00-2.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   2.00-3.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   3.00-4.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   4.00-5.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   5.00-6.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   6.00-7.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   7.00-8.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   8.00-9.00   sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
[  4]   9.00-10.00  sec  1.15 GBytes  9.86 Gbits/sec    0   -1880960188.00 Bytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.4 GBytes  9.82 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.4 GBytes  9.82 Gbits/sec                  receiver
                              1 Reply Last reply Reply Quote 0
                              • G
                                gcu_greyarea
                                last edited by

                                Hi trmee,

                                Can flow control be disabled on the switch and your test host, too

                                1 Reply Last reply Reply Quote 0
                                • G
                                  gcu_greyarea
                                  last edited by

                                  Hi,

                                  just read your post again. I can see that with pf disabled you achieve the desired speed.
                                  In that Case it looks like the problem might indeed be with your specific setup.

                                  Do you have pfSense support ?

                                  I imagine that pfSense support team could look at diagnostic data and drill down to find the bottleneck.
                                  The fact that you get better speed with higher MTU means pfSense has to handle less data packages per time.

                                  The 2758 is an 8 core Atom, but I do not know what the expected throughput ought to be with that CPU. Of course it all depends on what the FW is doing (packages, NAT ?, SNORT? etc).

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.