Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    XG-7100 10 gbe throughput issue

    Official Netgate® Hardware
    3
    16
    1244
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pinemin last edited by

      One of the sfp+ ports on XG-7100 is configured as my LAN interface and is connected to a sfp+ port on my Mikrotik switch. Also connected to an sfp+ port on that switch is a TrueNAS. Using iperf3, data transfers from my TrueNAS to the XG-7100 are happening at 980Mbps (one tenth of what I was expecting). Any ideas of what the problem might be?

      1 Reply Last reply Reply Quote 0
      • Derelict
        Derelict LAYER 8 Netgate last edited by

        Sounds like something is negotiating at gigabit, though I would expect to see something around 940Mbit/sec in that case.

        During a test, watch top -aSH running on the pfSense node. Report what you see there.

        Chattanooga, Tennessee, USA
        The pfSense Book is free of charge!
        DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • johnpoz
          johnpoz LAYER 8 Global Moderator last edited by

          980 is pretty freaking high for gig.. I don't think that even possible really.. With Derelict here I would expect something more in the mid 900's if gig was the speed negotiated.. You could get that if talking 1518 frame with the inter-frame gap and preamble.. But if you take those away your more like 974 Max... Guess comes down to how your doing your math.. You could see that with Jumbo I guess.. Taking away all the overhead your prob at 987mbps with jumbo..

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

          1 Reply Last reply Reply Quote 0
          • P
            pinemin last edited by

            Here is the iperf3 output. I don't think the connection is being negotiated down to gigabit. Watched top -aSH while running the test. Don't know what I should have been looking for but the iperf3 process was at the top of the list.

            [admin@NAS1 ~]$ iperf3 -c 10.10.10.1
            Connecting to host 10.10.10.1, port 5201
            [ 5] local 10.10.10.10 port 59828 connected to 10.10.10.1 port 5201
            [ ID] Interval Transfer Bitrate Retr Cwnd
            [ 5] 0.00-1.00 sec 130 MBytes 1.09 Gbits/sec 69 208 KBytes
            [ 5] 1.00-2.00 sec 130 MBytes 1.09 Gbits/sec 64 223 KBytes
            [ 5] 2.00-3.00 sec 130 MBytes 1.09 Gbits/sec 20 223 KBytes
            [ 5] 3.00-4.00 sec 126 MBytes 1.06 Gbits/sec 93 125 KBytes
            [ 5] 4.00-5.00 sec 125 MBytes 1.05 Gbits/sec 11 224 KBytes
            [ 5] 5.00-6.00 sec 125 MBytes 1.05 Gbits/sec 81 224 KBytes
            [ 5] 6.00-7.00 sec 125 MBytes 1.05 Gbits/sec 23 224 KBytes
            [ 5] 7.00-8.00 sec 126 MBytes 1.06 Gbits/sec 7 224 KBytes
            [ 5] 8.00-9.00 sec 97.9 MBytes 821 Mbits/sec 14 224 KBytes
            [ 5] 9.00-10.00 sec 123 MBytes 1.03 Gbits/sec 11 224 KBytes


            [ ID] Interval Transfer Bitrate Retr
            [ 5] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec 393 sender
            [ 5] 0.00-10.00 sec 1.21 GBytes 1.04 Gbits/sec receiver

            1 Reply Last reply Reply Quote 0
            • johnpoz
              johnpoz LAYER 8 Global Moderator last edited by

              Your testing too pfsense? Test THRU pfsense..

              iperfclient -- pfsense -- iperfserver

              An intelligent man is sometimes forced to be drunk to spend time with his fools
              If you get confused: Listen to the Music Play
              Please don't Chat/PM me for help, unless mod related
              2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

              1 Reply Last reply Reply Quote 0
              • Derelict
                Derelict LAYER 8 Netgate last edited by

                For testing throughput you should be testing through the firewall, not with one end on the firewall. Not that I think that's exactly why you are seeing what you are seeing, but that methodology is wrong.

                Chattanooga, Tennessee, USA
                The pfSense Book is free of charge!
                DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • johnpoz
                  johnpoz LAYER 8 Global Moderator last edited by

                  ^exactly... When he said all he saw was iperf using up cpu at the top of his list when you asked him to check top when running iperf I take it that he is testing to pfsense..

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

                  1 Reply Last reply Reply Quote 0
                  • P
                    pinemin last edited by

                    Let me provide some context. I have 2 machines with 10 Gbe networking capability through sfp+. A TrueNAS Server and the Netgate XG 7100. In about 3 weeks, I will add a virtualization server that also has 10 Gbe sfp+. I am just in the process of setting up the systems for a small business with high throughput needs. The "in rack" networking is at 10 Gbe while for now all the clients are 1 Gbe. What I wanted to test was the throughput within the rack. Since I didn't have another iperf3 server capable of 10 Gbe I used the pfsense box. The other alternative would be to test the throughput to multiple 1 Gbe clients but I don't have that option right now either. Hence the test to the pfsense box. Help me out here but my thought was that if I can't get 10 Gbe (or close) to the pfsense, its unlikely that I will get it through the pfsense to the virtualization server when I set it up in a month's time.

                    1 Reply Last reply Reply Quote 0
                    • Derelict
                      Derelict LAYER 8 Netgate last edited by Derelict

                      Right now we don't know that it's not the switch. Have you tried directly connecting them?

                      Chattanooga, Tennessee, USA
                      The pfSense Book is free of charge!
                      DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • P
                        pinemin last edited by

                        Agreed. I am working on ruling that out. What I plan to do is attach the TrueNAS directly to the XG 7100 and run the same test to rule out a switch problem but the pfsense will still be the iperf3 server.

                        1 Reply Last reply Reply Quote 0
                        • Derelict
                          Derelict LAYER 8 Netgate last edited by

                          I have a couple units here. It'll take me a bit to get the test set up. It's going to be iperf3 between an XG-2758 and XG-7100. If anything is the limiting factor there it should be the 2758.

                          Chattanooga, Tennessee, USA
                          The pfSense Book is free of charge!
                          DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • P
                            pinemin last edited by

                            That would be great. Thank you.

                            1 Reply Last reply Reply Quote 0
                            • P
                              pinemin last edited by

                              Today, I attached the TrueNAS directly to the XG 7100 with a DAC sfp+ cable and ran an iperf3 test. The transfer rate was 2.2 Gbps. Twice what is was with the switch between them but still way below what I would have expected. Any suggestions would be greatly appreciated.

                              1 Reply Last reply Reply Quote 0
                              • P
                                pinemin last edited by

                                Clarification on the immediately prior post. On a direct connection of the TrueNAS to the XG 7100, I get a transfer rate of 2.2 Gbps if the XG 7100 is the client. It is ~ 1 Gbps if the XG 7100 is the server. That seems to me to rule out the Switch as the bottleneck. I got exactly the same rates with and without the switch in the path.

                                1 Reply Last reply Reply Quote 0
                                • Derelict
                                  Derelict LAYER 8 Netgate last edited by

                                  I was seeing the same sort of thing but it looked like it was the iperf3 process on the firewall itself becoming CPU bound.

                                  That brings us back around to testing through the firewall not to/from the firewall.

                                  Chattanooga, Tennessee, USA
                                  The pfSense Book is free of charge!
                                  DO NOT set a source port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • P
                                    pinemin last edited by

                                    Thanks for that Derelict. I guess I will just have to wait until the new server arrives and see what happens then. Appreciate your help.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post