Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense low throughput from 10GB to 1GB Interfaces

    Scheduled Pinned Locked Moved General pfSense Questions
    7 Posts 2 Posters 1.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      Kotty
      last edited by

      Hi,

      I have the following HA - Setup:

      2x HP Proliant DL360 gen9 with 1 onboard LAN ( HP Ethernet 1GB 331i), 1 PCI-E LAN (HP Ethernet 1GB 331FLR) and 1 PCI-E LAN (HP Ethernet 10Gb 560SFP+).
      pfSense Software 2.4.5 p1

      I have configured different VLANS on these interfaces. The routing, CARP, etc. works so far. Then I wanted to test the speed of each interface. The iPerf3 test gave the following results:

      From 1GB interface (HP 331i) to 10GB interface (HP 560):

      [ 4] local "IP" port 52811 connected to "IP" port 5201
      ID] Interval Transfer Bandwidth
      [ 4] 0.00-1.00 sec 54.2 MBytes 455 Mbits/sec
      [ 4] 1.00-2.00 sec 69.8 MBytes 585 Mbits/sec
      [ 4] 2.00-3.00 sec 73.0 MBytes 612 Mbits/sec
      [ 4] 3.00-4.00 sec 73.2 MBytes 615 Mbits/sec
      4] 4.00-5.00 sec 73.5 MBytes 616 Mbits/sec
      [ 4] 5.00-6.00 sec 73.9 MBytes 620 Mbits/sec
      [ 4] 6.00-7.00 sec 73.4 MBytes 615 Mbits/sec
      [ 4] 7.00-8.00 sec 73.2 MBytes 615 Mbits/sec
      [ 4] 8.00-9.00 sec 73.5 MBytes 616 Mbits/sec
      [ 4] 9.00-10.00 sec 73.6 MBytes 618 Mbits/sec

      ID] Interval Transfer Bandwidth
      [ 4] 0.00-10.00 sec 711 MBytes 597 Mbits/sec transmitter
      4] 0.00-10.00 sec 711 MBytes 597 Mbits/sec receiver

      From 10GB interface (HP 560) to 1GB interface (HP 331i):

      [ 4] local "IP" port 59909 connected to "IP" port 5201
      ID] Interval Transfer Bandwidth
      [ 4] 0.00-1.01 sec 51.8 MBytes 428 Mbits/sec
      [ 4] 1.01-2.01 sec 39.0 MBytes 328 Mbits/sec
      [ 4] 2.01-3.01 sec 51.5 MBytes 433 Mbits/sec
      [ 4] 3.01-4.01 sec 40.2 MBytes 338 Mbits/sec
      4] 4.01-5.01 sec 38.5 MBytes 323 Mbits/sec
      [ 4] 5.01-6.01 sec 43.5 MBytes 365 Mbits/sec
      [ 4] 6.01-7.00 sec 48.2 MBytes 405 Mbits/sec
      [ 4] 7.00-8.00 sec 41.2 MBytes 347 Mbits/sec
      [ 4] 8.00-9.00 sec 39.8 MBytes 334 Mbits/sec
      [ 4] 9.00-10.02 sec 38.9 MBytes 322 Mbits/sec

      ID] Interval Transfer Bandwidth
      [ 4] 0.00-10.02 sec 433 MBytes 362 Mbits/sec transmitter
      4] 0.00-10.02 sec 433 MBytes 362 Mbits/sec receiver

      As you can see, we have significantly worse performance from the 10GB to the 1GB interface than vice versa. However, the speed from 1GB to 10GB does not satisfy us either.

      I did an iperf3 test directly from the pfSense to the individual targets and found the following values:

      From the pfSense (1GB interface) to the first destination:

      [ 5] local "IP" port 56729 connected to "IP" port 5201
      ID] Interval Transfer Bitrate Retr Cwnd
      [ 5] 0.00-1.00 sec 113 MBytes 947 Mbits/sec 1 208 KBytes
      [ 5] 1.00-2.00 sec 113 MBytes 945 Mbits/sec 2 208 KBytes
      [ 5] 2.00-3.00 sec 113 MBytes 945 Mbits/sec 2 208 KBytes
      [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
      [ 5] 4.00-5.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
      5] 5.00-6.00 sec 113 MBytes 945 Mbits/sec 2 153 KBytes
      [ 5] 6.00-7.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes
      [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 1 208 KBytes
      [ 5] 8.00-9.00 sec 113 MBytes 945 Mbits/sec 1 208 KBytes
      [ 5] 9.00-10.00 sec 113 MBytes 946 Mbits/sec 1 208 KBytes

      [ ID] Interval Transfer Bitrate Retr
      [ 5] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 13 sender
      [ 5] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec receiver

      From pfSense (10GB) to second destination:

      [ 5] local "IP" port 42314 connected to "IP" port 5201
      [ ID] Interval Transfer Bitrate Retr Cwnd
      [ 5] 0.00-1.00 sec 111 MBytes 933 Mbits/sec 32 209 KBytes
      [ 5] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 7 120 KBytes
      [ 5] 2.00-3.00 sec 110 MBytes 924 Mbits/sec 0 208 KBytes
      [ 5] 3.00-4.00 sec 108 MBytes 907 Mbits/sec 1 159 KBytes
      [ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes
      5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 209 KBytes
      [ 5] 6.00-7.00 sec 109 MBytes 913 Mbits/sec 3 192 KBytes
      [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 209 KBytes
      [ 5] 8.00-9.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes
      [ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec 0 209 KBytes

      [ ID] Interval Transfer Bitrate Retr
      [ 5] 0.00-10.00 sec 1.08 GBytes 925 Mbits/sec 43 sender
      [ 5] 0.00-10.00 sec 1.08 GBytes 925 Mbits/sec receiver

      Does anyone have any idea how i can fix this? We would be satisfied with a result close to 1GB.

      Thanks in advance.

      Kind regards,
      Kotty

      H 1 Reply Last reply Reply Quote 0
      • H
        heper @Kotty
        last edited by

        @kotty
        don't test TO or FROM pfsense ... that is pointless.
        it's a router. run the tests THROUGH pfsense with a client/server on each end

        1 Reply Last reply Reply Quote 1
        • K
          Kotty
          last edited by

          @heper

          Sorry for the misunderstandings. We have tested the throughput with a client/server.

          First test Client/Server -> pfSense in (1GB) -> pfSense out (10GB) -> Client/Server

          Second test Client/Server -> pfsense in (10GB) -> pfSense out(1GB) -> Client/Server.

          third test iperf pfSense out (1GB) -> Client/Server.

          fourth test iperf pfSense out(10GB) -> Client/Server

          I hope it is better to understand.

          Thank you for your help.

          H 1 Reply Last reply Reply Quote 0
          • H
            heper @Kotty
            last edited by

            @kotty try the tuning for your specific cards/drivers
            the 331i appears to be some sort of broadcom chipset
            the 560 seems to be an intel chipset

            https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#card-specific-issues

            1 Reply Last reply Reply Quote 0
            • K
              Kotty
              last edited by

              @heper

              Thank you.

              I have already gone through this post. This has improved the performance. I had problems with packet loss. After I went through the tuning guide, it got much better. But now there are still the performance problems. Could it be a problem with the network cards?

              H 1 Reply Last reply Reply Quote 0
              • H
                heper @Kotty
                last edited by

                @kotty
                I don't have similar hardware.... Might be a lot of things.

                Cabling/switch/driver/....

                1 Reply Last reply Reply Quote 0
                • K
                  Kotty
                  last edited by

                  Hi,

                  I have some news. After replacing the Intel 10GBE network card with an Intel X722-DA2 (ixL Interface), my performance problems are gone. But there is a new problem. When testing the 10GB performance I only get a speed of 2.7Gigabit. Here is an excerpt from iPerf3:

                  [ ID] Interval Transfer Bitrate Retr Cwnd
                  [ 5] 0.00-1.00 sec 276 MBytes 2.31 Gbits/sec 0 687 KBytes
                  [ 5] 1.00-2.00 sec 273 MBytes 2.29 Gbits/sec 0 687 KBytes
                  [ 5] 2.00-3.00 sec 276 MBytes 2.32 Gbits/sec 0 687 KBytes
                  [ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec 0 687 KBytes
                  [ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec 0 687 KBytes
                  5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 689 KBytes
                  [ 5] 6.00-7.00 sec 276 MBytes 2.32 Gbits/sec 0 689 KBytes
                  5] 7.00-8.00 sec 276 MBytes 2.32 Gbits/sec 0 689 KBytes
                  5] 8.00-9.00 sec 275 MBytes 2.30 Gbits/sec 0 689 KBytes
                  [ 5] 9.00-10.00 sec 277 MBytes 2.32 Gbits/sec 0 689 KBytes

                  [ ID] Interval Transfer Bitrate Retr
                  [ 5] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec 0 sender
                  [ 5] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver

                  With the command "top -p CC" I see that the test uses only one CPU Core to 100%. However, this only happens when testing the 10GB connection. 1GB tests are distributed over all cores. As it should be. Is there a command or a way to set the pfSense to multicore for 10GBE connections?

                  Thanks in advance.

                  Kind regards,
                  Kotty

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.