• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

10 GBit questions

Scheduled Pinned Locked Moved General pfSense Questions
25 Posts 6 Posters 1.4k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R
    rvdbijl
    last edited by Apr 29, 2023, 4:02 PM

    I'm running on a 2-core Core i7-2655LE @ 2.2GHz. It's been serving me well at 1 GBit. But now my ISP has upgraded to 2 GBit, and with that, I swapped out the i350 Quad board I had in this system for an i540-T2 10Gbit adapter.
    I moved all my local VLANs to one of the ports (going to a 10Gbit smart switch) and the other port on the board is a WAN port.
    With a PC connected on one of the other 10 GBit ports on the switch, I get 1.1 GBit max throughput through WAN (Speedtest with an ISP hosted server).

    Testing with iperf3 between that PC and the router yields some interesting results:
    Server on the router: Max 1.25Gbit throughput, CPU utilization on both cores spikes to 40% or so while the test is running (60% idle).
    Server on the PC: Max 5.6 Gbit throughput, CPU utilization on both cores around 10%.

    I did hook up another PC (but don't have a 10Gbit board, just a 2.5 Gbit), and I'm getting the full 2.5 Gbit between those systems in iperf3 in both directions, as well as when copying large files between those PCs. So I don't think the router or the two PC's have settings that could be causing issues.

    I did try to run two simultaneous speedtests from two PC's, and in that case I see (on the router) the traffic spike to 1.4 Gbit (700-ish on both PCs), but no higher than that in either direction.

    Question for the group -- is this CPU/system just not capable of running > 1.25/1.4 GBit? Or is there some other optimization or setting that could be causing issues here?

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by Apr 29, 2023, 10:38 PM

      That's not an especially fast CPU but I'd expect more than that through it. Testing to or from it directly is always going to give a lesser result.
      How are you measuring the CPU usage? I would use top -HaSP at the CLI.

      Check Status > Interfaces for errors on the NICs.

      Steve

      R 1 Reply Last reply Apr 29, 2023, 10:58 PM Reply Quote 0
      • R
        rvdbijl @stephenw10
        last edited by Apr 29, 2023, 10:58 PM

        @stephenw10 I was running the Diagnostics - System Activity which I think is the same output? Either way, the test ran long enough for me to see the CPU usage change.

        WAN interface has 5/0 errors and LAN has 0/184 errors. I assume that's cumulative since I rebooted last (about a week ago), so I'm assuming that's probably not the issue. Cables from the router to the switch are 2 feet long and CAT8 (shielded). But I have been plugging and unplugging them so it wouldn't surprise me if those errors were accumulated then. I did not see any movement on those counters in the last few minutes (even when running speedtests).

        Started looking on eBay for some faster systems.. The Xeon D2123IT (SYS-5019D-4C-FN8TP) looks like a decent option (without a humongous power draw)...

        D 1 Reply Last reply Apr 29, 2023, 11:52 PM Reply Quote 0
        • D
          Dobby_ @rvdbijl
          last edited by Apr 29, 2023, 11:52 PM

          @rvdbijl said in 10 GBit questions:

          The Xeon D2123IT (SYS-5019D-4C-FN8TP)

          If you have the chance to get hands on a device ending with an "N" or "NT" it is standing for network you may be better sorted with that. It is because you can count on that that
          devices are sorted with "all 4 options".

          • TurboBoost
          • Hyperthread
          • Intel QAT
          • AES-NI

          Pleas have a look at something like

          • Intel Xeon D-21xxNT
            or
          • Intel Xeon D-27xxN

          #~. @Dobby

          Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
          PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
          PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

          1 Reply Last reply Reply Quote 0
          • R
            rvdbijl
            last edited by Apr 30, 2023, 1:31 PM

            Looks like those *NT options add quite a bit to the cost ... At least, that I've found. Also to TDP (which I guess makes sense).

            It looks like the 2123IT has everything except QAT, which isn't implemented in CE 2.6.0 (which I run). So it doesn't sound like I should expect a big change between an *IT and *NT CPU, right?

            D 1 Reply Last reply Apr 30, 2023, 7:18 PM Reply Quote 0
            • T
              tman222
              last edited by tman222 Apr 30, 2023, 6:43 PM Apr 30, 2023, 6:10 PM

              Hi @rvdbijl - what are the rest of the specs of your current system? The i7-2655LE is fairly old (released in 2011) so I'm a little concerned that the issue might be insufficient PCI Express bandwidth due to an older motherboard.

              Having said that, some of the older Intel Xeon D-15xx family should be able to able to pass 2Gbit/s as well (if you're looking for something cheaper and/or used). I have run pfSense on a Supermicro 5018D-FN8T box for a number of years now and it can achieve 6-7 Gbit/s LAN1 to LAN2 (though the firewall) when running an iperf3 test. Real world performance (which would include smaller packets) is likely a bit less, but 3-4 Gbit/s, if not better, should still be achievable. Keep in mind that 10Gbit/s may require some hardware tuning as well.

              If you're looking for something newer and closer to cutting edge, this Xeon D-1718T based Supermicro box looks nice from a price/performance standpoint:

              https://www.supermicro.com/en/products/system/iot/1u/sys-510d-4c-fn6p

              The hardware is pretty recent though and likely will require at least pfSense 2.7 / pfSense Plus 23.01 for full support.

              Hope this helps.

              R 1 Reply Last reply Apr 30, 2023, 6:24 PM Reply Quote 0
              • R
                rvdbijl @tman222
                last edited by Apr 30, 2023, 6:24 PM

                @tman222 It's an old embedded system (Advantech Uno) with a PCIe x8 slot that the network board is mounted in. I got it for free a few years ago, and it was a good upgrade for my old Atom-based pfSense box when my network was 500MBit/50MBit and it was straining to keep up, especially with OpenVPN.
                Then I switched ISPs to 1G/1G, and it still kept up, seemingly with no issue (note that I do not run snort, or any other packages that could slow things down -- just firewall and some OpenVPN, DHCP and DNS). Now I'm at 2G/2G and I'm only seeing 1.1G of that with the Uno box (using Speedtest).

                I don't think it's constrained on its PCIe bus - it claims to be connected to the board at 8x link, etc. Also -- I can pump data TO it at 5-6 GBit/s based on iperf3. Just FROM it seems to be limited to 1.2 GBit/s (with iperf3).

                Either way -- I don't want to drop a ton of money on a new system ($500-$800 would be my limit), so the Xeon D-1718T is a bit out of my price range. I did just buy an older Dell R210-II with Xeon E3-1275 v2 on it off eBay for a little under $300, shipped. When that arrives, I'll give it a try. I know it's going to consume more power than my current system, but I can't justify >$1000 right now. If the Dell fails in getting me the full 2 GBit bandwidth (with some margin), I'll look for a faster system.

                T 1 Reply Last reply Apr 30, 2023, 6:52 PM Reply Quote 0
                • T
                  tman222 @rvdbijl
                  last edited by tman222 Apr 30, 2023, 6:53 PM Apr 30, 2023, 6:52 PM

                  @rvdbijl - that Dell R210-II system should be more than capable based on the CPU specs I can see. Also, I agree with you that it sounds like there is enough PCI Express bandwidth with the x8 slot on your current syste.

                  There may be another performance limitation / some network tuning required to try to overcome the current limit you're seeing. A couple good links to check out:

                  https://calomel.org/freebsd_network_tuning.html
                  https://wiki.freebsd.org/NetworkPerformanceTuning
                  https://docs.netgate.com/pfsense/en/latest/hardware/tune.html

                  Hope this helps.

                  S 1 Reply Last reply May 4, 2023, 5:03 PM Reply Quote 0
                  • D
                    Dobby_ @rvdbijl
                    last edited by Apr 30, 2023, 7:18 PM

                    @rvdbijl said in 10 GBit questions:

                    Looks like those *NT options add quite a bit to the cost ...

                    If you can get your hands on a used hardware, and there
                    will be two choices! One with IT and one with N or NT
                    and they are nearly the same price range, let us say at
                    something around $500 ish, my tip was to go with the
                    N or NT named boards (CPUs) over the IT named ones.

                    At least, that I've found. Also to TDP (which I guess
                    makes sense).

                    If this will be even more important to many of us, you
                    could also play with the idea to get hands on a small
                    Intel Denverton (C3000) hardware. 4 cores are enough
                    for your 1 GBit/s internet line if you are not using PPPoE.
                    Board ~389 plus shipping fee
                    case

                    Perhaps on top of this will be coming then if needed;

                    • RAM 2x DDR4 2400MHz
                    • M.2 SSD
                    • 2.5 GBit/s NIC

                    It looks like the 2123IT has everything except QAT, which isn't implemented in CE 2.6.0 (which I run). So it doesn't sound like I should expect a big change between an *IT and *NT CPU, right?

                    With pfSense CE you may be also able to use QAT but
                    not able to set it up so easy like with pfSense+ Plus.

                    The N or NT is giving you the full "potential" of the mostly
                    wished or "needed" points, but this is all different from user to user!

                    • QAT
                    • AES-NI
                    • TurboBoost
                    • HT
                    • Support DPDK

                    Mostly or often this may be the main points all are
                    talking about.

                    #~. @Dobby

                    Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
                    PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
                    PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

                    1 Reply Last reply Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator
                      last edited by Apr 30, 2023, 7:52 PM

                      You should make sure the 10G NICs show the expeced number of queues when they attach at boot. Especially since you're seeing traffic limited in one direction.

                      R 1 Reply Last reply May 1, 2023, 3:24 PM Reply Quote 1
                      • R
                        rvdbijl @stephenw10
                        last edited by stephenw10 May 1, 2023, 3:59 PM May 1, 2023, 3:24 PM

                        @stephenw10 said in 10 GBit questions:

                        You should make sure the 10G NICs show the expeced number of queues when they attach at boot. Especially since you're seeing traffic limited in one direction.

                        It looks like 2 queues are being allocated for both ix0/ix1 interfaces. Not sure if that's what it's supposed to be:

                        ix1: netmap queues/slots: TX 2/2048, RX 2/2048
                        ix1: eTrack 0x80000528 PHY FW V286
                        ix1: PCI Express Bus: Speed 5.0GT/s Width x8
                        ix1: Ethernet address: 80:61:5f:0e:8c:25
                        ix1: allocated for 2 rx queues
                        ix1: allocated for 2 queues
                        ix1: Using MSI-X interrupts with 3 vectors
                        ix1: Using 2 RX queues 2 TX queues
                        ix1: Using 2048 TX descriptors and 2048 RX descriptors
                        ix1: <Intel(R) X540-AT2> mem 0xf0000000-0xf01fffff,0xf0400000-0xf0403fff irq 18 at device 0.1 on pci2
                        ix0: netmap queues/slots: TX 2/2048, RX 2/2048
                        ix0: eTrack 0x80000528 PHY FW V286
                        ix0: PCI Express Bus: Speed 5.0GT/s Width x8
                        ix0: Ethernet address: 80:61:5f:0e:8c:24
                        ix0: allocated for 2 rx queues
                        ix0: allocated for 2 queues
                        ix0: Using MSI-X interrupts with 3 vectors
                        ix0: Using 2 RX queues 2 TX queues
                        ix0: Using 2048 TX descriptors and 2048 RX descriptors
                        ix0: <Intel(R) X540-AT2> mem 0xf0200000-0xf03fffff,0xf0404000-0xf0407fff irq 17 at device 0.0 on pci2
                        
                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by May 1, 2023, 4:05 PM

                          Yeah, that's probably correct Though that CPU looks like it's 2-cores with 2-threads per core so 4 virtual cores., if hyper-threading is enabled. If it shows 4 CPUs I'd expect 4 queues.

                          It's the same number of queues for Tx and Rx though so that doesn't look like a problem.

                          R 1 Reply Last reply May 1, 2023, 4:22 PM Reply Quote 1
                          • R
                            rvdbijl @stephenw10
                            last edited by May 1, 2023, 4:22 PM

                            @stephenw10 said in 10 GBit questions:

                            Yeah, that's probably correct Though that CPU looks like it's 2-cores with 2-threads per core so 4 virtual cores., if hyper-threading is enabled. If it shows 4 CPUs I'd expect 4 queues.

                            I was wondering about that myself -- I did turn Hyperthreading off to test to see if it got faster, but there was no appreciable difference. So I turned it back on. This is the log from the last boot and shows 2 queues, despite there being 4 cores (HT).

                            It's the same number of queues for Tx and Rx though so that doesn't look like a problem.

                            So it sounds like there aren't many more avenues to optimize this box. For whatever reason it's just not able to handle > 1GBit. My new box should be here in a week or so. Hopefully that one will run a lot faster...

                            (I did go through the optimization articles that were mentioned earlier, but none of the tricks made it any faster. Some actually made it slower -- like turning off HW offload options).

                            Thanks all for the tips and suggestions!

                            D 1 Reply Last reply May 1, 2023, 4:43 PM Reply Quote 0
                            • D
                              Dobby_ @rvdbijl
                              last edited by May 1, 2023, 4:43 PM

                              @rvdbijl

                              The Xeon D2123IT

                              4 Cores
                              8 Threads

                              max 3.0GHz

                              TurboBoost
                              HyperThreading
                              AES-NI
                              DPDK?

                              4 from 5!

                              If you are not using the PPPoE it will saturate a 1 GBit/s
                              with ease. And in theoretic it should be then able to
                              feed or support 8 queues, but you can also "tune" the;

                              • queue size
                              • queue length
                              • queue amount pending on the CPU "C/T"

                              perhaps you will be reporting back here if that box
                              was arriving.

                              #~. @Dobby

                              Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
                              PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
                              PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

                              R 1 Reply Last reply May 1, 2023, 4:49 PM Reply Quote 0
                              • R
                                rvdbijl @Dobby_
                                last edited by May 1, 2023, 4:49 PM

                                @dobby_
                                I ended up going with the Dell R210-II system with Xeon 1275v2 CPU. I'll be more than happy to report to this thread once I have some measurements with iperf3 / Speedtest!

                                D 1 Reply Last reply May 4, 2023, 9:40 PM Reply Quote 1
                                • S
                                  SpaceBass @tman222
                                  last edited by May 4, 2023, 5:03 PM

                                  @tman222 said in 10 GBit questions:

                                  Dell R210-II system should be more than capable

                                  In my testing, the R2x series cannot move more than about 1.8-2Gbps ... the CPUs simply max out on single thread routing

                                  R 1 Reply Last reply May 5, 2023, 12:04 PM Reply Quote 0
                                  • D
                                    Dobby_ @rvdbijl
                                    last edited by May 4, 2023, 9:40 PM

                                    @rvdbijl said in 10 GBit questions:

                                    Dell R210-II system with Xeon 1275v2 CPU

                                    3,5 - 3,9 GHZ
                                    CPU 4C/8T
                                    AES-NI
                                    TurboBoost
                                    Hyperthreading

                                    May be also an interesting choice! If you will not forced
                                    to use PPPoE it can be significant faster then imagine of.

                                    #~. @Dobby

                                    Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
                                    PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
                                    PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

                                    1 Reply Last reply Reply Quote 0
                                    • R
                                      rvdbijl @SpaceBass
                                      last edited by May 5, 2023, 12:04 PM

                                      @spacebass said in 10 GBit questions:

                                      @tman222 said in 10 GBit questions:

                                      Dell R210-II system should be more than capable

                                      In my testing, the R2x series cannot move more than about 1.8-2Gbps ... the CPUs simply max out on single thread routing

                                      What CPU did you test with on the R210-ii?

                                      S 1 Reply Last reply May 5, 2023, 2:22 PM Reply Quote 0
                                      • S
                                        SpaceBass @rvdbijl
                                        last edited by May 5, 2023, 2:22 PM

                                        @rvdbijl 1270 v5

                                        R 1 Reply Last reply May 5, 2023, 2:23 PM Reply Quote 0
                                        • R
                                          rvdbijl @SpaceBass
                                          last edited by May 5, 2023, 2:23 PM

                                          @spacebass
                                          The v5's work on the R210-ii? From what I read, it only supports up to the E3-12xx v2 series ...

                                          R 1 Reply Last reply May 11, 2023, 3:00 PM Reply Quote 0
                                          20 out of 25
                                          • First post
                                            20/25
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received