Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Hardware Recommendations

    Scheduled Pinned Locked Moved Hardware
    23 Posts 3 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dcreationsinc
      last edited by

      Hello,

      I am running into an issue. I have a 1 gbps by 1 gbps internet connection. This is a copper connection so no need for SPF connections. I am currently maxing out at 640 mbps by 890 mbps. When I plug directly into the modem (without pfsense) I get 980 mbps up and down. So it leads me to believe its something hardware-wise. It has the newest version of pfsense installed on it. It has a SuperMicro X8DTL motherboard loaded with dual Intel Xeon L5640's with a max speed of 2.266 ghz, 32 GB of RAM and 500 GB SSD drive. It has 3 NIC ports in it. The em0/em1 (embedded to the motherboard) using Intel 82574L. Then the PCI-e card has dual 82575EB using igb0/ibg1. I have tried both and get pretty close to the same speeds on both. What would your recommendations to achieve the speeds I am paying for. Thanks.

      • doug
      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Do you have powerd enabled? You may not be seeing turbo frequencies otherwise. Though I would expect that to pass 1Gbps even at 2.2GHz. Have you tested locally just with hosts directly on either side of the firewall?

        Do you have packages running? Traffic shaping?

        Steve

        1 Reply Last reply Reply Quote 0
        • D
          dcreationsinc
          last edited by

          Sorry about my lateness getting back to you. I do have powerd enabled. If i test behind pfsense i get 640/900. If i test with a computer connected right to the modem I get 980/980.

          I currently run pfBlocker, RRD_Summary and snort.

          Thank you for the help. Take care.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Ah, possibly Snort limiting it then. Does it pass full speed if you disable Snort on all interfaces?

            D 1 Reply Last reply Reply Quote 0
            • D
              dcreationsinc
              last edited by

              I already disabled snort and pfBlocker and ran speed test. Did not increase the speeds at all.

              1 Reply Last reply Reply Quote 0
              • T
                tman222
                last edited by tman222

                Hi @dcreationsinc - since you have a very high speed WAN connection, have you already tried tuning the network cards on the pfSense system? Check out these threads for more info on parameters than can be adjusted:

                https://forum.netgate.com/topic/101391/loader-conf-local-tuning-for-modern-hardware
                https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-upload

                This page may also be helpful for troubleshooting - check out the section, "Where is the bottleneck ?"

                https://bsdrp.net/documentation/technical_docs/performance

                Hope this helps.

                1 Reply Last reply Reply Quote 0
                • D
                  dcreationsinc @stephenw10
                  last edited by

                  I installed the command line version of speedtest on the pfsense box just now. Directly on it the highest it gets is 652/538. But yet clients connected behind it see a lower download and a faster upload. Can someone help make some sense out of that? It has me baffled.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Testing to/from pfSense like that is not a great test in absolute terms as pfSense it not at all optimised for TCP termination in the way that a server would be. It's a router. It can be useful on lower speed connections or in revealing a problem on the LAN side.

                    Try running top -aSH on pfSense whilst testing from a client behind it. See what load is being put on it and how that spreads across the cores.

                    Is your connection PPPoE?

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • D
                      dcreationsinc
                      last edited by

                      No our connection isnt PPPoE. Even when I run speed test from client computers the max i see is 500/890. I know I can get a lot better than that. I have tried it from multiple machines behind the pfsense box.

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        So run top -aSH on the pfSense box whilst you are testing and see what sort of CPU usage you get there.

                        Steve

                        D 1 Reply Last reply Reply Quote 0
                        • D
                          dcreationsinc @stephenw10
                          last edited by

                          @stephenw10 said in Hardware Recommendations:

                          top -aSH

                          On both upload and download the most i saw was 94% idle.

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            We need to see the actual output there, what is actually using the CPU and how it's spread across the cores.

                            D 2 Replies Last reply Reply Quote 0
                            • D
                              dcreationsinc @stephenw10
                              last edited by

                              @stephenw10

                              1 Reply Last reply Reply Quote 0
                              • D
                                dcreationsinc @stephenw10
                                last edited by

                                @stephenw10 last pid: 34544; load averages: 0.31, 0.21, 0.26 up 9+10:56:36 10:21:03
                                442 processes: 26 running, 336 sleeping, 80 waiting
                                CPU: 0.6% user, 3.7% nice, 0.0% system, 1.2% interrupt, 94.5% idle
                                Mem: 297M Active, 6615M Inact, 753M Wired, 159M Buf, 24G Free
                                Swap: 3852M Total, 3852M Free

                                PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
                                11 root 155 ki31 0K 384K CPU8 8 225.8H 100.00% [idle{idle: cpu8}]
                                11 root 155 ki31 0K 384K CPU2 2 225.8H 100.00% [idle{idle: cpu2}]
                                11 root 155 ki31 0K 384K CPU11 11 225.8H 100.00% [idle{idle: cpu11}]
                                11 root 155 ki31 0K 384K CPU7 7 225.2H 100.00% [idle{idle: cpu7}]
                                11 root 155 ki31 0K 384K CPU16 16 224.0H 100.00% [idle{idle: cpu16}]
                                11 root 155 ki31 0K 384K CPU15 15 224.0H 100.00% [idle{idle: cpu15}]
                                11 root 155 ki31 0K 384K CPU0 0 225.5H 98.04% [idle{idle: cpu0}]
                                11 root 155 ki31 0K 384K CPU4 4 224.0H 97.26% [idle{idle: cpu4}]
                                11 root 155 ki31 0K 384K CPU5 5 224.5H 96.96% [idle{idle: cpu5}]
                                11 root 155 ki31 0K 384K RUN 14 224.0H 96.92% [idle{idle: cpu14}]
                                11 root 155 ki31 0K 384K CPU6 6 224.8H 95.21% [idle{idle: cpu6}]
                                11 root 155 ki31 0K 384K CPU1 1 225.5H 93.02% [idle{idle: cpu1}]
                                11 root 155 ki31 0K 384K CPU3 3 225.8H 76.00% [idle{idle: cpu3}]
                                12 root -92 - 0K 1280K WAIT 3 36:37 24.02% [intr{irq259: igb0:que 3}]
                                12 root -92 - 0K 1280K WAIT 1 49:37 6.99% [intr{irq257: igb0:que 1}]
                                12 root -92 - 0K 1280K WAIT 6 74:02 4.71% [intr{irq263: igb1:que 2}]
                                12 root -92 - 0K 1280K WAIT 5 82:00 3.17% [intr{irq262: igb1:que 1}]
                                74499 root 20 0 9860K 5336K CPU14 14 0:00 3.08% top -aSH
                                12 root -92 - 0K 1280K WAIT 4 112:35 2.60% [intr{irq261: igb1:que 0}]
                                12 root -92 - 0K 1280K WAIT 0 52:52 1.81% [intr{irq256: igb0:que 0}]
                                9549 root 20 0 12904K 8152K select 8 0:00 0.30% sshd: root@pts/0 (sshd)
                                43572 root 20 0 10200K 5716K select 0 4:51 0.14% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
                                12 root -92 - 0K 1280K WAIT 2 34:38 0.05% [intr{irq258: igb0:que 2}]
                                36727 root 20 0 6900K 2456K nanslp 15 0:08 0.04% [dpinger{dpinger}]
                                37627 root 20 0 6900K 2456K nanslp 11 0:08 0.04% [dpinger{dpinger}]
                                13310 dhcpd 20 0 12576K 8068K select 7 0:29 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf
                                12 root -92 - 0K 1280K WAIT 7 57:00 0.02% [intr{irq264: igb1:que 3}]
                                36727 root 20 0 6900K 2456K sbwait 16 0:03 0.02% [dpinger{dpinger}]
                                12 root -60 - 0K 1280K WAIT 0 6:33 0.01% [intr{swi4: clock (0)}]
                                11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
                                11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
                                11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
                                11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]

                                1 Reply Last reply Reply Quote 0
                                • stephenw10S
                                  stephenw10 Netgate Administrator
                                  last edited by

                                  @dcreationsinc said in Hardware Recommendations:

                                  11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
                                  11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
                                  11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
                                  11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]

                                  Hmm at least 4 CPUs are 0% idle.... that looks a little odd. What is that load if it's not shown....

                                  That system is 24 apparent cores right?

                                  The actual loading shown is not unexpected though it's not spread evenly at all. That was passing 600Mbps at the time?
                                  igb0 is WAN there? And igb1 was the internal interface used for that test?

                                  Steve

                                  D 1 Reply Last reply Reply Quote 0
                                  • D
                                    dcreationsinc @stephenw10
                                    last edited by

                                    @stephenw10 Yes its 24 cores. The load avgs are last pid: 34544; load averages: 0.31, 0.21, 0.26 ibg0 is WAN. That was passing 600 mbps at a time. Right now I am using ibg1 as my LAN port.

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Hmm, I don't have anything to compare that with directly but I expect to see idle processes there for all 24 cores and I expect to see them all mostly idle. I'm unsure what the 0% idle processes for the other cores indicate there...

                                      If we assume the load on cpu3 is the igb0 interrupt load it's still not a CPU limit. Did you try swapping the NICs in use there? Maybe put on of the em NICs on WAN as a test.

                                      Steve

                                      D 1 Reply Last reply Reply Quote 0
                                      • D
                                        dcreationsinc @stephenw10
                                        last edited by

                                        @stephenw10 The more i think about it I think it might be an incompatibility between the modem and pfsense. Primary the network cards used. The reason why I say this is because when I had cable internet (1 gbps/20 Mbps) I got 980 Mbps download threw the same pfsense box. Can you recommend a low profile network card preferably with at least dual ports on it that I can pick up? Thanks.

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Hmm, I mean those em chipsets you're using are very common, I wouldn't have expected any issues with them.

                                          I would look for something using the igb driver just so you know it's different. i350, i210 NICs are common and well tested.

                                          Steve

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            tman222
                                            last edited by

                                            Before running out and buying new hardware, have you tried tuning to see if that makes a performance difference?

                                            D 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.