Hardware Recommendations
-
Hello,
I am running into an issue. I have a 1 gbps by 1 gbps internet connection. This is a copper connection so no need for SPF connections. I am currently maxing out at 640 mbps by 890 mbps. When I plug directly into the modem (without pfsense) I get 980 mbps up and down. So it leads me to believe its something hardware-wise. It has the newest version of pfsense installed on it. It has a SuperMicro X8DTL motherboard loaded with dual Intel Xeon L5640's with a max speed of 2.266 ghz, 32 GB of RAM and 500 GB SSD drive. It has 3 NIC ports in it. The em0/em1 (embedded to the motherboard) using Intel 82574L. Then the PCI-e card has dual 82575EB using igb0/ibg1. I have tried both and get pretty close to the same speeds on both. What would your recommendations to achieve the speeds I am paying for. Thanks.
- doug
-
Do you have powerd enabled? You may not be seeing turbo frequencies otherwise. Though I would expect that to pass 1Gbps even at 2.2GHz. Have you tested locally just with hosts directly on either side of the firewall?
Do you have packages running? Traffic shaping?
Steve
-
Sorry about my lateness getting back to you. I do have powerd enabled. If i test behind pfsense i get 640/900. If i test with a computer connected right to the modem I get 980/980.
I currently run pfBlocker, RRD_Summary and snort.
Thank you for the help. Take care.
-
Ah, possibly Snort limiting it then. Does it pass full speed if you disable Snort on all interfaces?
-
I already disabled snort and pfBlocker and ran speed test. Did not increase the speeds at all.
-
Hi @dcreationsinc - since you have a very high speed WAN connection, have you already tried tuning the network cards on the pfSense system? Check out these threads for more info on parameters than can be adjusted:
https://forum.netgate.com/topic/101391/loader-conf-local-tuning-for-modern-hardware
https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-uploadThis page may also be helpful for troubleshooting - check out the section, "Where is the bottleneck ?"
https://bsdrp.net/documentation/technical_docs/performance
Hope this helps.
-
I installed the command line version of speedtest on the pfsense box just now. Directly on it the highest it gets is 652/538. But yet clients connected behind it see a lower download and a faster upload. Can someone help make some sense out of that? It has me baffled.
-
Testing to/from pfSense like that is not a great test in absolute terms as pfSense it not at all optimised for TCP termination in the way that a server would be. It's a router. It can be useful on lower speed connections or in revealing a problem on the LAN side.
Try running
top -aSHon pfSense whilst testing from a client behind it. See what load is being put on it and how that spreads across the cores.Is your connection PPPoE?
Steve
-
No our connection isnt PPPoE. Even when I run speed test from client computers the max i see is 500/890. I know I can get a lot better than that. I have tried it from multiple machines behind the pfsense box.
-
So run
top -aSHon the pfSense box whilst you are testing and see what sort of CPU usage you get there.Steve
-
@stephenw10 said in Hardware Recommendations:
top -aSH
On both upload and download the most i saw was 94% idle.
-
We need to see the actual output there, what is actually using the CPU and how it's spread across the cores.
-
-
@stephenw10 last pid: 34544; load averages: 0.31, 0.21, 0.26 up 9+10:56:36 10:21:03
442 processes: 26 running, 336 sleeping, 80 waiting
CPU: 0.6% user, 3.7% nice, 0.0% system, 1.2% interrupt, 94.5% idle
Mem: 297M Active, 6615M Inact, 753M Wired, 159M Buf, 24G Free
Swap: 3852M Total, 3852M FreePID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 384K CPU8 8 225.8H 100.00% [idle{idle: cpu8}]
11 root 155 ki31 0K 384K CPU2 2 225.8H 100.00% [idle{idle: cpu2}]
11 root 155 ki31 0K 384K CPU11 11 225.8H 100.00% [idle{idle: cpu11}]
11 root 155 ki31 0K 384K CPU7 7 225.2H 100.00% [idle{idle: cpu7}]
11 root 155 ki31 0K 384K CPU16 16 224.0H 100.00% [idle{idle: cpu16}]
11 root 155 ki31 0K 384K CPU15 15 224.0H 100.00% [idle{idle: cpu15}]
11 root 155 ki31 0K 384K CPU0 0 225.5H 98.04% [idle{idle: cpu0}]
11 root 155 ki31 0K 384K CPU4 4 224.0H 97.26% [idle{idle: cpu4}]
11 root 155 ki31 0K 384K CPU5 5 224.5H 96.96% [idle{idle: cpu5}]
11 root 155 ki31 0K 384K RUN 14 224.0H 96.92% [idle{idle: cpu14}]
11 root 155 ki31 0K 384K CPU6 6 224.8H 95.21% [idle{idle: cpu6}]
11 root 155 ki31 0K 384K CPU1 1 225.5H 93.02% [idle{idle: cpu1}]
11 root 155 ki31 0K 384K CPU3 3 225.8H 76.00% [idle{idle: cpu3}]
12 root -92 - 0K 1280K WAIT 3 36:37 24.02% [intr{irq259: igb0:que 3}]
12 root -92 - 0K 1280K WAIT 1 49:37 6.99% [intr{irq257: igb0:que 1}]
12 root -92 - 0K 1280K WAIT 6 74:02 4.71% [intr{irq263: igb1:que 2}]
12 root -92 - 0K 1280K WAIT 5 82:00 3.17% [intr{irq262: igb1:que 1}]
74499 root 20 0 9860K 5336K CPU14 14 0:00 3.08% top -aSH
12 root -92 - 0K 1280K WAIT 4 112:35 2.60% [intr{irq261: igb1:que 0}]
12 root -92 - 0K 1280K WAIT 0 52:52 1.81% [intr{irq256: igb0:que 0}]
9549 root 20 0 12904K 8152K select 8 0:00 0.30% sshd: root@pts/0 (sshd)
43572 root 20 0 10200K 5716K select 0 4:51 0.14% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
12 root -92 - 0K 1280K WAIT 2 34:38 0.05% [intr{irq258: igb0:que 2}]
36727 root 20 0 6900K 2456K nanslp 15 0:08 0.04% [dpinger{dpinger}]
37627 root 20 0 6900K 2456K nanslp 11 0:08 0.04% [dpinger{dpinger}]
13310 dhcpd 20 0 12576K 8068K select 7 0:29 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf
12 root -92 - 0K 1280K WAIT 7 57:00 0.02% [intr{irq264: igb1:que 3}]
36727 root 20 0 6900K 2456K sbwait 16 0:03 0.02% [dpinger{dpinger}]
12 root -60 - 0K 1280K WAIT 0 6:33 0.01% [intr{swi4: clock (0)}]
11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}] -
@dcreationsinc said in Hardware Recommendations:
11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]Hmm at least 4 CPUs are 0% idle.... that looks a little odd. What is that load if it's not shown....
That system is 24 apparent cores right?
The actual loading shown is not unexpected though it's not spread evenly at all. That was passing 600Mbps at the time?
igb0 is WAN there? And igb1 was the internal interface used for that test?Steve
-
@stephenw10 Yes its 24 cores. The load avgs are last pid: 34544; load averages: 0.31, 0.21, 0.26 ibg0 is WAN. That was passing 600 mbps at a time. Right now I am using ibg1 as my LAN port.
-
Hmm, I don't have anything to compare that with directly but I expect to see idle processes there for all 24 cores and I expect to see them all mostly idle. I'm unsure what the 0% idle processes for the other cores indicate there...
If we assume the load on cpu3 is the igb0 interrupt load it's still not a CPU limit. Did you try swapping the NICs in use there? Maybe put on of the em NICs on WAN as a test.
Steve
-
@stephenw10 The more i think about it I think it might be an incompatibility between the modem and pfsense. Primary the network cards used. The reason why I say this is because when I had cable internet (1 gbps/20 Mbps) I got 980 Mbps download threw the same pfsense box. Can you recommend a low profile network card preferably with at least dual ports on it that I can pick up? Thanks.
-
Hmm, I mean those em chipsets you're using are very common, I wouldn't have expected any issues with them.
I would look for something using the igb driver just so you know it's different. i350, i210 NICs are common and well tested.
Steve
-
Before running out and buying new hardware, have you tried tuning to see if that makes a performance difference?
-
@tman222 Yes I did try them. I had to remove them because my download speed went down by 100 mbps.
-
@dcreationsinc said in Hardware Recommendations:
@tman222 Yes I did try them. I had to remove them because my download speed went down by 100 mbps.
Have you tried to disable hyper-threading to see if that helps any? Is Turbo Boost enabled? This whole setup is running on bare metal (vs. virtualized), correct? Are there any other expansion cards installed in the system?
Also, under System / Advanced / Firewall & NAT do you by chance have the "IP Random id generation" enabled? If so, try disabling it to see if makes a difference in performance.
Hope this helps.
-
@tman222 There are no additional expansion cards in the system. This is running on bare metal. Keep in mind when I had cable internet (1 gbps / 20 mbps) I achieved 960-980 mbps download under the same configuration. I do not have random id generation enabled. I have tried it with and without hyperthreading. I have also tried it with and without turbo boost. Nothing seems to help. This is why I am leaning towards an incompatibility between the modem and the network card but I am open to ideas and suggestions prior to buying a new NIC. Thanks.
