Hardware Recommendations
-
Hi @dcreationsinc - since you have a very high speed WAN connection, have you already tried tuning the network cards on the pfSense system? Check out these threads for more info on parameters than can be adjusted:
https://forum.netgate.com/topic/101391/loader-conf-local-tuning-for-modern-hardware
https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-uploadThis page may also be helpful for troubleshooting - check out the section, "Where is the bottleneck ?"
https://bsdrp.net/documentation/technical_docs/performance
Hope this helps.
-
I installed the command line version of speedtest on the pfsense box just now. Directly on it the highest it gets is 652/538. But yet clients connected behind it see a lower download and a faster upload. Can someone help make some sense out of that? It has me baffled.
-
Testing to/from pfSense like that is not a great test in absolute terms as pfSense it not at all optimised for TCP termination in the way that a server would be. It's a router. It can be useful on lower speed connections or in revealing a problem on the LAN side.
Try running
top -aSH
on pfSense whilst testing from a client behind it. See what load is being put on it and how that spreads across the cores.Is your connection PPPoE?
Steve
-
No our connection isnt PPPoE. Even when I run speed test from client computers the max i see is 500/890. I know I can get a lot better than that. I have tried it from multiple machines behind the pfsense box.
-
So run
top -aSH
on the pfSense box whilst you are testing and see what sort of CPU usage you get there.Steve
-
@stephenw10 said in Hardware Recommendations:
top -aSH
On both upload and download the most i saw was 94% idle.
-
We need to see the actual output there, what is actually using the CPU and how it's spread across the cores.
-
-
@stephenw10 last pid: 34544; load averages: 0.31, 0.21, 0.26 up 9+10:56:36 10:21:03
442 processes: 26 running, 336 sleeping, 80 waiting
CPU: 0.6% user, 3.7% nice, 0.0% system, 1.2% interrupt, 94.5% idle
Mem: 297M Active, 6615M Inact, 753M Wired, 159M Buf, 24G Free
Swap: 3852M Total, 3852M FreePID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 384K CPU8 8 225.8H 100.00% [idle{idle: cpu8}]
11 root 155 ki31 0K 384K CPU2 2 225.8H 100.00% [idle{idle: cpu2}]
11 root 155 ki31 0K 384K CPU11 11 225.8H 100.00% [idle{idle: cpu11}]
11 root 155 ki31 0K 384K CPU7 7 225.2H 100.00% [idle{idle: cpu7}]
11 root 155 ki31 0K 384K CPU16 16 224.0H 100.00% [idle{idle: cpu16}]
11 root 155 ki31 0K 384K CPU15 15 224.0H 100.00% [idle{idle: cpu15}]
11 root 155 ki31 0K 384K CPU0 0 225.5H 98.04% [idle{idle: cpu0}]
11 root 155 ki31 0K 384K CPU4 4 224.0H 97.26% [idle{idle: cpu4}]
11 root 155 ki31 0K 384K CPU5 5 224.5H 96.96% [idle{idle: cpu5}]
11 root 155 ki31 0K 384K RUN 14 224.0H 96.92% [idle{idle: cpu14}]
11 root 155 ki31 0K 384K CPU6 6 224.8H 95.21% [idle{idle: cpu6}]
11 root 155 ki31 0K 384K CPU1 1 225.5H 93.02% [idle{idle: cpu1}]
11 root 155 ki31 0K 384K CPU3 3 225.8H 76.00% [idle{idle: cpu3}]
12 root -92 - 0K 1280K WAIT 3 36:37 24.02% [intr{irq259: igb0:que 3}]
12 root -92 - 0K 1280K WAIT 1 49:37 6.99% [intr{irq257: igb0:que 1}]
12 root -92 - 0K 1280K WAIT 6 74:02 4.71% [intr{irq263: igb1:que 2}]
12 root -92 - 0K 1280K WAIT 5 82:00 3.17% [intr{irq262: igb1:que 1}]
74499 root 20 0 9860K 5336K CPU14 14 0:00 3.08% top -aSH
12 root -92 - 0K 1280K WAIT 4 112:35 2.60% [intr{irq261: igb1:que 0}]
12 root -92 - 0K 1280K WAIT 0 52:52 1.81% [intr{irq256: igb0:que 0}]
9549 root 20 0 12904K 8152K select 8 0:00 0.30% sshd: root@pts/0 (sshd)
43572 root 20 0 10200K 5716K select 0 4:51 0.14% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
12 root -92 - 0K 1280K WAIT 2 34:38 0.05% [intr{irq258: igb0:que 2}]
36727 root 20 0 6900K 2456K nanslp 15 0:08 0.04% [dpinger{dpinger}]
37627 root 20 0 6900K 2456K nanslp 11 0:08 0.04% [dpinger{dpinger}]
13310 dhcpd 20 0 12576K 8068K select 7 0:29 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf
12 root -92 - 0K 1280K WAIT 7 57:00 0.02% [intr{irq264: igb1:que 3}]
36727 root 20 0 6900K 2456K sbwait 16 0:03 0.02% [dpinger{dpinger}]
12 root -60 - 0K 1280K WAIT 0 6:33 0.01% [intr{swi4: clock (0)}]
11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}] -
@dcreationsinc said in Hardware Recommendations:
11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]Hmm at least 4 CPUs are 0% idle.... that looks a little odd. What is that load if it's not shown....
That system is 24 apparent cores right?
The actual loading shown is not unexpected though it's not spread evenly at all. That was passing 600Mbps at the time?
igb0 is WAN there? And igb1 was the internal interface used for that test?Steve
-
@stephenw10 Yes its 24 cores. The load avgs are last pid: 34544; load averages: 0.31, 0.21, 0.26 ibg0 is WAN. That was passing 600 mbps at a time. Right now I am using ibg1 as my LAN port.
-
Hmm, I don't have anything to compare that with directly but I expect to see idle processes there for all 24 cores and I expect to see them all mostly idle. I'm unsure what the 0% idle processes for the other cores indicate there...
If we assume the load on cpu3 is the igb0 interrupt load it's still not a CPU limit. Did you try swapping the NICs in use there? Maybe put on of the em NICs on WAN as a test.
Steve
-
@stephenw10 The more i think about it I think it might be an incompatibility between the modem and pfsense. Primary the network cards used. The reason why I say this is because when I had cable internet (1 gbps/20 Mbps) I got 980 Mbps download threw the same pfsense box. Can you recommend a low profile network card preferably with at least dual ports on it that I can pick up? Thanks.
-
Hmm, I mean those em chipsets you're using are very common, I wouldn't have expected any issues with them.
I would look for something using the igb driver just so you know it's different. i350, i210 NICs are common and well tested.
Steve
-
Before running out and buying new hardware, have you tried tuning to see if that makes a performance difference?
-
@tman222 Yes I did try them. I had to remove them because my download speed went down by 100 mbps.
-
@dcreationsinc said in Hardware Recommendations:
@tman222 Yes I did try them. I had to remove them because my download speed went down by 100 mbps.
Have you tried to disable hyper-threading to see if that helps any? Is Turbo Boost enabled? This whole setup is running on bare metal (vs. virtualized), correct? Are there any other expansion cards installed in the system?
Also, under System / Advanced / Firewall & NAT do you by chance have the "IP Random id generation" enabled? If so, try disabling it to see if makes a difference in performance.
Hope this helps.
-
@tman222 There are no additional expansion cards in the system. This is running on bare metal. Keep in mind when I had cable internet (1 gbps / 20 mbps) I achieved 960-980 mbps download under the same configuration. I do not have random id generation enabled. I have tried it with and without hyperthreading. I have also tried it with and without turbo boost. Nothing seems to help. This is why I am leaning towards an incompatibility between the modem and the network card but I am open to ideas and suggestions prior to buying a new NIC. Thanks.