Hardware Recommendations

dcreationsinc

Hello,

I am running into an issue. I have a 1 gbps by 1 gbps internet connection. This is a copper connection so no need for SPF connections. I am currently maxing out at 640 mbps by 890 mbps. When I plug directly into the modem (without pfsense) I get 980 mbps up and down. So it leads me to believe its something hardware-wise. It has the newest version of pfsense installed on it. It has a SuperMicro X8DTL motherboard loaded with dual Intel Xeon L5640's with a max speed of 2.266 ghz, 32 GB of RAM and 500 GB SSD drive. It has 3 NIC ports in it. The em0/em1 (embedded to the motherboard) using Intel 82574L. Then the PCI-e card has dual 82575EB using igb0/ibg1. I have tried both and get pretty close to the same speeds on both. What would your recommendations to achieve the speeds I am paying for. Thanks.

doug

stephenw10

Do you have powerd enabled? You may not be seeing turbo frequencies otherwise. Though I would expect that to pass 1Gbps even at 2.2GHz. Have you tested locally just with hosts directly on either side of the firewall?

Do you have packages running? Traffic shaping?

Steve

dcreationsinc

Sorry about my lateness getting back to you. I do have powerd enabled. If i test behind pfsense i get 640/900. If i test with a computer connected right to the modem I get 980/980.

I currently run pfBlocker, RRD_Summary and snort.

Thank you for the help. Take care.

stephenw10

Ah, possibly Snort limiting it then. Does it pass full speed if you disable Snort on all interfaces?

dcreationsinc

I already disabled snort and pfBlocker and ran speed test. Did not increase the speeds at all.

tman222

Hi @dcreationsinc - since you have a very high speed WAN connection, have you already tried tuning the network cards on the pfSense system? Check out these threads for more info on parameters than can be adjusted:

https://forum.netgate.com/topic/101391/loader-conf-local-tuning-for-modern-hardware
https://forum.netgate.com/topic/117072/dsl-reports-speed-test-causing-crash-on-upload

This page may also be helpful for troubleshooting - check out the section, "Where is the bottleneck ?"

https://bsdrp.net/documentation/technical_docs/performance

Hope this helps.

dcreationsinc

I installed the command line version of speedtest on the pfsense box just now. Directly on it the highest it gets is 652/538. But yet clients connected behind it see a lower download and a faster upload. Can someone help make some sense out of that? It has me baffled.

stephenw10

Testing to/from pfSense like that is not a great test in absolute terms as pfSense it not at all optimised for TCP termination in the way that a server would be. It's a router. It can be useful on lower speed connections or in revealing a problem on the LAN side.

Try running top -aSH on pfSense whilst testing from a client behind it. See what load is being put on it and how that spreads across the cores.

Is your connection PPPoE?

Steve

dcreationsinc

No our connection isnt PPPoE. Even when I run speed test from client computers the max i see is 500/890. I know I can get a lot better than that. I have tried it from multiple machines behind the pfsense box.

stephenw10

So run top -aSH on the pfSense box whilst you are testing and see what sort of CPU usage you get there.

Steve

dcreationsinc

@stephenw10 said in Hardware Recommendations:

top -aSH

On both upload and download the most i saw was 94% idle.

stephenw10

We need to see the actual output there, what is actually using the CPU and how it's spread across the cores.

dcreationsinc

@stephenw10

dcreationsinc

@stephenw10 last pid: 34544; load averages: 0.31, 0.21, 0.26 up 9+10:56:36 10:21:03
442 processes: 26 running, 336 sleeping, 80 waiting
CPU: 0.6% user, 3.7% nice, 0.0% system, 1.2% interrupt, 94.5% idle
Mem: 297M Active, 6615M Inact, 753M Wired, 159M Buf, 24G Free
Swap: 3852M Total, 3852M Free

PID USERNAME PRI NICE SIZE 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 12 root -92 - 12 root -92 - 12 root -92 - 12 root -92 - 74499 root 12 root -92 - 12 root -92 - 9549 root 43572 root 12 root -92 - 36727 root 37627 root 13310 dhcpd 12 root -92 - 36727 root 12 root -60 - 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 11 root 155 ki31 RES STATE C TIME WCPU COMMAND
0K 384K CPU8 8 225.8H 100.00% [idle{idle: cpu8}]
0K 384K CPU2 2 225.8H 100.00% [idle{idle: cpu2}]
0K 384K CPU11 11 225.8H 100.00% [idle{idle: cpu11}]
0K 384K CPU7 7 225.2H 100.00% [idle{idle: cpu7}]
0K 384K CPU16 16 224.0H 100.00% [idle{idle: cpu16}]
0K 384K CPU15 15 224.0H 100.00% [idle{idle: cpu15}]
0K 384K CPU0 0 225.5H 98.04% [idle{idle: cpu0}]
0K 384K CPU4 4 224.0H 97.26% [idle{idle: cpu4}]
0K 384K CPU5 5 224.5H 96.96% [idle{idle: cpu5}]
0K 384K RUN 14 224.0H 96.92% [idle{idle: cpu14}]
0K 384K CPU6 6 224.8H 95.21% [idle{idle: cpu6}]
0K 384K CPU1 1 225.5H 93.02% [idle{idle: cpu1}]
0K 384K CPU3 3 225.8H 76.00% [idle{idle: cpu3}]
0K 1280K WAIT 3 36:37 24.02% [intr{irq259: igb0:que 3}]
0K 1280K WAIT 1 49:37 6.99% [intr{irq257: igb0:que 1}]
0K 1280K WAIT 6 74:02 4.71% [intr{irq263: igb1:que 2}]
0K 1280K WAIT 5 82:00 3.17% [intr{irq262: igb1:que 1}]
20 0 9860K 5336K CPU14 14 0:00 3.08% top -aSH
0K 1280K WAIT 4 112:35 2.60% [intr{irq261: igb1:que 0}]
0K 1280K WAIT 0 52:52 1.81% [intr{irq256: igb0:que 0}]
20 0 12904K 8152K select 8 0:00 0.30% sshd: root@pts/0 (sshd)
20 0 10200K 5716K select 0 4:51 0.14% /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
0K 1280K WAIT 2 34:38 0.05% [intr{irq258: igb0:que 2}]
20 0 6900K 2456K nanslp 15 0:08 0.04% [dpinger{dpinger}]
20 0 6900K 2456K nanslp 11 0:08 0.04% [dpinger{dpinger}]
20 0 12576K 8068K select 7 0:29 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf
0K 1280K WAIT 7 57:00 0.02% [intr{irq264: igb1:que 3}]
20 0 6900K 2456K sbwait 16 0:03 0.02% [dpinger{dpinger}]
0K 1280K WAIT 0 6:33 0.01% [intr{swi4: clock (0)}]
0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]

stephenw10

@dcreationsinc said in Hardware Recommendations:

11 root 155 ki31 0K 384K CPU9 9 225.8H 0.00% [idle{idle: cpu9}]
11 root 155 ki31 0K 384K CPU10 10 225.8H 0.00% [idle{idle: cpu10}]
11 root 155 ki31 0K 384K RUN 19 224.1H 0.00% [idle{idle: cpu19}]
11 root 155 ki31 0K 384K CPU13 13 224.1H 0.00% [idle{idle: cpu13}]

Hmm at least 4 CPUs are 0% idle.... that looks a little odd. What is that load if it's not shown....

That system is 24 apparent cores right?

The actual loading shown is not unexpected though it's not spread evenly at all. That was passing 600Mbps at the time?
igb0 is WAN there? And igb1 was the internal interface used for that test?

Steve

dcreationsinc

@stephenw10 Yes its 24 cores. The load avgs are last pid: 34544; load averages: 0.31, 0.21, 0.26 ibg0 is WAN. That was passing 600 mbps at a time. Right now I am using ibg1 as my LAN port.

stephenw10

Hmm, I don't have anything to compare that with directly but I expect to see idle processes there for all 24 cores and I expect to see them all mostly idle. I'm unsure what the 0% idle processes for the other cores indicate there...

If we assume the load on cpu3 is the igb0 interrupt load it's still not a CPU limit. Did you try swapping the NICs in use there? Maybe put on of the em NICs on WAN as a test.

Steve

dcreationsinc

@stephenw10 The more i think about it I think it might be an incompatibility between the modem and pfsense. Primary the network cards used. The reason why I say this is because when I had cable internet (1 gbps/20 Mbps) I got 980 Mbps download threw the same pfsense box. Can you recommend a low profile network card preferably with at least dual ports on it that I can pick up? Thanks.

stephenw10

Hmm, I mean those em chipsets you're using are very common, I wouldn't have expected any issues with them.

I would look for something using the igb driver just so you know it's different. i350, i210 NICs are common and well tested.

Steve

tman222

Before running out and buying new hardware, have you tried tuning to see if that makes a performance difference?