10gbps performance issue
-
Hi all. Here is the setup:
linux server [bond0] = 10gb switch = [lagg1] pfsense [lagg0]= ISP
pfsense uses Qlogic 10Gb NICs:$ grep bxe /var/log/dmesg.boot | grep QL bxe0: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.90 bxe1: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.90 bxe2: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.90 bxe3: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.90ISP connection is 1Gbps.
When LRO is off the links perform as follows:
linux-pfsense - 2Gbps
pfsense-ISP - 1Gbps
linux-ISP - 1Gbps(Speed is measured with iperf)
After switching LRO on (to achieve 10Gbps at the linux-pfsense link), we get:
linux-pfsense - 10Gbps
pfsense-ISP - 1Gbps
linux-ISP - 2MbpsI tried to tune pfSense like outlined at https://forum.netgate.com/post/738428 - didn't help at all.
Please, help me to figure out how to tune pfsense to keep speeds at their highest for all connections.
-
Those are all download speeds I assume? How are you running iperf?
What CPU is that box running?
bxe may have a sysctl to set LRO per interface. I don't have access to anything running it to check.
Steve
-
Hi @stephenw10 , thanks for a quick reply!
Iperf is launched pretty simple: iperf -c at the client and iperf -s at pfsense. Tried to manipulate with the tcp window size (-w) - didn't help. Also I tried to specify multiple flows (-P 10) and was able to get ~4.5Gbps. Further increasing the flows amount did not affect the rate.
pfSense has the following CPU:CPU Type Intel(R) Xeon(R) CPU E5620 @ 2.40GHz Current: 2394 MHz, Max: 2395 MHz 8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (active)Setting up LRO for any interface is not an option, since it degrades performance for linux-ISP connection (I gave it a try).
Interesting, that running iperf (single flow) in the opposite direction (iperf -c at pfsense and iperf -s at linux) shows 10Gbps throughput. So, the issue is just with linux-pfsense direction.It's also worth to note, that there're no errors or collisions on any interfaces (linux, pfsense, switch).
Any ideas are kindly appreciated!
-
I assumed you enabled LRO globally in that test where you were seeing only 2Mbps from the client to the ISP?
Where was the iperf server in the tests involving the ISP?
You can enable/disable LRO per interface using ifconfig as a test. Since that affected the Linux-ISP connection so badly I would assume that was a download test with pfSense receiving on the WAN?
Steve
-
Not trying to step on the discussion here, but:
> Iperf is launched pretty simple: iperf -c at the client and iperf -s at pfsense
If you're actually launching "iperf -s" on your pfSense box, then you're testing speeds TO pfSense not THROUGH pfSense. You probably want to run iperf on the server behind your firewall.
just my $.02
-
- I tried enabling LRO globally as well as per-interface (ifconfig lagg1 lro; ifconfig lagg0 -lro). 2Mbps rate happened for any variant of enabled LRO
- The ISP has its own iperf server - iperf.he.net
- iperf -c is generating traffic, so pfSense was receiving traffic on LAN and forwarded it to WAN
A lot of docs do not recommend to turn LRO on a router (which does sound reasonable), so I'd like to achieve 10Gbps on linux-pfSense link w/o LRO (if that's possible).
-
@divsys , you're absolutely right. I tried running iperf -c against pfsense as well as the ISP's iperf server.
-
Yes testing directly to or from pfSense is not representative of throughput but it can be useful for pinning down a throttling problem.
Here you were seeing 10Gb to pfSense and 1Gb from pfSense to the ISP but only 2Mb through both. Which is odd.However without LRO you're seeing the full 1Gb from the client to the ISP.
What actually bandwidth throttli8ng are you seeing there? Between internal interfaces perhaps?
Steve
-
I guess the bandwidth between linux and pfsense should be 10Gbps without enabling LRO. If I understood your question correctly.
-
You might expect that but when running as a router/firewall connections are not normally terminated on the firewall. The exception might be if you're running Squid for example.
Since your WAN is 1Gbps the actual firewall throughput for a connection to/from the internet cannot exceed that. So if you're seeing 2Gbps to the firewall it's not throttling that.
With LRO disabled you are seeing 1Gbps from a Linux client to your ISP. That's the maximum you can get. SO where are you actually seeing less bandwidth than you expect other than testing to the firewall itself which never normally happens?
Steve
-
We're in the process of settings things up for a new environment and would like to make sure that they work properly. I agree there're a limited number of tasks when such a high throughput required against pfSense itself, but they exist and we wouldn't like to get into a situation when we'll have to troubleshoot things on the live production system.
Please, help me to find a reason for 2Gbps rate from a server to pfSense?
-
What is the CPU usage when you are running that test?
Try running
top -aSHin another console window. Are any CPU threads running at or near 100%?Steve
-
Here is the top output during the iperf test (linux is a client, pfsense is a server). I do not see an overload here.
last pid: 54739; load averages: 0.27, 0.15, 0.10 up 3+02:29:33 05:12:44 264 processes: 12 running, 198 sleeping, 54 waiting CPU: 1.4% user, 0.0% nice, 9.3% system, 12.6% interrupt, 76.8% idle Mem: 67M Active, 680M Inact, 574M Wired, 84M Buf, 10G Free Swap: 3881M Total, 3881M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -92 - 0K 880K CPU1 1 2:44 99.93% [intr{irq273: bxe2:fp01}] 11 root 155 ki31 0K 128K RUN 7 74.4H 98.14% [idle{idle: cpu7}] 11 root 155 ki31 0K 128K CPU0 0 74.4H 97.33% [idle{idle: cpu0}] 11 root 155 ki31 0K 128K CPU6 6 74.4H 95.98% [idle{idle: cpu6}] 11 root 155 ki31 0K 128K RUN 5 74.4H 86.53% [idle{idle: cpu5}] 54395 root 84 0 28552K 4508K CPU7 7 0:06 84.71% iperf -s{iperf} 11 root 155 ki31 0K 128K CPU3 3 74.4H 80.39% [idle{idle: cpu3}] 11 root 155 ki31 0K 128K RUN 2 74.4H 79.07% [idle{idle: cpu2}] 11 root 155 ki31 0K 128K CPU4 4 74.4H 72.06% [idle{idle: cpu4}] 54395 root 20 0 28552K 4508K nanslp 4 0:00 0.74% iperf -s{iperf} 12120 root 40 20 683M 524M CPU2 2 0:25 0.18% /usr/local/bin/snort -R 41368 -D -q --suppress-config-log -l /var/ 12 root -60 - 0K 880K WAIT 0 3:30 0.08% [intr{swi4: clock (0)}] 11 root 155 ki31 0K 128K RUN 1 74.4H 0.07% [idle{idle: cpu1}] 54739 root 20 0 22116K 4816K CPU5 5 0:00 0.07% top -aSH 12587 root 40 20 51952K 17220K nanslp 5 0:08 0.03% /usr/local/bin/barnyard2 -r 41368 -f snort_41368_lagg0.u2 --pid-pa 12 root -92 - 0K 880K WAIT 0 0:09 0.02% [intr{irq267: bxe1:fp00}]iperf result:
[2.4.3-RELEASE][admin@pfSense]/root: iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 53986 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 2.52 GBytes 2.16 Gbits/secThe top output for the opposite direction (linux is a server, pfsense is a client):
last pid: 21988; load averages: 0.13, 0.16, 0.10 up 3+02:32:16 05:15:27 263 processes: 9 running, 199 sleeping, 55 waiting CPU: 0.1% user, 0.0% nice, 8.4% system, 8.4% interrupt, 83.0% idle Mem: 66M Active, 681M Inact, 575M Wired, 84M Buf, 10G Free Swap: 3881M Total, 3881M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 128K CPU7 7 74.4H 100.00% [idle{idle: cpu7}] 11 root 155 ki31 0K 128K RUN 1 74.4H 99.88% [idle{idle: cpu1}] 11 root 155 ki31 0K 128K CPU2 2 74.4H 98.56% [idle{idle: cpu2}] 11 root 155 ki31 0K 128K CPU5 5 74.4H 88.79% [idle{idle: cpu5}] 11 root 155 ki31 0K 128K CPU3 3 74.4H 85.63% [idle{idle: cpu3}] 11 root 155 ki31 0K 128K CPU4 4 74.4H 81.41% [idle{idle: cpu4}] 11 root 155 ki31 0K 128K CPU6 6 74.4H 72.80% [idle{idle: cpu6}] 21988 root 52 0 26376K 3852K sbwait 2 0:03 68.84% iperf -c 10.10.10.20{iperf} 12 root -92 - 0K 880K WAIT 0 2:57 62.64% [intr{irq272: bxe2:fp00}] 11 root 155 ki31 0K 128K RUN 0 74.4H 37.34% [idle{idle: cpu0}] 0 root -92 - 0K 832K - 5 0:00 1.00% [kernel{bxe2_fp0_tq}] 12120 root 40 20 683M 524M bpf 6 0:25 0.13% /usr/local/bin/snort -R 41368 -D -q --suppress-config-log -l /var/ 54739 root 20 0 22116K 4816K CPU1 1 0:00 0.10% top -aSHiperf result:
[2.4.3-RELEASE][admin@pfSense]/root: iperf -c 10.10.10.20 ------------------------------------------------------------ Client connecting to 10.10.10.20, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 10.10.10.254 port 15711 connected with 10.10.10.20 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 11.0 GBytes 9.41 Gbits/secJust in case, here is the local iperf test:
[2.4.3-RELEASE][admin@pfSense]/root: iperf -c localhost ------------------------------------------------------------ Client connecting to localhost, TCP port 5001 TCP window size: 144 KByte (default) ------------------------------------------------------------ [ 3] local 127.0.0.1 port 13072 connected with 127.0.0.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 18.0 GBytes 15.5 Gbits/sec -
You have one CPU core running at 100% (~0% idle):
11 root 155 ki31 0K 128K RUN 1 74.4H 0.07% [idle{idle: cpu1}]You probably have (at least) 4 queues per NIC so it would be worth running that test with
-P 4at the client to spread the load better.Steve
-
[2.4.3-RELEASE][admin@pfSense]/root: iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 53996 [ 5] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 53998 [ 6] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 54000 [ 7] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 54002 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.36 GBytes 1.16 Gbits/sec [ 5] 0.0-10.0 sec 1.34 GBytes 1.15 Gbits/sec [ 6] 0.0-10.0 sec 1.32 GBytes 1.13 Gbits/sec [ 7] 0.0-10.0 sec 1.32 GBytes 1.13 Gbits/sec [SUM] 0.0-10.0 sec 5.34 GBytes 4.58 Gbits/seclast pid: 34460; load averages: 1.15, 0.32, 0.16 up 3+03:52:37 06:35:48 267 processes: 17 running, 199 sleeping, 51 waiting CPU: 5.9% user, 0.0% nice, 30.7% system, 50.0% interrupt, 13.4% idle Mem: 67M Active, 683M Inact, 576M Wired, 84M Buf, 10G Free Swap: 3881M Total, 3881M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root -92 - 0K 880K CPU0 0 3:08 99.93% [intr{irq272: bxe2:fp00}] 12 root -92 - 0K 880K CPU3 3 3:02 99.91% [intr{irq275: bxe2:fp03}] 12 root -92 - 0K 880K CPU2 2 3:02 99.88% [intr{irq274: bxe2:fp02}] 12 root -92 - 0K 880K CPU1 1 2:55 99.86% [intr{irq273: bxe2:fp01}] 34352 root 83 0 35080K 6772K CPU6 6 0:06 77.43% iperf -s{iperf} 34352 root 52 0 35080K 6772K CPU4 4 0:06 76.79% iperf -s{iperf} 34352 root 52 0 35080K 6772K CPU7 7 0:06 76.59% iperf -s{iperf} 34352 root 52 0 35080K 6772K CPU6 6 0:06 76.52% iperf -s{iperf} 11 root 155 ki31 0K 128K RUN 7 75.8H 22.62% [idle{idle: cpu7}] 11 root 155 ki31 0K 128K RUN 6 75.8H 22.55% [idle{idle: cpu6}] 11 root 155 ki31 0K 128K RUN 5 75.8H 22.49% [idle{idle: cpu5}] 11 root 155 ki31 0K 128K RUN 4 75.8H 22.46% [idle{idle: cpu4}] 34352 root 20 0 35080K 6772K nanslp 6 0:00 2.14% iperf -s{iperf} 11 root 155 ki31 0K 128K RUN 2 75.8H 0.12% [idle{idle: cpu2}] 11 root 155 ki31 0K 128K RUN 3 75.8H 0.12% [idle{idle: cpu3}] 11 root 155 ki31 0K 128K RUN 0 75.7H 0.12% [idle{idle: cpu0}] 11 root 155 ki31 0K 128K RUN 1 75.8H 0.11% [idle{idle: cpu1}] 34460 root 20 0 22116K 4820K CPU5 5 0:00 0.10% top -aSH 12 root -60 - 0K 880K WAIT 5 3:34 0.09% [intr{swi4: clock (0)}]Wow.. This does seem as a CPU limit.. Wondering why linux box's CPU (Intel(R) Xeon(R) CPU X5670 @ 2.93GHz) "eats" 10Gbps w/o issues:
top - 06:45:18 up 4 days, 21:26, 3 users, load average: 0.09, 0.03, 0.01 Threads: 426 total, 2 running, 424 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.0 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 0.0 us, 1.3 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 1.3 si, 0.0 st %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu9 : 0.3 us, 55.9 sy, 0.0 ni, 43.4 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu12 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu13 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu14 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu15 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu16 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu17 : 0.3 us, 0.0 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu18 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu19 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu20 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu21 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu22 : 0.0 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu23 : 0.0 us, 2.2 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st KiB Mem : 65965828 total, 63945296 free, 394004 used, 1626528 buff/cache KiB Swap: 67096572 total, 67096572 free, 0 used. 65039256 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20962 myuser 20 0 236696 2208 1924 R 60.6 0.0 0:01.83 iperf -s 125 root 20 0 0 0 0 S 1.0 0.0 0:00.87 [ksoftirqd/23] 34 root 20 0 0 0 0 S 0.7 0.0 0:00.26 [ksoftirqd/5] 14 root 20 0 0 0 0 S 0.3 0.0 0:00.29 [ksoftirqd/1] 55 root 20 0 0 0 0 S 0.3 0.0 0:00.36 [ksoftirqd/9] 16776 root 20 0 0 0 0 S 0.3 0.0 0:03.47 [kworker/9:1][2.4.3-RELEASE][admin@pfSense]/root: iperf -c 10.10.10.20 ------------------------------------------------------------ Client connecting to 10.10.10.20, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 3] local 10.10.10.254 port 52919 connected with 10.10.10.20 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 10.8 GBytes 9.26 Gbits/secAny ideas?
-
What are the NICs in the Linux box? If the NICs are the same, most likely the drivers are different.
Also try:
- disabling Hyperthreading on the pfsense box.
- disabling flow-controll everywhere including the switch
- increasing the interrupt max rate of interrupts
To be honest it's a bit pointless to test througput. Better test PPS through the pfsense box. This will expose your PPS limit based on the CPU/NIC/Settings/Firewall Configuration combination.
You can check with netstat -ihw 1 where the drop happens.
-
The load due to pf shows in those values in pfSense. Is the Linux box running any sort of firewall?
Try disabling pf temporarily as a test.
But this is still not a test of the firewall throughput. I'm still unsure what you're trying to achieve here. Your WAN is 1Gbps and you are able to see that fully from a client behind pfSense. If you want to test more than that use a 10Gbps WAN to see what it can pass.
Steve
-
@stephenw10 with pf disabled it shows slightly better perfromance:
pf disabled [2.4.3-RELEASE][admin@pfSense]/root: [2.4.3-RELEASE][admin@pfSense]/root: iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 4] local 10.10.10.254 port 5001 connected with 10.10.10.20 port 54942 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 3.91 GBytes 3.35 Gbits/sec4 flows test gives:
[SUM] 0.0-10.0 sec 8.03 GBytes 6.88 Gbits/secI don't think the ISP can provide us 10Gbps link at the moment. But later it's possible. And it wouldn't be great to face such an issue when all systems are in production.
I'm trying to figure out why the speed is not the expected one. The next steps in the list is to disable hyperthreading and upgrade the CPU. I'll post the results here.
Anyway, if you have any other ideas why the CPU is so slow comparing to the linux box, I'd be more than happy to check them. -
I believe the devs should remove iperf from base installs....
These iperf threads keep popping up every month & conclusion is always the same:
Don't run iperf on pfsenseThe only way to measure throughput is like this:
(Iperf-server)----(pfsense)----(iperf-client)
All other measurements are pointless and inaccurate. -
@heper hope devs would not follow your suggestion. it's like "we've got a headache. let's cut the head out". very wise.