600MB down to 10MB Performance Issue
-
My pfSense box performance is awful and I just cant seem to find a good explanation. Hoping someone can point me in the right direction.
Hardware: HP Proliant DL360G6. 4xCPU, 4GB RAM, 256G HDD
ISP: Comcast Business Class @ 100MB
Users: ~250System is configured pretty vanilla. OpenVPN is configured for admin access, about 5 inbound rules, no proxys or filters running.
Here is the symptoms of what I am seeing:
- Client going through the firewall is averaging around 13MB
- Client bypassing the firewall directly into the Comcast router is averaging 98MB
- Client running iperf is getting ~600MB to the pfSense box on both the LAN and WAN sides
Most of the articles I find on performance point to network cards but I am getting great performance running iperf tests. Below is my top system activity which doesnt seem to show hardly any load on the system. All the advanced network settings are out of the box defaults.
I have plenty of horsepower, network performance seems good, the Internet connection itself seems fine. Any suggestions on where to look next?
_last pid: 39659; load averages: 0.02, 0.05, 0.06 up 56+07:32:34 16:58:08
138 processes: 5 running, 106 sleeping, 2 zombie, 25 waitingMem: 76M Active, 79M Inact, 121M Wired, 552K Cache, 112M Buf, 3213M Free
Swap: 8192M Total, 8192M FreePID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 32K CPU1 1 1337.0 100.00% [idle{idle: cpu1}]
11 root 171 ki31 0K 32K CPU0 0 1318.6 100.00% [idle{idle: cpu0}]
11 root 171 ki31 0K 32K RUN 2 1315.1 100.00% [idle{idle: cpu2}]
11 root 171 ki31 0K 32K CPU3 3 1293.9 99.85% [idle{idle: cpu3}]
12 root -68 - 0K 200K WAIT 0 17.3H 2.59% [intr{irq257: bce0}]
12 root -68 - 0K 200K WAIT 0 756:53 1.76% [intr{irq258: bce1}]_ -
What CPU do you have?
Intel chips can under perform when the L2 cache gets its wrong with out of branch execution and some other code instructions. Whether that would equate to the drop in performance that you see I dont know.
What if you rolled things back a stage, like see how things are before OpenVPN is installed, or try a real default install to see if you see the same performance drops.
You might spot which config change caused the problem, ie have you changed the MTU's as an example?
-
Interesting on the L2 cache. These CPUs also have L3. Not sure I can disable that but its worth a shot to try.
CPU Type Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
4 CPUs: 1 package(s) x 4 core(s)I did not roll back OpenVPN yet but I did stop the services and test with no luck. In fact I shut down every service I could and still poor performance.
Have not changed any network settings, including MTUs.
I do plan on going back to default and testing but have to wait for a downtime window that I can get in there and do it.
Thanks for the advice.
-
Have you checked for eny interface speed mismatch settings??
-
Yes. Both interfaces are 1G / FD. I thought of that but what is interesting is when I run a iperf test between a client and the server I get great throughput.
-
My first check with that level of throttling would be an interface mismatch too. Check the Status: Interfaces: page for errors or collisions.
Read this: https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
You're seeing good bandwidth bewteen a client and the LAN interface so the throttling is presumed to be on the LAN side. You can check that by running a download test on the firewall itself though. At the command line:
fetch -o /dev/null http://cachefly.cachefly.net/10mb.testYou have listed everything in Bytes (B) but I assume some of that is bits (b). That can confuse matters, a lot! ;)
Steve
-
Thanks for the reply. I thought about the interface mismatch as well and have checked that. Both are good:
WAN interface (bce0):
Media 1000baseT <full-duplex,flowcontrol,rxpause,txpause>In/out packets 25154099/13742928 (29.89 GB/2.68 GB)
In/out packets (pass) 25154099/13742928 (29.89 GB/2.68 GB)
In/out packets (block) 44073/479 (4.46 MB/55 KB)
In/out errors 0/0
Collisions 0LAN interface (bce1):
Media 1000baseT <full-duplex,flowcontrol,master,rxpause,txpause>In/out packets 12789344/24380268 (2.35 GB/29.48 GB)
In/out packets (pass) 12789344/24380268 (2.35 GB/29.48 GB)
In/out packets (block) 19664/2 (2.13 MB/152 bytes)
In/out errors 0/0
Collisions 0I ran the download on the firewall and the result was 1246 kBps. If I convert that right I am at 9.734375 Mbps.
Sorry about the B vs b, you are right big difference. I have been going through the troubleshooting & tuning document. Unfortunately I cant reboot at the moment so I will have to wait and see.
I have confirmed I dont have any traffic shaping on. One thing I have noticed is the issue only seems to be on incoming traffic. The line is 100 Mbps down and 20 Mbps up. I can hit the upstream limit every time, the downstream is only getting ~ 10-13 Mbps.</full-duplex,flowcontrol,master,rxpause,txpause></full-duplex,flowcontrol,rxpause,txpause>
-
Can you set it to 1000 Auto on both ends and no flowcontrol and rx/tx off??
And try again….
-
Ok, I set both sides to:
1000baseT <full-duplex>and set the switch to the same. No change in the result.</full-duplex>
-
Can you set it to full auto pls. On all 3 attached NIC's :)
-
Can you set it to full auto pls. On all 3 attached NIC's :)
Try what Supermule suggested – "auto" instead of hard-coding the duplex. 1000BaseT links expect the duplex settings to be "auto". It wants to auto-negotiate with the endpoint. If you hard-code, one side can silently fall back to half-duplex on you.
Bill
-
What do you have upstream of the WAN interface? Some sort of Comcast CPE box I assume. Do you have access to it? Is it showing any errors?
That download result from cachefly is really only valid if you prove you can download the test file far faster with a direct connection. Though 10Mbps is about what you're seeing at the clients. ;)
Steve
-
Thanks for the suggestion. I have tried every mode on the NICs with no change in the results (it is a dual port NIC). I have also been playing with the advanced network setting and trying different combinations. I am really beginning to believe this is probably related to the NIC driver itself based on some more research I have been doing. I have spent so much time on this I am about to just pull the trigger and buy one preconfigured from pfSense so I know it works and has support. I usually dont want to give up and want to find the answer but I am spending way too much time on this and its taking away from my other projects. Any downside to the preconfigured pfSense boxes you are aware of?
-
Pretty sure its a Broadcom chip. Try use dual or quad port INtel's.
