Poor network performance
-
Cable modem is a Mototal SB6141, 8 channels bonded downstream. Bridging. A laptop with a GigE interface plugged directly in gets 177Mb/s down.
Pfsense box is a dual Atom D2550, 1.86GHz hyperthreaded, 4GB RAM, running 2.1.2-release.
Packages running are apinger, avahi, bandwidthd, darkstat, dhcpd, dnsmasq, ntpd, openvpn, and snort.
CPU sits at 13% idle, moderate traffic takes it to 25-75%, its been as high as 100% for a few seconds.
Bandwidth through pfsense runs between 50-70Mb/s. Prior pfsense system was a T7200-based laptop, 2Ghz dual core, 4GB RAM, and had a 100mb interface on the WAN side – it got 50-70Mb/s. The Atom has dual GigE interfaces, both running full duplex and full speed.
Not sure why I'm seeing a half (or more) drop in network speeds.
Any ideas?
-
If your CPU is at 100% when hitting 70mbps, you have a CPU bottleneck. See if any one process is largely responsible. Snort is a likely culprit. Try disabling it or play with the search method it uses.
-
Well disabling snort took me to 105Mbs. And now, during speed tests and 70-80MB/s traffic going, I never get above 15% CPU. So now I'm totally confused, and not running snort, which I really would like to run.
-
Snort is a resource hog but it can be tamed to some extent, mostly by choosing the correct pattern matcher. Check the various Snort threads and here if you haven't already:
https://doc.pfsense.org/index.php/Setup_Snort_Package
Just off hand that figure, 50-70Mbps, is suspiciously exactly what I expect the OpenVPN throughput to be.
There's no reason, without Snort, that you shouldn't be seeing the full 170Mbps. That CPU should be good for >500Mbps of firewall/NAT.Steve
-
Well, turning off ALL services, I can get 117Mbs down. Turning them on one by one slowly eats up the bandwidth. Turing a couple of them on results in some others (dependencies?) starting as well.
Not sure what to do here. Even with snort off, other services (bandwidth monitor, darkstat, openvpn) seem to sap bandwidth. Incidentally, when I turn openvpn off, I can no longer get speedtest.net to work (although there is no openvpn connection coming in). With it on, it works again.
I'm at a loss here. Not sure what is going on.
-
Possibly there is a persistent state that is trying to use the VPN connection to get to speedtest.net, try clearing the state table. Other sites are accessible I assume?
Since the box is capable of >500Mbps I would expect to be able to put a few services on there without reducing the throughput to <120Mbps. Are you saying as soon as you add any service the bandwidth drops?
Steve
-
tucansam, how you measure performance? Iperf or similar tool?
-
tucansam, how you measure performance? Iperf or similar tool?
stephenw10: I've reset the states, and indeed the firewall, with the same effects. I can get max bandwidth at about 70Mb/s, with no services running, and slowly creep it down to 50Mb/s once I have everything running. Some services seem to take more bw away than others.
Cutler: Well, initially I was using speedtest.net. With my laptop connected directly to my cable modem (bridge), with a FDX GigE connection, 177Mb/s is the fastest I've seen (off-peak hours, averaged over a few days). When I plug my network back in, the only easily accessible machine is an Atom-based XP system, into which I RDP and then run speedtest.net's app under a Chrome browser. In the past this has given me 100Mbs+ speeds, as it is also FDX GigE to the switch. Other devices on the network are linux or wireless, and I just found the speedtest tool to be the easiest.
I did some googling and ran the "fetch" command with a variety of hosted files of 100MB in size, ran directly from pfsense, and was never able to top 25Mb/s, which I assume means the bottleneck is on each servers' end (speedtest under the XP system continued to show 50-55Mb/s down during these tests)
I will have to google Iperf and see how to use it as I am unfamiliar.
I am tempted to restore factory defaults (or re-load from a fresh ISO image completely) and reconfigure. What would be the better option?
-
Well this is interesting:
C:\Documents and Settings\Desktop>iperf -c 192.168.0.1
–----------------------------------------------------------
Client connecting to 192.168.0.1, TCP port 5001
TCP window size: 64.0 KByte (default)[ 3] local 192.168.0.7 port 2618 connected with 192.168.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 33.1 MBytes 27.6 Mbits/secC:\Documents and Settings\Desktop>
This is with iperf running (server) on pfsense, and client on the aforementioned XP machine. All links are GigE FDX through a Netgear business class switch. Speeds are atrocious! Intel NICs on the pfsense machine; crappy Realtek on the XP box but speeds have never been this slow. "Green ethernet" is enabled on the XP system, which I have only read a little about, but the cable length is 3' so it shouldn't matter. With "Green ethernet" turned off I get 29Mbits/s
More testing needed here.
-
Yeah replying to myself is poor form.
OK, well, running the same iperf test back to back several times, I get 29Mb/s, 110, 84, 137, 64, 148, 148, 147…. and pfsense shows CPU never tops 29% use
What could be causing such differences? No other traffic is presently on the network.
-
Cable modem is a Mototal SB6141, 8 channels bonded downstream. Bridging. A laptop with a GigE interface plugged directly in gets 177Mb/s down.
Pfsense box is a dual Atom D2550, 1.86GHz hyperthreaded, 4GB RAM, running 2.1.2-release.
Packages running are apinger, avahi, bandwidthd, darkstat, dhcpd, dnsmasq, ntpd, openvpn, and snort.
CPU sits at 13% idle, moderate traffic takes it to 25-75%, its been as high as 100% for a few seconds.
Bandwidth through pfsense runs between 50-70Mb/s. Prior pfsense system was a T7200-based laptop, 2Ghz dual core, 4GB RAM, and had a 100mb interface on the WAN side – it got 50-70Mb/s. The Atom has dual GigE interfaces, both running full duplex and full speed.
Not sure why I'm seeing a half (or more) drop in network speeds.
Any ideas?
For starters, I will reinforce that if you want to run intensive services such as snort and just everything really but snort is definitely causing issues with you as others have mentioned. I'm not even going to blame any of it really on pfsense simply because I could not imagine running my box at 1.86ghz. I do remember some time ago when I was going to upgrade my internet speed and my ISP actually recommended certain levels of performance for processors or it would just bog down the whole system.
A good example would be my old Dell XPS that was 850mhz single core with 768MB of memory. Everything was fine when we just had 5MB down and 1MB up. For an 850mhz system it was very snappy with those internet speeds. As soon as we upgraded to 12MB down (16MB with boost, something to do with comcast) and 2 MB up because they were just cheap like that. My DELL XPS starting acting like a Pentium II even though it was a Pentium III. Everything was slower except for when I did bandwidth tests.
Sure enough they would read true at anywhere from (remember comcast has boost) 16MB to 20MB down and still only 2MB up. So sometimes the speeds would actually increase and I think it was just something that they were doing to make you feel like you had something good. The only time you would see the speed really is when downloading. Web page surfing forget about it. It felt like dial up sometimes but I believe that it had to do with my pc at the time being too slow for the internet.
Not only that, but it will make your overall pc performance slower if it cannot keep up with the internet speed simply because your processor still has to process all of that data coming in even if you have a separate nic. One recommendation that I would give is to get yourself an AMD AM3+ motherboard. Doesn't have to be fancy. Just something around $80 and and an AMD FX 4130 3.8GHZ or Phenom 965 Black edition. I only say that because right now it is about the cheapest processors that you can get with the best performance. Keep in mind that I'm not one that's trying to save the planet through electricity. I don't buy into that hype. You can still run it efficiently because it will only go as fast as it needs to if you keep amd cool and quiet on. If you get that and say 8GB of DDR3 memory which is really cheap now.
As of now I am on Verizon with 70MB down and 50MB up. I am running into the same problem sometimes with my 965 x4 black edition running at 3.8GHZ. I 'm just running into it. It's not full blown slowness yet but I can tell that upgrading to something that is 4ghz or more would help my pc performance. In hindsight though I realize that I really don't need the internet speed that they are providing. So it's also a good idea to choose the package that you want and make sure that your not going to have to upgrade a pc over it unless that 's what you want to do. The only reason I say this is because you will never see that true speed unless you're looking at an internet speed test or downloading a file. At times I'm willing to bet that even with no pfsense services it probably feels like dial up when your just surfing.
-
@Cmellons, I take it you're running a load of packages? your machine spec is way, way higher than anything I'm running.
@tucansam I had an issue testing my connection speed some time ago that turned out to be a problem with my client machine, an older Windows XP box. When I booted the same machine from a live Linux CD I was suddenly able to max out the connection no problems. I did investigate the problem and I think it turned out to be the Windows default TCP window size but don't quote me on that.
When you're looking at the pfSense CPU usage you cannot use the dashboard bar graph if your box has multiple CPU cores. That graph shows the average use across all cores. The D2550 is dual core with hyperthreading so it appears as 4 cores. If you have one core maxed out at 100% and only 10% use on the other cores the graph will show 32.5% but in fact the pf process has hit the cpu limit on one core. To get a much better idea run 'top -SH' at the console. That will show you the idle percentage for each core.
Where were you fetching the file from directlt in pfSense? There's probably a better source nearer to you.Steve
-
Sure enough they would read true at anywhere from (remember comcast has boost) 16MB to 20MB down and still only 2MB up. So sometimes the speeds would actually increase and I think it was just something that they were doing to make you feel like you had something good. The only time you would see the speed really is when downloading. Web page surfing forget about it. It felt like dial up sometimes but I believe that it had to do with my pc at the time being too slow for the internet.
This is essentially what I am seeing. File downloads go OK, mostly. Youtube stutters a bit, and only half loads otherwise. Web surfing comes and goes. Sometimes its fast, sometimes not so much. This is on all systems on the network, wired or wireless, fast or slow. At home I am running Core2Duos, Phenom II X4s, i5s… Clients are plenty fast for what we are doing.
Not only that, but it will make your overall pc performance slower if it cannot keep up with the internet speed simply because your processor still has to process all of that data coming in even if you have a separate nic. One recommendation that I would give is to get yourself an AMD AM3+ motherboard. Doesn't have to be fancy. Just something around $80 and and an AMD FX 4130 3.8GHZ or Phenom 965 Black edition. I only say that because right now it is about the cheapest processors that you can get with the best performance. Keep in mind that I'm not one that's trying to save the planet through electricity. I don't buy into that hype. You can still run it efficiently because it will only go as fast as it needs to if you keep amd cool and quiet on. If you get that and say 8GB of DDR3 memory which is really cheap now.
My Atom draws less than 25W at the wall. Its silent (in the entertainment center, the only place I can put it), and runs very cool. I'm also running dual Intel NICs, on which I insist. I also need the mini-ITX form factor, in a chassis no larger than the cardboard box the MB came on. Power onboard the MB, 4GB RAM… No way I can justify a 95W CPU, and I'm ot aware of any mini-ITX AM3/+ boards with dual NICs. Only option would be an Intel-based dual-NIC mini-ITX, and probably an i5 at the minimum given your recommendations. That would make my firewall the fastest system in the house, which is contrary to everything I've ever read about pfsense ;D
As of now I am on Verizon with 70MB down and 50MB up. I am running into the same problem sometimes with my 965 x4 black edition running at 3.8GHZ. I 'm just running into it. It's not full blown slowness yet but I can tell that upgrading to something that is 4ghz or more would help my pc performance. In hindsight though I realize that I really don't need the internet speed that they are providing. So it's also a good idea to choose the package that you want and make sure that your not going to have to upgrade a pc over it unless that 's what you want to do. The only reason I say this is because you will never see that true speed unless you're looking at an internet speed test or downloading a file. At times I'm willing to bet that even with no pfsense services it probably feels like dial up when your just surfing.
Agreed. I am buying the highest package my ISP offers for the monthly bandwidth allowance, not the speed. Still, seeing blinding speeds at the modem, but 1/2-1/3 as fast behind pfsense, makes me wonder if I've got another problem. Frankly, I don't know what i would ever do with 177Mb/s down, but if I'm not capable of getting it, it means something is amiss.
I appreciate your recommendations, but building a firewall that is many orders of magnitude better config'd than my fastest workstation doesn't seem like the right approach. Low power, low heat, "appliance" type devices is what turned me onto pfsense in the first place.
-
@tucansam I had an issue testing my connection speed some time ago that turned out to be a problem with my client machine, an older Windows XP box. When I booted the same machine from a live Linux CD I was suddenly able to max out the connection no problems. I did investigate the problem and I think it turned out to be the Windows default TCP window size but don't quote me on that.
That's a stellar idea. I will have to give this a shot. All of the PCs in the house exhibit the same symptoms, but I'll wire up a laptop and boot off a live CD to compare.
When you're looking at the pfSense CPU usage you cannot use the dashboard bar graph if your box has multiple CPU cores. That graph shows the average use across all cores. The D2550 is dual core with hyperthreading so it appears as 4 cores. If you have one core maxed out at 100% and only 10% use on the other cores the graph will show 32.5% but in fact the pf process has hit the cpu limit on one core. To get a much better idea run 'top -SH' at the console. That will show you the idle percentage for each core.
Stellar, thank you. I had no idea. Yep, I've been using the dashboard, and occasionally top, but not with -SH I will keep an eye on things with that from now on.
Where were you fetching the file from directlt in pfSense? There's probably a better source nearer to you.
Yep, directly from pfsense. I'll dig around for servers closer to me and test again.
-
If you used a fetch command example I posted anywhere I probably pointed to a thinkbroadband test file. They're great if you're in the UK but not so much from the US. ;) Chris (cmb) once posted a similar site with test files he uses in the US but I can't find it now.
Steve
-
If you used a fetch command example I posted anywhere I probably pointed to a thinkbroadband test file. They're great if you're in the UK but not so much from the US. ;) Chris (cmb) once posted a similar site with test files he uses in the US but I can't find it now.
Steve
Ha. Yep, pretty sure it was your thread I read.
An an aside, after uninstalling snort a few days ago, I just now reinstalled it, and its running. My media downloader is showing 4.9-5.3MB/s download speeds, and 'top -SH' is showing 83-89% idle with that traffic passing. I typically run it at 200KB/s, at which point 'top -SH' shows 94-99% idle. 3.5MB free memory during the duration.
Do not believe this is a CPU issue…
Just as a point of curiosity, has anyone ever ranked the most system-resource-hungry packages from top to bottom? I know some of what I am running is probably unnecessary, and I'd like to leave enough headroom for other things. For one thing, I am trying to get rules working to restrict the bandwidth of some devices, as well as schedules for those devices. Not sure how much, if any, processing power that would take up. I'll also be revisiting squid at some point (which I have never seemed to get installed correctly despite following numerous youtube tutorials) as well as squidguard (same traffic).
-
'top -SH' is showing 83-89% idle with that traffic passing.
How is that divided between the cores? The central firewall/NAT process, pf, can curently only use one core so that's usually the limit. Snort will be able to use other cores though.
Steve
-
cachefly is the one I tend to use that Steve referenced, they have links to a 10 MB and 100 MB test file on their site.
http://cachefly.cachefly.net/10mb.test
http://cachefly.cachefly.net/100mb.testAs a CDN, they should be fast pretty much everywhere because you should end up at a server that's relatively close to you. Granted that depends on where you are, your ISP, and many other factors.
-
You should really take that XP box out back and shoot it. Or load some supported OS on it. Already a nice 0 day that isn't going to get patched on XP.
http://arstechnica.com/security/2014/04/active-0day-attack-hijacking-ie-users-threatens-a-quarter-of-browser-market/Those are just going to keep coming and coming. XP is dead, it's been time to move on for years.
-
'top -SH' is showing 83-89% idle with that traffic passing.
How is that divided between the cores? The central firewall/NAT process, pf, can curently only use one core so that's usually the limit.
SteveOnly true before pfSense 2.2.