Poor network performance
-
Yeah replying to myself is poor form.
OK, well, running the same iperf test back to back several times, I get 29Mb/s, 110, 84, 137, 64, 148, 148, 147…. and pfsense shows CPU never tops 29% use
What could be causing such differences? No other traffic is presently on the network.
-
Cable modem is a Mototal SB6141, 8 channels bonded downstream. Bridging. A laptop with a GigE interface plugged directly in gets 177Mb/s down.
Pfsense box is a dual Atom D2550, 1.86GHz hyperthreaded, 4GB RAM, running 2.1.2-release.
Packages running are apinger, avahi, bandwidthd, darkstat, dhcpd, dnsmasq, ntpd, openvpn, and snort.
CPU sits at 13% idle, moderate traffic takes it to 25-75%, its been as high as 100% for a few seconds.
Bandwidth through pfsense runs between 50-70Mb/s. Prior pfsense system was a T7200-based laptop, 2Ghz dual core, 4GB RAM, and had a 100mb interface on the WAN side – it got 50-70Mb/s. The Atom has dual GigE interfaces, both running full duplex and full speed.
Not sure why I'm seeing a half (or more) drop in network speeds.
Any ideas?
For starters, I will reinforce that if you want to run intensive services such as snort and just everything really but snort is definitely causing issues with you as others have mentioned. I'm not even going to blame any of it really on pfsense simply because I could not imagine running my box at 1.86ghz. I do remember some time ago when I was going to upgrade my internet speed and my ISP actually recommended certain levels of performance for processors or it would just bog down the whole system.
A good example would be my old Dell XPS that was 850mhz single core with 768MB of memory. Everything was fine when we just had 5MB down and 1MB up. For an 850mhz system it was very snappy with those internet speeds. As soon as we upgraded to 12MB down (16MB with boost, something to do with comcast) and 2 MB up because they were just cheap like that. My DELL XPS starting acting like a Pentium II even though it was a Pentium III. Everything was slower except for when I did bandwidth tests.
Sure enough they would read true at anywhere from (remember comcast has boost) 16MB to 20MB down and still only 2MB up. So sometimes the speeds would actually increase and I think it was just something that they were doing to make you feel like you had something good. The only time you would see the speed really is when downloading. Web page surfing forget about it. It felt like dial up sometimes but I believe that it had to do with my pc at the time being too slow for the internet.
Not only that, but it will make your overall pc performance slower if it cannot keep up with the internet speed simply because your processor still has to process all of that data coming in even if you have a separate nic. One recommendation that I would give is to get yourself an AMD AM3+ motherboard. Doesn't have to be fancy. Just something around $80 and and an AMD FX 4130 3.8GHZ or Phenom 965 Black edition. I only say that because right now it is about the cheapest processors that you can get with the best performance. Keep in mind that I'm not one that's trying to save the planet through electricity. I don't buy into that hype. You can still run it efficiently because it will only go as fast as it needs to if you keep amd cool and quiet on. If you get that and say 8GB of DDR3 memory which is really cheap now.
As of now I am on Verizon with 70MB down and 50MB up. I am running into the same problem sometimes with my 965 x4 black edition running at 3.8GHZ. I 'm just running into it. It's not full blown slowness yet but I can tell that upgrading to something that is 4ghz or more would help my pc performance. In hindsight though I realize that I really don't need the internet speed that they are providing. So it's also a good idea to choose the package that you want and make sure that your not going to have to upgrade a pc over it unless that 's what you want to do. The only reason I say this is because you will never see that true speed unless you're looking at an internet speed test or downloading a file. At times I'm willing to bet that even with no pfsense services it probably feels like dial up when your just surfing.
-
@Cmellons, I take it you're running a load of packages? your machine spec is way, way higher than anything I'm running.
@tucansam I had an issue testing my connection speed some time ago that turned out to be a problem with my client machine, an older Windows XP box. When I booted the same machine from a live Linux CD I was suddenly able to max out the connection no problems. I did investigate the problem and I think it turned out to be the Windows default TCP window size but don't quote me on that.
When you're looking at the pfSense CPU usage you cannot use the dashboard bar graph if your box has multiple CPU cores. That graph shows the average use across all cores. The D2550 is dual core with hyperthreading so it appears as 4 cores. If you have one core maxed out at 100% and only 10% use on the other cores the graph will show 32.5% but in fact the pf process has hit the cpu limit on one core. To get a much better idea run 'top -SH' at the console. That will show you the idle percentage for each core.
Where were you fetching the file from directlt in pfSense? There's probably a better source nearer to you.Steve
-
Sure enough they would read true at anywhere from (remember comcast has boost) 16MB to 20MB down and still only 2MB up. So sometimes the speeds would actually increase and I think it was just something that they were doing to make you feel like you had something good. The only time you would see the speed really is when downloading. Web page surfing forget about it. It felt like dial up sometimes but I believe that it had to do with my pc at the time being too slow for the internet.
This is essentially what I am seeing. File downloads go OK, mostly. Youtube stutters a bit, and only half loads otherwise. Web surfing comes and goes. Sometimes its fast, sometimes not so much. This is on all systems on the network, wired or wireless, fast or slow. At home I am running Core2Duos, Phenom II X4s, i5s… Clients are plenty fast for what we are doing.
Not only that, but it will make your overall pc performance slower if it cannot keep up with the internet speed simply because your processor still has to process all of that data coming in even if you have a separate nic. One recommendation that I would give is to get yourself an AMD AM3+ motherboard. Doesn't have to be fancy. Just something around $80 and and an AMD FX 4130 3.8GHZ or Phenom 965 Black edition. I only say that because right now it is about the cheapest processors that you can get with the best performance. Keep in mind that I'm not one that's trying to save the planet through electricity. I don't buy into that hype. You can still run it efficiently because it will only go as fast as it needs to if you keep amd cool and quiet on. If you get that and say 8GB of DDR3 memory which is really cheap now.
My Atom draws less than 25W at the wall. Its silent (in the entertainment center, the only place I can put it), and runs very cool. I'm also running dual Intel NICs, on which I insist. I also need the mini-ITX form factor, in a chassis no larger than the cardboard box the MB came on. Power onboard the MB, 4GB RAM… No way I can justify a 95W CPU, and I'm ot aware of any mini-ITX AM3/+ boards with dual NICs. Only option would be an Intel-based dual-NIC mini-ITX, and probably an i5 at the minimum given your recommendations. That would make my firewall the fastest system in the house, which is contrary to everything I've ever read about pfsense ;D
As of now I am on Verizon with 70MB down and 50MB up. I am running into the same problem sometimes with my 965 x4 black edition running at 3.8GHZ. I 'm just running into it. It's not full blown slowness yet but I can tell that upgrading to something that is 4ghz or more would help my pc performance. In hindsight though I realize that I really don't need the internet speed that they are providing. So it's also a good idea to choose the package that you want and make sure that your not going to have to upgrade a pc over it unless that 's what you want to do. The only reason I say this is because you will never see that true speed unless you're looking at an internet speed test or downloading a file. At times I'm willing to bet that even with no pfsense services it probably feels like dial up when your just surfing.
Agreed. I am buying the highest package my ISP offers for the monthly bandwidth allowance, not the speed. Still, seeing blinding speeds at the modem, but 1/2-1/3 as fast behind pfsense, makes me wonder if I've got another problem. Frankly, I don't know what i would ever do with 177Mb/s down, but if I'm not capable of getting it, it means something is amiss.
I appreciate your recommendations, but building a firewall that is many orders of magnitude better config'd than my fastest workstation doesn't seem like the right approach. Low power, low heat, "appliance" type devices is what turned me onto pfsense in the first place.
-
@tucansam I had an issue testing my connection speed some time ago that turned out to be a problem with my client machine, an older Windows XP box. When I booted the same machine from a live Linux CD I was suddenly able to max out the connection no problems. I did investigate the problem and I think it turned out to be the Windows default TCP window size but don't quote me on that.
That's a stellar idea. I will have to give this a shot. All of the PCs in the house exhibit the same symptoms, but I'll wire up a laptop and boot off a live CD to compare.
When you're looking at the pfSense CPU usage you cannot use the dashboard bar graph if your box has multiple CPU cores. That graph shows the average use across all cores. The D2550 is dual core with hyperthreading so it appears as 4 cores. If you have one core maxed out at 100% and only 10% use on the other cores the graph will show 32.5% but in fact the pf process has hit the cpu limit on one core. To get a much better idea run 'top -SH' at the console. That will show you the idle percentage for each core.
Stellar, thank you. I had no idea. Yep, I've been using the dashboard, and occasionally top, but not with -SH I will keep an eye on things with that from now on.
Where were you fetching the file from directlt in pfSense? There's probably a better source nearer to you.
Yep, directly from pfsense. I'll dig around for servers closer to me and test again.
-
If you used a fetch command example I posted anywhere I probably pointed to a thinkbroadband test file. They're great if you're in the UK but not so much from the US. ;) Chris (cmb) once posted a similar site with test files he uses in the US but I can't find it now.
Steve
-
If you used a fetch command example I posted anywhere I probably pointed to a thinkbroadband test file. They're great if you're in the UK but not so much from the US. ;) Chris (cmb) once posted a similar site with test files he uses in the US but I can't find it now.
Steve
Ha. Yep, pretty sure it was your thread I read.
An an aside, after uninstalling snort a few days ago, I just now reinstalled it, and its running. My media downloader is showing 4.9-5.3MB/s download speeds, and 'top -SH' is showing 83-89% idle with that traffic passing. I typically run it at 200KB/s, at which point 'top -SH' shows 94-99% idle. 3.5MB free memory during the duration.
Do not believe this is a CPU issue…
Just as a point of curiosity, has anyone ever ranked the most system-resource-hungry packages from top to bottom? I know some of what I am running is probably unnecessary, and I'd like to leave enough headroom for other things. For one thing, I am trying to get rules working to restrict the bandwidth of some devices, as well as schedules for those devices. Not sure how much, if any, processing power that would take up. I'll also be revisiting squid at some point (which I have never seemed to get installed correctly despite following numerous youtube tutorials) as well as squidguard (same traffic).
-
'top -SH' is showing 83-89% idle with that traffic passing.
How is that divided between the cores? The central firewall/NAT process, pf, can curently only use one core so that's usually the limit. Snort will be able to use other cores though.
Steve
-
cachefly is the one I tend to use that Steve referenced, they have links to a 10 MB and 100 MB test file on their site.
http://cachefly.cachefly.net/10mb.test
http://cachefly.cachefly.net/100mb.testAs a CDN, they should be fast pretty much everywhere because you should end up at a server that's relatively close to you. Granted that depends on where you are, your ISP, and many other factors.
-
You should really take that XP box out back and shoot it. Or load some supported OS on it. Already a nice 0 day that isn't going to get patched on XP.
http://arstechnica.com/security/2014/04/active-0day-attack-hijacking-ie-users-threatens-a-quarter-of-browser-market/Those are just going to keep coming and coming. XP is dead, it's been time to move on for years.
-
'top -SH' is showing 83-89% idle with that traffic passing.
How is that divided between the cores? The central firewall/NAT process, pf, can curently only use one core so that's usually the limit.
SteveOnly true before pfSense 2.2.
-
@cmb:
You should really take that XP box out back and shoot it.
Indeed, and that's coming from a die hard XP fan. I have seen little point in upgrading Windows versions until now. XP did everything I needed it to without too much system bloat. 2K was better! ;) However I've now switched everything I had running XP to Xubuntu which runs great on older hardware. Also playing with GhostBSD which is nice with XFCE. Even so I still have one machine set to dual boot to XP which I had to use yesterday to open BIOS update distributed as a windows executable. >:(
Steve
-
'top -SH' is showing 83-89% idle with that traffic passing.
How is that divided between the cores? The central firewall/NAT process, pf, can curently only use one core so that's usually the limit. Snort will be able to use other cores though.
Steve
Just ran fetch on the 100mb cachefly file and got only 1944kBps, one cpu was 100% idle, other three were 90-94%, snort never went above 25%
-
@cmb:
You should really take that XP box out back and shoot it. Or load some supported OS on it. Already a nice 0 day that isn't going to get patched on XP.
http://arstechnica.com/security/2014/04/active-0day-attack-hijacking-ie-users-threatens-a-quarter-of-browser-market/Those are just going to keep coming and coming. XP is dead, it's been time to move on for years.
Working on it, although I'll save my bullets for other purposes. I need to migrate some things from that machine to another one, and I need to build the new one first. So, yeah, working on it.
-
@cmb:
You should really take that XP box out back and shoot it.
Indeed, and that's coming from a die hard XP fan. I have seen little point in upgrading Windows versions until now. XP did everything I needed it to without too much system bloat. 2K was better! ;) However I've now switched everything I had running XP to Xubuntu which runs great on older hardware. Also playing with GhostBSD which is nice with XFCE. Even so I still have one machine set to dual boot to XP which I had to use yesterday to open BIOS update distributed as a windows executable. >:(
Steve
Agreed, 2000 > XP > Win7 > DOS 1.0 > Win8.
Actually scratch that, put "punch cards" ahead of Win8.
My speeds are remaining consistent, just consistently slower than I had anticipated given my setup.
-
[Just ran fetch on the 100mb cachefly file and got only 1944kBps[/quote]
Then I think you'll have to test your connection speed to cachefly without the pfSense box because that's slower that anything else.
scratch that, put "punch cards" ahead of Win8.
Technically I don't think punch cards count as an operation system. Then again you could say the same for Win8. ;)
Steve
-
I failed to mention that it was just lying around collecting dust. It was a previous gaming pc. However, I do realize the kind of damage that I was causing to the environment and now I am running pfsense in a virtual machine. Then again I just have that itch again to build it back up and run it. I need to get a dynamat for the box though. It was only $40 so as you could guess it is way too loud. Something like that really does not draw that much power with amd cool and quiet on. Most of the time it ran at 800mhz and .75v. The tdp was embarrassingly high at 1.47v and 140w at full load so yes I had a big thermaltake maxorb cooling it.
edited in the interest of not messing up a topic. I must hold back on the caffeine intake late at night.
-
I just wanted to jump in on this as I am seeing the same type of issue.
I bought an OPNsense appliance running Intel Atom 1.6Ghz, 2GB RAM, 2GB CF, and in a production environment on a 100Mbit fibre connection it gives me 15/87. Sent it back on warranty and got a reply that it was a config / software mismatch, but when it got back I rebuilt the entire config manually and the problem persists. I use some advanced NAT (reflection, Virtual IP's, 1:1 etc) but almost no packages except dhcpd, dns and whatever is default.
I'm thinking hardware issue but I'm not sure. Seems strange since the hardware is brand new. Maybe you've got the same issue as I do?
-
Not quite sure what numbers you're giving us there. You're seeing 15Mbps down on a 100Mbps connection?
If that's the case look for a duplex mismatch or possibly some flow control issue. Check the Status: Interfaces: page for errors/collisions.
Test directly on the box to see which interface is throttling the connection. Look at 'top -SH' at the console to see if it's a CPU or interrupt problem.Steve
-
I do realize the kind of damage that I was causing to the environment
Don't underestimate what damage you are saving by not buying new hardware. Of course if you already have a VM host running then yes, no excuse! ;)
Steve
