SG-1000 100% CPU Usage

[deadmalc]

I've got it setup with openvpn (one outbound and an inbound server) and unbound.
It works fine - infact it works amazingly well, lower latency than my previous router.
I've not tried pfBlockerNG, I took a look but I've not used it before so I've removed it for the moment.
The outbound vpn connection works really well and saves me hastle.
The inbound connection is for when my wife is at work and needs files of the NAS.
Not had any complaints, in fact she commented that it was great and "just worked".

It is still in beta, so I'm expecting a few issues.
If you update every day, given it's in beta - I would expect some pain here and there.

I've not seen any load issues, or had any reboots. (touch wood)
But I tried a few other packages:
squid - way too sluggish, transformed from fibre to 9600 baud modem!
acme - not worth the effort, good idea though. [ CORRECTION - not worth the effort for a "home firewall" ]
pfBlockerNG - not got the time at the moment to understand and configure it properly.

[stephenw10]

Hmm, Ok.
Not seeing high CPU usage there. Not sure what that panic might be. Did you get a crash report at reboot?

Steve

[chrcoluk]

you can reduce the interrupt usage by fiddling in loader.conf.

you can reduce the timer to 100hz refresh rate instead of 1000hz, usually I only tune this on virtual machines but I would consider tuning it also on very low end hardware.

I will paste a link to the FreeBSD page as it has other tunables as well.

https://wiki.freebsd.org/TuningPowerConsumption

Also it might be a good idea to add a swap to that system as 512meg maximum addressable memory is quite low when you doing things like using pfblockerNG and especially if using with unbound.

[W4RH34D]

I can't get in to the GUI or the console - getting a resource busy.  I think i fried it.  I handed it to a friend who is going to take a crack at it.

[cplmayo]

@stephenw10:

Hmm, Ok.
Not seeing high CPU usage there. Not sure what that panic might be. Did you get a crash report at reboot?

Steve

I noticed that when before I sent it. The crash is what really worries me. My thought is the CPU usage and performance will come with finalized software.

[stephenw10]

I certainly hope so.

If you can catch it running at high CPU though and grab an output from top that will help us track down whatever that is.

It doesn't seem to be that widespread, not an issue effecting all users.

Steve

[pfbolt]

I'm seeing very high load on mine, and very low throughput. The CPU usage in top says it's mostly idle, but the load average hovers around 1.
Also, I'm getting very low throughput. Simply running curl on a large file and piping to /dev/null, I get <10mbit both on the SG-1000 itself and when I connect a computer to the LAN port.

I took the thing with me to work today, to try it in another network, but the results are the same.

Really disappointed :(

If there's any output I can give you that would help, please let me know. I really want to use this, but if it can't get close to my 100mbit connection I can't.
Currently it has the 20170207 image freshly flashed, and I've only run through the initial web config. No other changes.

[stephenw10]

Same thing I suggested above should show it. Run at the console:

top -aSH

Let it run for a few seconds to get all the info then quit and copy paste it here.

Steve

[pfbolt]

It seems it depends very much on what I do. Testing with speedtest.net, I get the full 100mbit down, and almost the same up.
However, downloading a single file from a site that normally maxes out my 100mbit, both on wireless and when stealing the WAN cable from the SG-1000, I get around 10mbit. Back to a smooth 100mbit as soon as I steal the WAN cable and plug it into my client machine.

Attached is the output of top, as requested, after having ran top and the download for about a minute.

top.txt

[pfbolt]

Any ideas? Anything else I can provide?

[deadmalc]

The load on mine always hovers around 1, but it can handle my internet speed OK, it's the max speed I'm pegged at.

So I don't think the load is the problem.
If you want load less than 1, disable ssh, only use the console and don't use the web-interface.
Obviously that's not never use the web interface!
The 20% interrupt seems very high

last pid:  7799;  load averages:  0.95,  1.26,  1.07    up 0+00:33:30  21:55:09
117 processes: 2 running, 93 sleeping, 22 waiting
CPU: 16.7% user,  0.0% nice, 44.9% system,  5.1% interrupt, 33.3% idle
Mem: 59M Active, 227M Inact, 115M Wired, 56M Buf, 81M Free
Swap:

The only things I can think of immediately are:
Is the traffic already saturated on the WAN port (install iftop and use from command line)
Is there a duplex mismatch? Can you put a(nother) switch between the WAN cable and the SG-1000
Are you logging any packets on the firewall? (that may help, try not logging anything)

[pfbolt]

It's absolutely stock, so I doubt anything is being logged. However, speedtest also runs well on my SG-1000. I get the full 100mbit. However, when downloading a single file, the story is entirely different. Could you by any chance give downloading http://ipv4.download.thinkbroadband.com/1GB.zip a try? (it's a test file from http://www.thinkbroadband.com/download.html)

[deadmalc]

If speedtest is working fine, then there is something weird going on.
It doesn't make much sense, which means we are missing something (probably obvious!)

I get(ish) around 4-5MB/sec (30-40Mbit) downloading the file.
That seems around right for me, downloading from standard endpoints.

If it's stock, then I'd definitely look at the settings.
I'm still a newbie when it comes to PFSense (even though I've been using it for a long time) but
If I was having these problems I would:

• swear a lot ;-)

• "Reset to factory defaults"

• disable ssh

• don't use the web-ui whilst you are doing any performance testing. (web browsers are the work of the devil anyway)

• On the WAN, I would recommend a rule saying drop everything, don't log.

Something like this:

The SG-1000, from my understanding of gossip, blogs and forums is that it's currently rated to be around 100Mbit.
I could be wrong there, that is purely my guess and bad memory at work!
I know they are and have been working on improving this.
virus scanning, proxy etc. IMHO is the last thing that should be on it at the moment if you are using it, or want to use it as your main firewall.

So anything you can do to reduce the overhead is a good thing.
It maybe painful to start again from factory, but if it's a niggling issue that can't be solved the only way is one step at a painful time.
Make one change and re-test every time.

Another question, are you plugging directly into the SG-1000 or via a switch - for a laugh if you can try both.
Hardware is also the work of the devil - in fact anything to do with infrastructure is there to bend your mind!

[pfbolt]

I am plugging directly into the SG-1000, both with my laptop at home and the machine at work. That's also the two different networks I've tried it in with the same results. I've reflashed the thing multiple times with no improvements in throughput. I also see the same performance when downloading the file directly to the SG-1000 through the console with curl, writing to /dev/null, so I doubt the connection between my machines and the SG-1000 has anything to do with it, at least.

[deadmalc]

Using it on two different networks with two different machines doesn't allow any comparisons as there are too many variables involved.
There are many, many things that can make a difference.
I would concentrate on debugging it where you are going to use it.
The only conclusion I could draw at the moment, given I'm not seeing any problems is that something is interfering with the downloads between the SG-1000 and the WAN connection.
I'm not saying that is right, but it's a starting point.

When you are using this at home, how is your WAN connection configured.
i.e. are you connected directly to the internet with it, or is it going through a router (or multiple routers) first.
For example, my setup I have my old wifi router on my LAN (in case sg-1000 breaks I can reconfigure that as the WAN connection).
My SG-1000 is connected via the WAN port using a shielded network cable to my ISP's fibre router.
My SG-1000 does pppoe on the wan port to connect me to the internet and the LAN port connects to my old wifi router (which is acting as a switch and NAS box, until I can afford a decent switch, NAS and wifi access point[cisco/synology?/Ubiquiti])

[cplmayo]

So it took me a while to get all this together; system wouldn't even boot until I had a fan blowing directly on it.

All these Screen shots were done while I was logged in and looking at the dashboard the last two that show lower usage came after I logged off. I also included a screen shot of the installed packages.

Side not if it would help I could ship it back to netgate or bring it down for you to look at. I am just an hour away to your north.

1st  Picture: is an attempt to boot up the SG-1000
2nd Picture: still trying to boot
3rd Picture: Yup still won't boot
4th Picture: No Joy
5th Picture: ok finally booted and updated from console; no traffic, one user viewing the Dashboard
6th Picture: no traffic, one user viewing the Dashboard
7th Picture: no traffic, one user viewing the Dashboard
8th Picture: Installed Packages
9th Picture: no traffic, one user viewing the Dashboard
10th Picture:  no traffic, one users logged in

.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)
.png)
.png_thumb)

[deadmalc]

Wow! If you saying "system wouldn't even boot until I had a fan blowing directly on it",
I'd speak to netgate and look at a replacement.
My initial guess would have been there's a dodgy package causing that, but you're seeing it during initial boot!
My next guess would be a bad flash of the image.
Assuming you have reflashed the image onto it, the next guess (only one left) is hardware fault.
But bang a call into netgate and see what they come back with.

[pfbolt]

@deadmalc:

Using it on two different networks with two different machines doesn't allow any comparisons as there are too many variables involved.
There are many, many things that can make a difference.

That's the whole point. I use it on two different networks with two different machines, and the download speed is equally bad on both. Thus I am thinking the environment has nothing to do with it.

[doktornotor]

@cplmayo:

So it took me a while to get all this together; system wouldn't even boot until I had a fan blowing directly on it.

Well, that thing is clearly faulty and needs to be RMA-ed.

[deadmalc]

@pfbolt:

That's the whole point. I use it on two different networks with two different machines, and the download speed is equally bad on both. Thus I am thinking the environment has nothing to do with it.

Possibly, but you are using it in two totally different environments. All that really proves is that whatever is the same in both environments isn't causing the issue.
It's not a case of proving whether it's specifically environmental, that won't get you far.
What you want to do is find out what is causing the issue on the SG-1000, that maybe some environmental factor that is specific to adding the SG-1000 to it or a bug in PFSense.
There is an issue that causes lower than gigabit downloads when using pppoe:
https://redmine.pfsense.org/issues/4821
But if you aren't using pppoe…then that may not be the problem.
So at home are you going through a router? or using the SG-1000 as a router?
It's a case of stepping through every part of your infrastructure, slowly and logically.