New SG-4860 - Only getting ~400Mb? 100% system CPU usage
-
I just received my SG-4860 today and have begun playing around with it. I'm running the stock software load out of the box (2.2.1-RELEASE) and at home I have a 1Gbps FTTH connection from Centurylink and I'm terminating the PPPoE connection directly on the pfsense box.
The connection is stable, but I'm only able to pull ~400Mb/s down through the SG-4860 and weirdly, I can get about 500Mb/s up. I've run pfsense for a long time, and the SG-4860 is an upgrade from an Atom D525 based system that I used for five years or so. On the D525 system I get about 250Mb/s, so I was hoping that the SG-4860 would be a significantly better performer. If I use the centurylink provided hardware, I get the near gigabit speeds that I expect.
I've done some research on tunables and tried several things, but none seem to make much of a difference. The system also fails to check for updates with the URL that it is trying to access returning a 404 (https://firmware.netgate.com/auto-update/ADI/amd64). I can't seem to locate the special USB image for restoring the SG-4860 that is supposed to be available either, which is a little disappointing.
I've had some experience tuning boxes for high network throughput in the past, so I took a look at CPU usage to see if interrupts were causing it, and they appear to be spread across cores, but when I'm doing download tests I always see one core that gets pegged at 100% system CPU usage:
151 processes: 7 running, 101 sleeping, 43 waiting CPU 0: 0.0% user, 0.0% nice, 0.8% system, 25.1% interrupt, 74.1% idle CPU 1: 0.0% user, 0.0% nice, 0.4% system, 29.0% interrupt, 70.6% idle CPU 2: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 3: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle Mem: 39M Active, 91M Inact, 171M Wired, 51M Buf, 7590M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K CPU2 2 55:23 86.38% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 0 55:31 81.59% idle{idle: cpu0} 11 root 155 ki31 0K 64K RUN 1 55:52 80.57% idle{idle: cpu1} 0 root -92 0 0K 480K CPU3 3 1:35 60.79% kernel{igb2 que} 11 root 155 ki31 0K 64K RUN 3 55:45 44.38% idle{idle: cpu3} 12 root -92 - 0K 704K WAIT 0 0:48 16.70% intr{irq256: igb0:que} 12 root -92 - 0K 704K CPU1 1 0:47 16.55% intr{irq257: igb0:que} 0 root -92 0 0K 480K - 2 1:12 5.57% kernel{igb0 que} 12 root -92 - 0K 704K WAIT 0 0:56 5.47% intr{irq262: igb2:que} 0 root -92 0 0K 480K - 2 0:57 4.20% kernel{igb0 que} 12 root -92 - 0K 704K WAIT 1 0:03 1.17% intr{irq263: igb2:que} 13 root -16 - 0K 64K sleep 0 0:00 0.10% ng_queue{ng_queue3} 17115 root 20 0 21996K 3208K CPU1 1 0:00 0.10% top 0 root -16 0 0K 480K swapin 0 0:38 0.00% kernel{swapper} 13 root -16 - 0K 64K sleep 3 0:23 0.00% ng_queue{ng_queue1}
However, when I'm testing upload throughput I see that the system usage is spread more evenly across cores:
last pid: 48743; load averages: 1.61, 0.72, 0.32 up 0+00:59:33 23:52:46 151 processes: 10 running, 99 sleeping, 42 waiting CPU 0: 0.0% user, 0.0% nice, 2.3% system, 69.9% interrupt, 27.7% idle CPU 1: 0.0% user, 0.0% nice, 21.1% system, 27.7% interrupt, 51.2% idle CPU 2: 0.0% user, 0.0% nice, 51.0% system, 0.0% interrupt, 49.0% idle CPU 3: 0.0% user, 0.0% nice, 41.0% system, 0.4% interrupt, 58.6% idle Mem: 39M Active, 91M Inact, 171M Wired, 52M Buf, 7590M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K RUN 1 57:21 76.76% idle{idle: cpu1} 11 root 155 ki31 0K 64K RUN 2 56:56 62.16% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 0 56:51 49.17% idle{idle: cpu0} 11 root 155 ki31 0K 64K RUN 3 56:52 46.68% idle{idle: cpu3} 0 root -92 0 0K 480K CPU2 2 1:12 44.29% kernel{igb0 que} 0 root -92 0 0K 480K CPU1 1 1:29 38.28% kernel{igb0 que} 12 root -92 - 0K 704K WAIT 0 1:10 30.76% intr{irq262: igb2:que} 0 root -92 0 0K 480K CPU3 3 2:06 21.19% kernel{igb2 que} 12 root -92 - 0K 704K WAIT 0 0:58 15.58% intr{irq256: igb0:que} 12 root -92 - 0K 704K WAIT 1 0:58 14.16% intr{irq257: igb0:que} 12 root -92 - 0K 704K RUN 1 0:03 0.20% intr{irq263: igb2:que} 0 root -16 0 0K 480K swapin 3 0:38 0.00% kernel{swapper} 13 root -16 - 0K 64K sleep 0 0:23 0.00% ng_queue{ng_queue1} 56230 root 20 0 21160K 4536K select 1 0:03 0.00% miniupnpd 12 root -60 - 0K 704K WAIT 0 0:03 0.00% intr{swi4: clock}
Has anyone experienced similar throughput issues on the SG-4860? Is this a fixable software issue, or is this all I can expect from this box?
Cheers,
Aaron -
i think you should open a support ticket
-
Hello Aaron,
If I use the centurylink provided hardware, I get the near gigabit speeds that I expect.
Here is another story centurylink FTTH based and there was talked about something like a router
called FR1000Z that owns also a Fiber modem and at the WAN port a VLAN Tag was setting up!
Here in Germany this is usual VLAN / Tagged, perhaps this would be triggering you to the right
point perhaps? Fiber Optic Modem Recommendation please read the entire full thread down to
the bottom where you will be finding 2 more links, I really think this is the real problem you have
and not SG-4860 or pfSense based! In some really rare cases a device such as a fiber modem is
needed, but often from the ONT a RJ45 cable is drawn to the apartment and you are able to stitch
in any kind of router you will, but not ever. ;)Perhaps the ISP provided modem/router is also able to run in the so called "bridge mode"
and then acting only likes a pure modem, that would be better for the pfSense box, no
VLAN (7) Tag must be taken at the WAN port but then more speed I would expecting
from this device pair. -
An update which will hopefully be helpful to others just getting their shiny new SG-4860s/SG-2440s (and maybe the SG-8860 when it ships).
I slept on it and remembered noticing that the CPU frequency was being displayed as @ 1200Mhz on the pfsense dashboard while playing around last night. A quick check verified that was the case, and that it wasn't adaptively throttling up and down as load increased (you can run 'sysctl dev.cpu0.freq' to view such things).
So it turns out that the out of the box config doesn't enable powerd (which manages the p-states of the CPU). Once I enabled this and the CPU frequency was able to jump up to 2400Mhz, I started seeing much better performance and am now getting >700Mb/s through the box. Victory!
I've got powerd set to hiadaptive so that the system can still clock itself down when the load isn't high, but I'm curious if the thermal budget of this guy would allow it to be set to maximum. But for now, hiadaptive seems like a good compromise and will probably result in less power consumption overall than being locked at 1200Mhz out of the box.
Cheers,
Doki -
Thanks for that. The usual behaviour is that without powerd enabled it will remain at the maximum frequency. Clearly worth investigating.
Steve
-
I did some quick benchmarking (using a Prime95 torture test) with/without powerd on my 2440 and sure enough…
With powerd enabled:
dev.cpu.0.freq: 525 dev.cpu.0.freq: 350 dev.cpu.0.freq: 350 dev.cpu.0.freq: 175 dev.cpu.0.freq: 700 dev.cpu.0.freq: 525 dev.cpu.0.freq: 350 dev.cpu.0.freq: 350 dev.cpu.0.freq: 350 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100 dev.cpu.0.freq: 2100
Without:
dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225 dev.cpu.0.freq: 1225
As Doki has noted, the solution here is to enable powerd, but it might be better if pfSense/FreeBSD clocked the CPU (a C2358 in this case) at the max speed of 2.1 GHz or the stock speed of 1.7 GHz out of the box.
-
Thanks for the report. I enabled powerd by default on the configs we ship with 2.2.2 and newer on the SG platforms.
The auto-update URL isn't populated yet as there isn't anything older or newer out there for that platform. It will be with 2.2.2 (coming in days) upon release.
-
@cmb:
The auto-update URL isn't populated yet as there isn't anything older or newer out there for that platform. It will be with 2.2.2 (coming in days) upon release.
Will it continue to be necessary and/or recommended to use a custom URL?
If so, will there be a corresponding URL for beta testing?
Thanks
-
Will it continue to be necessary and/or recommended to use a custom URL?
We point each platform to a custom URL so you get a customized update for that platform. Leave it to its default. There are certain customizations you'll lose otherwise, like no longer having the correct default config if you reset to factory defaults.
If so, will there be a corresponding URL for beta testing?
There aren't available pre-release updates for those available at this time. You can manual or auto update to stock snapshot builds, then upgrade later via auto-update to the platform specific stable release, if it's a dev/testing/hacking box.