Topton N100 Reporting 402 MHz
-
@TheNarc
Just one question:
In the graphs, where do you get cpu_0..3 from?
I only get tz0, and that always stays at 27.9°C.
(Btw, on Topton N5105.)Found the reference to the posting above.
OK, a simple reset of monitor data did the trick. -
Hello everyone,
Unfortunately, I'm afraid I have very similar symptoms.
Initially I was happy to see this thread, but as time goes by and I start to doubt whether there is a solution.I have one of these Topton Intel N100 devices, 4 x 2.5 GBs Intel 226 NIC
My bios is BK-1264NP Ver: 1.5, 09/28/2023 17:23:35
Unfortunately, I don't see the "Performance temperature" settings in the BIOS :-(
However, I only noticed the possibility of changing "CPU Flex Ratio Settings"
The default value of "Disabled" can only be changed to "7". Then pfsense shows me the max speed is 691 Mhz
I have experimented with "7" and "8" (max speed is 800 Mhz ) but in all cases I still see the issue.
The issue is that during the idle mode, the pfsense is reporting quite high values of "Current CPU speed".
However, if I startiperf3test, than the speed decreases from 2 GHz to mentioned 402 MhzI can observe this also with iperf3 transfer speed which decreases from 2.5 Gbps to ~ 1 Gbps
This is weird to me. Summarizing the issue is about:
- when there is no traffic, the current CPU speed is 2GHz
- when I start iperf3 , which gives my pfsense lot of work to do :-), then CPU speed decreases to 402 MHz
-
@roxy Have you run the openssl benchmark referenced in this thread? That may help to determine whether the N100 is being throttled. Do you see CPU usage hitting 100% when your transfer speed drops?
I only got access to the power level settings in the BIOS that seemed to make the difference for me by loading a modded BIOS (also referenced in this thread). My whole LAN is only 1Gbps though so I can't run your same 2.5Gbps test as a point of comparison.
-
Yes, and the benchmark result depends how often I run it.
If the pfsense is in the idle mode, then I got very good results :-)You have chosen to measure elapsed time instead of user CPU time. Doing AES-256-CBC for 3s on 16 size blocks: 106710142 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 64 size blocks: 35718014 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 256 size blocks: 9240105 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 1024 size blocks: 2329749 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 8192 size blocks: 293649 AES-256-CBC's in 3.02s Doing AES-256-CBC for 3s on 16384 size blocks: 146186 AES-256-CBC's in 3.00s version: 3.0.12 built on: reproducible build, date unspecified options: bn(64,64) compiler: clang CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x98c007bc239ca7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes AES-256-CBC 569120.76k 761984.30k 788488.96k 795220.99k 795641.59k 798370.47kBut, when I run the benchmakr several times, or if I start
iperf3, so the CPU is getting slower (~503 MHz this time) and slower, then I see the following resultsRESULTS when CPU speed is 503 MHz You have chosen to measure elapsed time instead of user CPU time. Doing AES-256-CBC for 3s on 16 size blocks: 20752586 AES-256-CBC's in 3.61s Doing AES-256-CBC for 3s on 64 size blocks: 11339018 AES-256-CBC's in 3.81s Doing AES-256-CBC for 3s on 256 size blocks: 2117270 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 1024 size blocks: 474071 AES-256-CBC's in 3.20s Doing AES-256-CBC for 3s on 8192 size blocks: 62937 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 16384 size blocks: 22461 AES-256-CBC's in 3.00s version: 3.0.12 built on: reproducible build, date unspecified options: bn(64,64) compiler: clang CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x98c007bc239ca7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes AES-256-CBC 91994.15k 190346.79k 180673.71k 151925.27k 171859.97k 122667.01k -
That seems like you might be hitting thermal throttling. Check the per core temperatures.
-
I see the following output of
thermal sensorshw.acpi.thermal.tz0.temperature: 27.9C dev.cpu.3.temperature: 44.0C dev.cpu.2.temperature: 43.0C dev.cpu.1.temperature: 44.0C dev.cpu.0.temperature: 45.0CHowever, after few minutes of benchmark, the CPU speed increased to 600 - 800 MHz, and the transfer speed is about 1.9 Gbps
-
Hmm, seems fine if that's during the test.
-
S stephenw10 referenced this topic on
-
It's quite fine, but not bad ;-)
I really would like to have 2.5 GBps :-( all the time ,
I see 2.35 at the beginning and 1.0 after 15 seconds and 1.8 back after 30 secondsThis could be the result of bad cooling (but temperature is about 45 degrees).
I tried to simplify pfsense firewall rules (single rule with Pass all traffic) but didnt get 2.5I am wondering if NetGate hw really gets 2.5 GBps. I was considering to buy NetGate 4200 to check, but I habe no spare $500
Connecting to host 192.168.18.100, port 5201 [ 5] local 10.0.2.21 port 56216 connected to 192.168.18.100 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 283 MBytes 2.36 Gbits/sec [ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.00-3.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 3.01-4.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 5.01-6.00 sec 275 MBytes 2.31 Gbits/sec [ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 7.00-8.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 8.01-9.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 9.01-10.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 10.01-11.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 11.00-12.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 12.01-13.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 13.00-14.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 14.00-15.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 15.01-16.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 16.00-17.00 sec 280 MBytes 2.35 Gbits/sec [ 5] 17.00-18.01 sec 281 MBytes 2.35 Gbits/sec [ 5] 18.01-19.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 19.01-20.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 20.01-21.01 sec 196 MBytes 1.65 Gbits/sec [ 5] 21.01-22.00 sec 226 MBytes 1.91 Gbits/sec [ 5] 22.00-23.01 sec 204 MBytes 1.71 Gbits/sec [ 5] 23.01-24.00 sec 190 MBytes 1.60 Gbits/sec [ 5] 24.00-25.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 25.00-26.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 26.00-27.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 27.00-28.01 sec 157 MBytes 1.32 Gbits/sec [ 5] 28.01-29.01 sec 155 MBytes 1.30 Gbits/sec [ 5] 29.01-30.01 sec 153 MBytes 1.28 Gbits/sec [ 5] 30.01-31.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 31.00-32.00 sec 156 MBytes 1.31 Gbits/sec [ 5] 32.00-33.00 sec 158 MBytes 1.32 Gbits/sec [ 5] 33.00-34.00 sec 157 MBytes 1.32 Gbits/sec [ 5] 34.00-35.00 sec 113 MBytes 947 Mbits/sec [ 5] 35.00-36.01 sec 103 MBytes 860 Mbits/sec [ 5] 36.01-37.01 sec 187 MBytes 1.57 Gbits/sec [ 5] 37.01-38.01 sec 193 MBytes 1.62 Gbits/sec [ 5] 38.01-39.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 39.01-40.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 40.01-41.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 41.01-42.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 42.01-43.01 sec 212 MBytes 1.78 Gbits/sec [ 5] 43.01-44.01 sec 214 MBytes 1.80 Gbits/sec [ 5] 44.01-45.00 sec 213 MBytes 1.80 Gbits/sec [ 5] 45.00-46.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 46.00-47.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 47.00-48.01 sec 214 MBytes 1.79 Gbits/sec [ 5] 48.01-49.00 sec 212 MBytes 1.79 Gbits/sec [ 5] 49.00-50.01 sec 199 MBytes 1.66 Gbits/sec [ 5] 50.01-51.01 sec 215 MBytes 1.80 Gbits/sec [ 5] 51.01-52.01 sec 217 MBytes 1.82 Gbits/sec [ 5] 52.01-53.01 sec 213 MBytes 1.79 Gbits/sec [ 5] 53.01-54.00 sec 213 MBytes 1.79 Gbits/sec [ 5] 54.00-55.00 sec 214 MBytes 1.79 Gbits/sec [ 5] 55.00-56.00 sec 218 MBytes 1.83 Gbits/sec -
You see retransmissions at the other end when that's shown?
Do you see dropped packets in the interface stats?
-
I have iperf sunning in
SERVICESand the client inDMZ
I am not sure how to check number of dropped packages. I think the rules are "Pass" - so I should not see any dropped packets, right?
-
One end of the iperf test usually reports retransmission counts. In a nice result there should be no retransmissions.
Interface stats can be seen in Status > Interfaces or from
netstat -i.One other user seeing this recently had disabled flow control on the NIC and required it on that particular connection. Some devices just don't play nicely with/without it so it's worth testing.
-
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll igc2 1500 <Link#3> 00:d0:b4:02:1c:b8 645357575 0 0 21639068 0 0 igc2 - fe80::%igc2/64 fe80::2d0:b4ff:fe02:1cb8%igc2 0 - - 1 - - igc2 - 10.0.2.0/24 10.0.2.1 24 - - 28181 - - igc3 1500 <Link#4> 00:d0:b4:02:1c:b9 21751498 0 0 646024233 0 0 igc3 - fe80::%igc3/64 fe80::2d0:b4ff:fe02:1cb9%igc3 0 - - 0 - - igc3 - 192.168.18.0/24 192.168.18.1 3130 - - 3115 - -
-
After 10 minutes the speed test converges to 2 Gbps ;-)
and the power consumption increased by ~ 5W
-
Hmm, well that seems suspicious!
No retries/retransmissions shown at either end? You might have to run it verbose perhaps. I expect to see something like:
steve@steve-NUC9i9QNX:~$ iperf3 -c 4200.stevew.lan Connecting to host 4200.stevew.lan, port 5201 [ 5] local 172.21.16.8 port 43826 connected to 172.21.16.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 0 397 KBytes [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 417 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec 0 417 KBytes [ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 417 KBytes [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 417 KBytes [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 417 KBytes [ 5] 8.00-9.00 sec 113 MBytes 945 Mbits/sec 0 417 KBytes [ 5] 9.00-10.00 sec 112 MBytes 936 Mbits/sec 0 417 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done.Check for ASPM in the BIOS if you haven't already. That can cause a lot of problems.
-
I will chack the ASPM tomorrow, but I am afraid I have no option in my version.
Here is the result of
iperfrunning for 10 minutes❯ iperf3 -c 192.168.18.200 -t 600 ─╯ Connecting to host 192.168.18.200, port 5201 [ 5] local 10.0.2.21 port 56698 connected to 192.168.18.200 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 284 MBytes 2.37 Gbits/sec [ 5] 1.01-2.01 sec 280 MBytes 2.35 Gbits/sec [ 5] 2.01-3.00 sec 279 MBytes 2.35 Gbits/sec [ 5] 3.00-4.01 sec 282 MBytes 2.35 Gbits/sec [ 5] 4.01-5.01 sec 279 MBytes 2.34 Gbits/sec [ 5] 5.01-6.00 sec 279 MBytes 2.35 Gbits/sec . . . [ 5] 594.00-595.01 sec 182 MBytes 1.52 Gbits/sec [ 5] 595.01-596.01 sec 178 MBytes 1.50 Gbits/sec [ 5] 596.01-597.00 sec 174 MBytes 1.47 Gbits/sec [ 5] 597.00-598.00 sec 179 MBytes 1.50 Gbits/sec [ 5] 598.00-599.01 sec 170 MBytes 1.42 Gbits/sec [ 5] 599.01-600.00 sec 168 MBytes 1.42 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-600.00 sec 108 GBytes 1.54 Gbits/sec sender [ 5] 0.00-600.01 sec 108 GBytes 1.54 Gbits/sec receiver -
Try running with
-Vfor more verbose output. It would be interesting to see if it's reducing he window size because or retries. That is common. -
@stephenw10
iperf -V#server side - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver rcv_tcp_congestion cubic iperf 3.9 Linux iperf 6.5.13-3-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-3 (2024-03-20T10:45Z) x86_64 # client side [ 5] 57.00-58.00 sec 222 MBytes 1.86 Gbits/sec [ 5] 58.00-59.01 sec 237 MBytes 1.98 Gbits/sec [ 5] 59.01-60.00 sec 223 MBytes 1.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec sender [ 5] 0.00-60.00 sec 12.9 GBytes 1.84 Gbits/sec receiver CPU Utilization: local/sender 8.1% (0.1%u/8.0%s), remote/receiver 7.6% (0.3%u/7.4%s) rcv_tcp_congestion cubic -
Odd. Your output seems to have a space where retr and cwnd would be shown, but isn't. Same iperf version.
-
@roxy Apologies if I've somewhat lost the thread here but do you have a control, so to speak? That is to say, have you run iperf to the pfSense box from more than one host on your LAN and seen the same results? And, ideally, have you also run iperf tests directly between two hosts on your LAN and seen sustained 2.5Gbps throughput? Just trying to isolate variables. I wish I could help because it sounds like we have nearly (maybe exactly) identical pfSense hardware, except my whole LAN is still 1Gbps. I noticed from BIOS screen shots I took that I actually have ASPM enabled; haven't noticed any ill effects, but may disable it when I get a chance.
Also have you tried a UDP iperf test? I think
iperf3 -c <pfsense_ip> -u -b 2.5Gshould work. I'm not sure what we'll necessarily learn if that goes worse/same/better but it's another data point to gather. -
@roxy I think we have the same hardware and BIOS. I am able to sustain about ~2.3 Gbits/sec constantly using iperf now that I have new 2.5Gbe switches. Based on your openssl benchmarks also getting slower over time while you say temperatures are fine I'd guess you are hitting power limits just like @TheNarc seemed to be doing. I'm not sure why my box never experienced these issues with the same BIOS version but I guess the default settings which are normally hidden might be different. You might have to flash the same custom BIOS @TheNarc did and play with the PL1 settings it can unlock.
An explanation could be that the manufacturer of these boxes have realized they are not really built to passively cool a sustained heatload from these CPUs and have therefore added low sustained power limits (PL1) to remove crashes (which is an issue I experienced instead). So even though my box was performing much better out-of-the-box it would instead crash from sustained high temperatures and the only solution I found was to add an external fan blowing on the case. The CPU is fine with much higher temperatures but the other components in the very small and very badly ventilated box will cook the RAM and NVMe drives inside making them unstable because of the high ambient temperatures the CPU causes.