Topton N100 Reporting 402 MHz
-
Mmm, in my experience powerd will show 'wanted' values between the actual available steps too. So it showing wanting more than can be achieved is not entirely surprising.
-
@stephenw10 Well I loaded another modded BIOS from here that exposes power & performance options so now I can enable or disable SpeedStep and SpeedShift, C-states, change PL1 and PL2, etc. But nothing seems to meaningfully move the needle on this . . . incredibly frustrating. If anyone else who has been following this is running N100-based hardware, I'd be curious to know the results of:
openssl speed -elapsed -evp aes-256-cbc
for you. I can't help but wonder whether this is really highly specific to the hardware/BIOS combination I have, or if performance may be degraded generally for the N100 in FreeBSD, just not so much that it's generally noticeable when used for applications such as pfSense. That seems highly unlikely, but any points of comparison would be welcome. Thank you! -
One other observation - likely not related, but just piling on the strangeness here - is that iperf tests to this machine exhibit strange behavior. If it's been sitting idle for a bit, usually the first test shows about the expected 1Gbps speed (950+Mbps). But if I re-run several times, it consistently drops until it hits ~400Mbps and seems to stay there. I realize that iperf tests to the firewall as opposed to through it are of limited value, but I'm again using the N5105 box I have as a point of comparison, which gets consistent 950+Mbps iperf results. That machine has i225 NICs as opposed to the i226 in my N100 machine, but are otherwise configured identically. Getting ready to cut my losses and toss this Topton into an industrial shredder, but I sure would love to gain some understanding of what's going on before doing so, ha.
-
Have you checked the heatsink on that device is actually attached in a useful way? Wouldn't be the first China sourced device people have seen with the wrong shim or no thermal paste that was just running into thermal throttling all the time.
-
@stephenw10 Funny you should mention that, because my mind was going in the same direction. I think I'll take a closer look today. Being aware of such issues, I did peek under the board when I installed the RAM and SSD and was able to see the interface to the heatsink with no obvious gap. But I have some decent thermal compound on hand so I'm thinking I may disassemble it, clean off the stock compound, then re-mount the board before adding more compound and see whether there's a visible gap then . . . will report back again
-
Of course I realize only after writing this that I have the (seemingly) compelling data point of booting Linux and having it consistently knock the openssl test out of the park. That said, I've got no other ideas and not much to lose, so I'll still look at the heatsink situation . . .
-
@TheNarc Here are my results:
openssl speed -elapsed -evp aes-256-cbc You have chosen to measure elapsed time instead of user CPU time. Doing AES-256-CBC for 3s on 16 size blocks: 103729914 AES-256-CBC's in 3.01s Doing AES-256-CBC for 3s on 64 size blocks: 35140334 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 256 size blocks: 9071472 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 1024 size blocks: 2274966 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 8192 size blocks: 281410 AES-256-CBC's in 3.00s Doing AES-256-CBC for 3s on 16384 size blocks: 143837 AES-256-CBC's in 3.00s version: 3.0.12 built on: reproducible build, date unspecified options: bn(64,64) compiler: clang CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x98c007bc239ca7eb The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes AES-256-CBC 551789.26k 749660.46k 774098.94k 776521.73k 768436.91k 785541.80kI think my board should also be compatible with the BIOS you flashed but I'm still running the original bios. I have speed shift enabled with the bar dragged all the way to "Energy efficency". I have also set the system tunable "hw.acpi.cpu.cx_lowest" to "Cmax" which seems like it allows the CPU cores to enter C2 and C3 states.
Scores seem much better than yours in pfSense but strangely worse than yours from Linux for small sizes but even better for larger sizes?
I'm not sure what to make of the CPU frequency readings, in the pfSense GUI the highest I've seen on mine is ~2200 MHz I think, but it always says 806 MHz max. The Zone 0 temperature is also permanently stuck at 27.9 °C but the temperatures for the cores seem correct. My box came with Windows 11 Pro preinstalled and in Windows it did report speeds up to 3.4 GHz in task manager. When running the benchmark the CPU frequency in the pfSense GUI drops though instead of increase. Freqs reported by sysctl dev.cpu.x.freq seem to always be between 1410 - ~1700, doesn't seem to match at all what the pfSense web GUI reports.
Not sure if my box is performing as it should under pfSense but it's still more than fast enough for what I need it for. Instead though it doesn't seem like it's fully stable. When I let the preinstalled windows install some updates it crashed once with DRIVER_IRQL_NOT_LESS_OR_EQUAL but continued installing the update succesfully after. pfSense initially seemed stable but after 1-2 days it crashed and rebooted with a "general protection fault". I'm still troubleshooting this but if I can somehow stabilize this machine I will be super happy with it.
-
@AnonymousRetard Thank you for that! Very good to have a point of comparison. I may try the Cmax setting too, although I thought that only allowed it to go into deeper energy saving states, and therefore wouldn't have an impact on the clock speed not going high enough. Although I don't know what's going on at this point :)
I definitely had my system in the state you describe at one point too where the reported frequency dropped under load instead of increased, but the benchmark numbers seemed to indicate that the speed was indeed being boosted, not throttled.
I'm also not even sure how relevant or good of a choice this openssl benchmark is for me to be using, but it's the only thing I'm aware of that I can use without installing other packages. Thanks again and I'll post back after checking my thermal interface to the heatsink and/or trying that Cmax setting.
-
@TheNarc Good luck! I had not really thought about benchmarking my performance at all before reading this because the box just performs extremely much better than my old PCEngines APU. I no longer need to limit the amount of pfBlockerNG IP or DNSBL lists I want to load, and I've been able to run suricata inline IPS mode and FQ_CODEL limiters to make sure my latency no longer increases as I saturate my uplink (1Gbps) without loosing much throughput at all and it looks like the CPU usage barely budges still. On my previous router I couldn't run all these things and had to do a LOT of tweaking to be able to almost saturate my 1 Gbps uplink.
Therefore I have just been focusing on trying to reduce the power usage as much as possible which I think the Cmax setting help with. I doubt it increases performance but I guess it might when you run a single-threaded test like this openssl one (if you add -multi 4 as an argument though it seems to fully load all cores), because if the other cores go to sleep there should be a larger power budget left for the remaining core.
-
@AnonymousRetard Thanks! By the way, I don't want to ask you to try anything else unless you feel like it, but if you do . . . I'd be curious to know whether you see similar iperf results as me or if you're always steady near a full 1Gbps. On my system, the fact that it's initially what I'd expect and then it drops if I keep running it does make me wonder about thermal throttling.
-
@TheNarc No problem! I have not ran iperf before but appearantly iperf3 was already installed on my Gentoo machine in the LAN so I installed it on pfSense as well. Here's what I get when pfSense acts as the server:
iperf3 -c 192.168.1.1 -p 5201 Connecting to host 192.168.1.1, port 5201 [ 5] local 192.168.1.11 port 49202 connected to 192.168.1.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 114 MBytes 951 Mbits/sec 0 502 KBytes [ 5] 1.00-2.00 sec 108 MBytes 909 Mbits/sec 0 502 KBytes [ 5] 2.00-3.00 sec 104 MBytes 876 Mbits/sec 0 502 KBytes [ 5] 3.00-4.00 sec 112 MBytes 935 Mbits/sec 0 502 KBytes [ 5] 4.00-5.00 sec 113 MBytes 947 Mbits/sec 0 502 KBytes [ 5] 5.00-6.00 sec 112 MBytes 943 Mbits/sec 0 502 KBytes [ 5] 6.00-7.00 sec 114 MBytes 953 Mbits/sec 0 765 KBytes [ 5] 7.00-8.00 sec 112 MBytes 935 Mbits/sec 0 765 KBytes [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 765 KBytes [ 5] 9.00-10.00 sec 112 MBytes 939 Mbits/sec 0 765 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec receiverI ran it multiple times and there's not much of a difference. Worst bitrate I saw was 814, some runs had no bitrates below 900, but the speeds definately don't degrade after multiple runs. Under "htop" it looks like all cores reach almost 100% during this test but I have no idea what frequency they run at during the test, since I have speed shift set to 100% efficency I guess they might not be clocking to the highest they can. I also tried disabling/enabling suricata but it really didn't make any difference. Oh and I also have the tunable "dev.igc.0.fc" (WAN) and "dev.igc.1.fc" (LAN) both set to 0 because it seemed to give more consistent speeds on the Ookla speedtest website.
But about the temperatures I can tell you my box was initially idling with core temperatures around 50-52C with the default pfSense settings. After enabling speed shift set to 100% effiency, disabling PowerD and enabling all C-states through the tunable the idle temperatures dropped to ~45-46C. At the moment though the idle temperatures are only ~27C because after the last crash I brought a USB desk fan from work to blow on the case to see if it helps with my stability issue. I don't think ~50C should be a problem for the CPU but perhaps the ambient temp inside the case gets too high for the RAM to be fully stable is my hope... I really should run memtest86 on the machine but it's currently "in production" already running my home network and the wife gets a bit mad every time I reboot the machine or tinker with something else which brings the internet down.
Since just blowing on the top of the case with a fan is able to drop the CPU core temps by close to 20C and almost down to ambient room temp it feels like the CPU must already have pretty good thermal coupling to these fins on the outside. But I doubt the NVME drive and SODIMM ram stick inside has any heatsinks at all and theres no working "ambient" thermal sensor on the inside I've been able to find so because the case has no holes I wouldn't be surprised if the ambient temp inside the case becomes close to the CPU core temps.
-
@AnonymousRetard Thank you, much appreciated! So you're definitely getting the results I'd expect. I really don't know what the deal with my machine is. I disassembled it and the heatsink solution isn't what you'd love to see: an aluminum block sandwiched between the CPU dies and the (painted!) top inside surface of the case. There were no obvious gaps though. So I cleaned off the existing thermal compound, used a crappy little rotary tool I have to get off as much of the paint as I was able to with a wire wheel, cleaned everything, and reassembled using my thermal compound. That did have the effect of dropping my temps a few degrees I think, but they were only getting up to the low 40s before.
Curiously (or maybe not, since C-states are low power states) I'm finding that I get better performance on the openssl test with I disable C-states in the BIOS. It sure feels like FreeBSD is just throttling the CPU as extremely as it can. I did notice that it's not logging the temps in System > Monitoring, which my N5105 system does. But the 4 core temps do show up and seem accurate on the dashboard.
Honestly I was out of my depth a number of posts ago in this thread, and I'm not sure I have the stamina to try changing one BIOS setting at a time and rebooting to see whether it makes any difference. Just very frustrating . . . if it weren't for the normal performance I'm seeing under Linux, I could at least write it off to a potential hardware defect, but that not being the case there's the tantalizing possibility of being able to get it to work like it should if I could only tweak the right things.
Anyway, thanks again for your time. And I certainly empathize with the difficulty of fooling around with these things in a "live" environment! Thankfully for the time being I've got the old apu2 swapped back in.
-
This post is interesting and seems to suggest that the FreeBSD scheduler with respect to E-cores is lacking when compared to Linux. Of course, it's just a random forum post, but the N100 has no P-cores, and the poster was specifically referring to variability in results of the same openssl speed test I've been running.
-
@TheNarc No problem! Really strange problem you have... I took another round thinking what I could be and realized that we probably do not have the same board. I never followed the link to the BIOS you said you used until now and when I did I realized it's not the same one I added a link to of possible things to try later. For me it's this one: https://forums.servethehome.com/index.php?threads/cwwk-topton-nxxx-quad-nic-router.39685/page-75#post-405132 . I haven't double-checked yet that it matches 100% and I might never do that but I do remember my bios said "BK-1264NP Ver 1.5" somewhere as I was setting up the device. I have not yet checked dates and numbers on stickers but I'm pretty sure that system matches mine.
I guess my specific board/BIOS version could be better supported in FreeBSD than what yours is? It does really sound like your CPU is not being allowed to clock as high as it should in FreeBSD, especially with the temperatures you say you've had. I mean mine started out at 50-52 before any tweaking for lower power use...
Edit: Oh nvm.. you did link to the same BIOS... it's just you had another link as well to the unlocking procedure which talks about another BIOS... This makes it even stranger if we started from the same board and BIOS... Are you also running the latest pfSense 2.7.2?
-
@AnonymousRetard Yeah it definitely seems like we have the same machine. And I am also running 2.7.2. I even tried booting the live FreeBSD distribution NanoBSD and got similarly poor results there, so it's not just a pfSense thing. It definitely is confusing; if I didn't have the data point of Linux seeming to perform as I'd expect, I'd definitely chalk it up to losing the hardware lottery. But I can't do that so it will just keep driving me crazy until I give up at some point :)
-
Well, in late-breaking news, something I wangjangled in the BIOS seems to have made a difference. The results are still different than yours, but kind of on-par, plus I can see pfSense claiming it's boosting up to 3+GHz now:
/home/john/better_openssl_speed.jpegNow the bad news is: I touched a lot of settings in the BIOS this last time. I was kind of going for broke. I may try to get pictures of all the screens (in fact I probably should in case the settings ever get defaulted and I need to get back to this point), but it's going to be a lot of screens! I've wasted enough time on this today that I may wait until tomorrow to do that. And then I need to decide whether I care enough to methodically try to determine which setting or few settings out of the ones I changed actually made the difference.
-
I do still see the drop-off in the iperf testing though. That one's odd because I can now see the CPU speed going up and corresponding to the more expected results of 900+ Mbps. But if I keep re-running the test, it's as if something decides it no longer needs to increase the CPU speed, and I see it fall again along with the iperf speeds. That I don't care as much about so long as traffic through the system isn't impacted, but to test that I'll need to get it fully back in service again and run some Internet speed tests.
-
Hmm, fun*!
-
@TheNarc Very interesting! In my original BIOS I don't have these settings unlocked but in general I think these CPUs should have PL1 setting which says what the maximum power consumption for a low load scenario is. Then there should be a PL2 setting which says what the maximum wattage should be while boosting. Then there should also be a configured time limit for how long the CPU should be allowed to be in the PL2 state. After this time limit it should be forced back to the maximum power usage from the PL1 setting.
If your CPU is not thermal throttling, could it perhaps be that you reached the configured time limit for the PL2 state?
If you are lucky maybe you can find these settings in your unlocked BIOS and tweak them further?
For example maybe with a higher PL1 setting you could remove the visible iperf performance drop even after the PL2 time limit has ran out?Bad PL1 and PL2 settings don't quite explain though how you could get better performance in Linux... so yeah if you could find out what you changed to finally see the 3GHz in pfSense I would be very happy to learn that as well.
I would be very interested on anything you can learn about this board and BIOS really. I might also change from my original BIOS in the future and would like to learn as much as possible. If you decide to keep digging and find out exactly what fixed or at least partly fixed your issue I would be very interested in that as well. But no rush at all. It's time for me to go to bed for today as well so you're not likely to get more responses from me tonight either.
-
@AnonymousRetard Thanks! Have a good night. I did play around some with the PL1 and PL2 settings and those changes alone did not seem to make a difference, but I didn't configure time limits on them either. I also lied and went ahead and took pictures of all the BIOS screens on which I know I changed at least one setting from the default. Which ones? It's anyone's guess, as I had lost any incentive to be methodical by the point I was fiddling with a bunch of them :) So I am not asking anyone to analyze or postulate on these, but posting them here just in case they're of interest.
Of note, both of the modded BIOSes that I tried were indistinguishable (version-wise) from the one that was originally loaded. So perhaps it only looked like we had the same BIOS, but behind the scenes they had different default values. Who knows.
Anyway, here's a bunch of low-quality phone pics of BIOS setting screens :)
/home/john/Pictures/bios1.jpeg
/home/john/Pictures/bios2.jpeg
/home/john/Pictures/bios3.jpeg
/home/john/Pictures/bios4.jpeg
/home/john/Pictures/bios5.jpeg
/home/john/Pictures/bios6.jpeg
/home/john/Pictures/bios7.jpeg
/home/john/Pictures/bios8.jpeg
/home/john/Pictures/bios9.jpeg
/home/john/Pictures/bios10.jpeg
/home/john/Pictures/bios11.jpeg
/home/john/Pictures/bios12.jpeg
/home/john/Pictures/bios13.jpeg
/home/john/Pictures/bios14.jpeg
/home/john/Pictures/bios15.jpeg
