Netgate 7100 freezes when temperature above 50°C
-
@geek_at Okay one more update.
I opened the 7100 up and found out that the heatsinks were installed in the wrong direction
They should align with the airflow but are rotated 90°
This doesn't seem to be normal as in the official documentation of the 7100 the heatsinks are aligned correctly
https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/m-2-sata-installation.html
It seems there was a mistake in the production process around the time my clients bought their Firewalls. This explains the overheating and random crashes my clients were experiencing.
-
It still seems as though something else is happening there because 50C really isn't that hit for that CPU.
But if that lowers the temps and prevents it....
-
@stephenw10 but temps are where the measurement are taken, I would think it possible without proper cooling that while the measurement point might only show 50, that its hotter at some other spot.. And maybe that is where the problem is?
It is a good finding.. Is this a known issue where the heat sink does seems to be installed in the wrong orientation?
Simple google for heat sinks and air flow does point to the wrong orientation being problematic for proper cooling.
-
Well the heatsink only cools the CPU in the 7100. The measurement is using the on-die sensors in the CPU package. Having the heatsink oriented incorrectly causes it to run hotter and hence the fans run faster It probably actually lowers the temps of everything else.
-
Okay I'm back with new thermal paste and correctly aligned heatsinks.
I ran a CPU benchmark for 45 minutes. It started out with a CPU temp of 40°C
The first half it ran just on default settings at 100% CPU. Temp went up to to 58°C without crashes. So the orientation did improve things greatly which also adds to my suspicion that the problem in the production process was to blame for my freezing problems.
I also tested running the "smart fan control" script which reduced the temperatures back to ~42°C even at 100% load. Here's the graph:
I will check all remaining 3 firewalls of my clients to see if the heatsinks are aligned wrong there too
-
I run that script here and have never seen any issues with it. However there is a risk that if it crashes or is otherwise killed for any reason the fans will just remain at whatever speed there are running. That means if it was very light load it may then be insufficient at higher loads.
As an alternative we do have a script that resets the lookup tables in the fan controller but leaves the controller in charge, independent of the CPU. It's not as good as actually maintaining the temperature though since it still relies on the board sensors.Do you have a temperature value for the CPU without the script running? Just comparing the before and after remounting the heatsink?
Steve
-
@stephenw10 In my last screenshot, the when the temperature rises, this was without the script.
I started the script at between 19:25 and 19:33 and you can see it working and the temperature falling. The script works perfectly (well except for the error you get when the script tries to spin up the fans over a value of 256 which results in errors)
I don't really have a "before" benchmark with CPU load but I will try to do it at another customers location soon
-
Ok great. I'd guess it's not dramatically different. Those numbers seems to indicate that.
-
I wanted to give an update on the matter.
I have opened and checked all of the Netgate boxes of my clients and my suspicions were correct. All clients who have experienced outages and random crashes indeed hat the heatsinks mounted in the wrong direction (against airflow) and all clients who had no problems had them in the correct orientation (same as in the pictures of the official documentation)
After fixing the heatsinks I had no more crashes even with forced heat and burn-in-tests. So this really was the cause for the crashes.
@stephenw10 please talk with your QA people about this
Best wishes from Austria
-
I will do. Thanks for checking that.
