Netgate 7100 freezes when temperature above 50°C
-
Most of our 7100 are in climatisated server room, so I never had an issue like that. I am looking in one 7100 in France, and in GUI is 37.0°C .I assume you should check enviroment and make somu tuning of rack temperature. Can high temperature went out of rack? Is there ventilation?
-
@stephenw10 Can the fan controller be controlled manually?
This thread has a similar issue but doesn't have an answer if it can be tuned https://forum.netgate.com/topic/142945/how-to-control-fans-on-xg7100
-
There are options for changing the fan controller behaviour. It shouldn't normally be required but can be useful as a test. Mostly we have used it to reduce the fan speed and hence noise in a desktop environment.
I can send you a script to test if you're local to the unit. I wouldn't recommend it for remote use, at least not initially. -
@stephenw10 Yes please. One solution I thought about was strapping a few more fans on the sides so if you have a script that would increase the fan speed I'd be happy wiith it
-
Update: With the script of @stephenw10 I was able to lower the temperature to levels that would stop the crashes. So this topic should be resolved. Thanks for the help
-
@geek_at this got me curious to what my temps were at.. I use to run around 50ish, but after the update to 23.09 there seems to have be a drastic drop in normal running temp
Where I see the drastic drop seems to correspond to when 23.09 came out. Its possible I stop using some package or something? But I don't recall turning off anything? Or doing any sort of significant change that could account for the drastic drop off in overall temp.. I looked at the number of processes running or the cpu use over the same period and don't see any changes there that could account for the drastic drop?
And temp in the house or my office where pfsense sure didn't change all that much..
Curious to why the sudden drop off now - maybe reading of the senors changed? My sg4860 doesn't even have any fans that I am aware of - sure can't hear anything.. So I don't think it could of been a change to how the fans run..
Curious - I do like the lower temps..
Also curious to why it was running hot there for a while.. Either way even when it was running on the warm side of 50+ I never had any issues with it hanging or rebooting or anything - the thing runs, and the only time it reboots is when I update it to new version of pfsense.
50ish seems a bit low to be causing any sort of the thermal problem..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.03 | Lab VMs 2.7.2, 24.03 -
@johnpoz said in Netgate 7100 freezes when temperature above 50°C:
50ish seems a bit low to be causing any sort of the thermal problem..
It does. Turning up the fans seems to confirm it's a thermal issue but I wonder if it's actually some other component that's overheating.
I'm not sure what might have affected a 4860 like that. Do the CPU graphs show a load reduction?
-
@stephenw10 no I looked no cpu reduction nor processes running that would reflect such a change..
-
Hmm, interesting. I'm nowt aware of anything that would affect that in the power/thermal management.
-
@johnpoz that's actually a very good point. I too was still on 22 and just upgraded to 23. Let's see if that changes anything without the fan script
-
Update: after the update to the latest version the 7100 didn't find any boot partition anymore.. so I'm done with this and I'm going to virtualize from now on.
-
@geek_at Okay one more update.
I opened the 7100 up and found out that the heatsinks were installed in the wrong direction
They should align with the airflow but are rotated 90°
This doesn't seem to be normal as in the official documentation of the 7100 the heatsinks are aligned correctly
https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/m-2-sata-installation.html
It seems there was a mistake in the production process around the time my clients bought their Firewalls. This explains the overheating and random crashes my clients were experiencing.
-
It still seems as though something else is happening there because 50C really isn't that hit for that CPU.
But if that lowers the temps and prevents it....
-
@stephenw10 but temps are where the measurement are taken, I would think it possible without proper cooling that while the measurement point might only show 50, that its hotter at some other spot.. And maybe that is where the problem is?
It is a good finding.. Is this a known issue where the heat sink does seems to be installed in the wrong orientation?
Simple google for heat sinks and air flow does point to the wrong orientation being problematic for proper cooling.
-
Well the heatsink only cools the CPU in the 7100. The measurement is using the on-die sensors in the CPU package. Having the heatsink oriented incorrectly causes it to run hotter and hence the fans run faster It probably actually lowers the temps of everything else.
-
Okay I'm back with new thermal paste and correctly aligned heatsinks.
I ran a CPU benchmark for 45 minutes. It started out with a CPU temp of 40°C
The first half it ran just on default settings at 100% CPU. Temp went up to to 58°C without crashes. So the orientation did improve things greatly which also adds to my suspicion that the problem in the production process was to blame for my freezing problems.
I also tested running the "smart fan control" script which reduced the temperatures back to ~42°C even at 100% load. Here's the graph:
I will check all remaining 3 firewalls of my clients to see if the heatsinks are aligned wrong there too
-
I run that script here and have never seen any issues with it. However there is a risk that if it crashes or is otherwise killed for any reason the fans will just remain at whatever speed there are running. That means if it was very light load it may then be insufficient at higher loads.
As an alternative we do have a script that resets the lookup tables in the fan controller but leaves the controller in charge, independent of the CPU. It's not as good as actually maintaining the temperature though since it still relies on the board sensors.Do you have a temperature value for the CPU without the script running? Just comparing the before and after remounting the heatsink?
Steve
-
@stephenw10 In my last screenshot, the when the temperature rises, this was without the script.
I started the script at between 19:25 and 19:33 and you can see it working and the temperature falling. The script works perfectly (well except for the error you get when the script tries to spin up the fans over a value of 256 which results in errors)
I don't really have a "before" benchmark with CPU load but I will try to do it at another customers location soon
-
Ok great. I'd guess it's not dramatically different. Those numbers seems to indicate that.
-
I wanted to give an update on the matter.
I have opened and checked all of the Netgate boxes of my clients and my suspicions were correct. All clients who have experienced outages and random crashes indeed hat the heatsinks mounted in the wrong direction (against airflow) and all clients who had no problems had them in the correct orientation (same as in the pictures of the official documentation)
After fixing the heatsinks I had no more crashes even with forced heat and burn-in-tests. So this really was the cause for the crashes.
@stephenw10 please talk with your QA people about this
Best wishes from Austria
