Netgate 7100 freezes when temperature above 50°C

Gertjan

Noop. Look for yourself. This is a 4100 with a company network and a captive portal.
I've purged the stats from before October 2023, and I'm sure during summer it was over 50 degrees °C (a couple of degrees).

My two measurements are the on-die Intel sensors. It doesn't know anything about the surrounding chips on the mother board, of course.
The Intel processor would pull the plug, true, but not at 50 °C, more like '100'. I presume something else fails, taking the system with it.

stephenw10

Are those the coretemp CPU values? 50C is nowhere near the CPU limit for the on-die sensor.

Are there any errors logged when this happens?

Are the fans running?

geek_at

@stephenw10 Yes These are the CPU temperature values from the Interface.

The fans are spinning (very silently though) and no errors are reported after the crash. No crash dump, no errors before the crash either in the logs.

The 3 firewalls at 3 customers locations were also not bought at the same time, about a year apart each

stephenw10

If they are rack mounted where noise isn't an issue it's possible to retune the fan controller. I would expect the fans to be running at their higher speed anyway at those CPU temps though. Unless the ambient temp is especially low perhaps.

GeorgeCZ58

Most of our 7100 are in climatisated server room, so I never had an issue like that. I am looking in one 7100 in France, and in GUI is 37.0°C .I assume you should check enviroment and make somu tuning of rack temperature. Can high temperature went out of rack? Is there ventilation?

geek_at

@stephenw10 Can the fan controller be controlled manually?

This thread has a similar issue but doesn't have an answer if it can be tuned https://forum.netgate.com/topic/142945/how-to-control-fans-on-xg7100

stephenw10

There are options for changing the fan controller behaviour. It shouldn't normally be required but can be useful as a test. Mostly we have used it to reduce the fan speed and hence noise in a desktop environment.
I can send you a script to test if you're local to the unit. I wouldn't recommend it for remote use, at least not initially.

geek_at

@stephenw10 Yes please. One solution I thought about was strapping a few more fans on the sides so if you have a script that would increase the fan speed I'd be happy wiith it

geek_at

Update: With the script of @stephenw10 I was able to lower the temperature to levels that would stop the crashes. So this topic should be resolved. Thanks for the help

johnpoz

@geek_at this got me curious to what my temps were at.. I use to run around 50ish, but after the update to 23.09 there seems to have be a drastic drop in normal running temp

Where I see the drastic drop seems to correspond to when 23.09 came out. Its possible I stop using some package or something? But I don't recall turning off anything? Or doing any sort of significant change that could account for the drastic drop off in overall temp.. I looked at the number of processes running or the cpu use over the same period and don't see any changes there that could account for the drastic drop?

And temp in the house or my office where pfsense sure didn't change all that much..

Curious to why the sudden drop off now - maybe reading of the senors changed? My sg4860 doesn't even have any fans that I am aware of - sure can't hear anything.. So I don't think it could of been a change to how the fans run..

Curious - I do like the lower temps..

Also curious to why it was running hot there for a while.. Either way even when it was running on the warm side of 50+ I never had any issues with it hanging or rebooting or anything - the thing runs, and the only time it reboots is when I update it to new version of pfsense.

50ish seems a bit low to be causing any sort of the thermal problem..

stephenw10

@johnpoz said in Netgate 7100 freezes when temperature above 50°C:

50ish seems a bit low to be causing any sort of the thermal problem..

It does. Turning up the fans seems to confirm it's a thermal issue but I wonder if it's actually some other component that's overheating.

I'm not sure what might have affected a 4860 like that. Do the CPU graphs show a load reduction?

johnpoz

@stephenw10 no I looked no cpu reduction nor processes running that would reflect such a change..

stephenw10

Hmm, interesting. I'm nowt aware of anything that would affect that in the power/thermal management.

geek_at

@johnpoz that's actually a very good point. I too was still on 22 and just upgraded to 23. Let's see if that changes anything without the fan script

geek_at

Update: after the update to the latest version the 7100 didn't find any boot partition anymore.. so I'm done with this and I'm going to virtualize from now on.

geek_at

@geek_at Okay one more update.

I opened the 7100 up and found out that the heatsinks were installed in the wrong direction

They should align with the airflow but are rotated 90°

This doesn't seem to be normal as in the official documentation of the 7100 the heatsinks are aligned correctly

https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/m-2-sata-installation.html

It seems there was a mistake in the production process around the time my clients bought their Firewalls. This explains the overheating and random crashes my clients were experiencing.

stephenw10

It still seems as though something else is happening there because 50C really isn't that hit for that CPU.

But if that lowers the temps and prevents it....

johnpoz

@stephenw10 but temps are where the measurement are taken, I would think it possible without proper cooling that while the measurement point might only show 50, that its hotter at some other spot.. And maybe that is where the problem is?

It is a good finding.. Is this a known issue where the heat sink does seems to be installed in the wrong orientation?

Simple google for heat sinks and air flow does point to the wrong orientation being problematic for proper cooling.

stephenw10

Well the heatsink only cools the CPU in the 7100. The measurement is using the on-die sensors in the CPU package. Having the heatsink oriented incorrectly causes it to run hotter and hence the fans run faster It probably actually lowers the temps of everything else.

geek_at

Okay I'm back with new thermal paste and correctly aligned heatsinks.

I ran a CPU benchmark for 45 minutes. It started out with a CPU temp of 40°C

The first half it ran just on default settings at 100% CPU. Temp went up to to 58°C without crashes. So the orientation did improve things greatly which also adds to my suspicion that the problem in the production process was to blame for my freezing problems.

I also tested running the "smart fan control" script which reduced the temperatures back to ~42°C even at 100% load. Here's the graph:

I will check all remaining 3 firewalls of my clients to see if the heatsinks are aligned wrong there too