Netgate 7100 freezes when temperature above 50°C
-
Hello!
I'm a MSP and 4 of my clients have a Netgate 7100. Three of them have the problem that some times the firewall becomes completely unresponsive to web requests, routing and even USB console.
After many hours of checking factors and testing I came to the conclusion that it only happens when the temperature rises slightly (but nowhere near "dangerous" temperatures). Usually this happens when the captive portal is in use quite a bit and the CPU temperature rises to 50 or above.
I have to add that the three clients experiencing this problem don't have ventilated server racks.
What made the situation a bit better (from daily freezes to only weekly or monthly) was me putting the Netgate 7100 in the bottom of the shelf so the heat from other appliances don't increase the temp of the firewall.
But I'm still confused because how can a €1000+ firewall crash at such low(ish) temperatures? Anyone else have seen this issue?
-
Noop. Look for yourself. This is a 4100 with a company network and a captive portal.
I've purged the stats from before October 2023, and I'm sure during summer it was over 50 degrees °C (a couple of degrees).My two measurements are the on-die Intel sensors. It doesn't know anything about the surrounding chips on the mother board, of course.
The Intel processor would pull the plug, true, but not at 50 °C, more like '100'. I presume something else fails, taking the system with it. -
Are those the coretemp CPU values? 50C is nowhere near the CPU limit for the on-die sensor.
Are there any errors logged when this happens?
Are the fans running?
-
@stephenw10 Yes These are the CPU temperature values from the Interface.
The fans are spinning (very silently though) and no errors are reported after the crash. No crash dump, no errors before the crash either in the logs.
The 3 firewalls at 3 customers locations were also not bought at the same time, about a year apart each
-
If they are rack mounted where noise isn't an issue it's possible to retune the fan controller. I would expect the fans to be running at their higher speed anyway at those CPU temps though. Unless the ambient temp is especially low perhaps.
-
Most of our 7100 are in climatisated server room, so I never had an issue like that. I am looking in one 7100 in France, and in GUI is 37.0°C .I assume you should check enviroment and make somu tuning of rack temperature. Can high temperature went out of rack? Is there ventilation?
-
@stephenw10 Can the fan controller be controlled manually?
This thread has a similar issue but doesn't have an answer if it can be tuned https://forum.netgate.com/topic/142945/how-to-control-fans-on-xg7100
-
There are options for changing the fan controller behaviour. It shouldn't normally be required but can be useful as a test. Mostly we have used it to reduce the fan speed and hence noise in a desktop environment.
I can send you a script to test if you're local to the unit. I wouldn't recommend it for remote use, at least not initially. -
@stephenw10 Yes please. One solution I thought about was strapping a few more fans on the sides so if you have a script that would increase the fan speed I'd be happy wiith it
-
Update: With the script of @stephenw10 I was able to lower the temperature to levels that would stop the crashes. So this topic should be resolved. Thanks for the help
-
@geek_at this got me curious to what my temps were at.. I use to run around 50ish, but after the update to 23.09 there seems to have be a drastic drop in normal running temp
Where I see the drastic drop seems to correspond to when 23.09 came out. Its possible I stop using some package or something? But I don't recall turning off anything? Or doing any sort of significant change that could account for the drastic drop off in overall temp.. I looked at the number of processes running or the cpu use over the same period and don't see any changes there that could account for the drastic drop?
And temp in the house or my office where pfsense sure didn't change all that much..
Curious to why the sudden drop off now - maybe reading of the senors changed? My sg4860 doesn't even have any fans that I am aware of - sure can't hear anything.. So I don't think it could of been a change to how the fans run..
Curious - I do like the lower temps..
Also curious to why it was running hot there for a while.. Either way even when it was running on the warm side of 50+ I never had any issues with it hanging or rebooting or anything - the thing runs, and the only time it reboots is when I update it to new version of pfsense.
50ish seems a bit low to be causing any sort of the thermal problem..
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
@johnpoz said in Netgate 7100 freezes when temperature above 50°C:
50ish seems a bit low to be causing any sort of the thermal problem..
It does. Turning up the fans seems to confirm it's a thermal issue but I wonder if it's actually some other component that's overheating.
I'm not sure what might have affected a 4860 like that. Do the CPU graphs show a load reduction?
-
@stephenw10 no I looked no cpu reduction nor processes running that would reflect such a change..
-
Hmm, interesting. I'm nowt aware of anything that would affect that in the power/thermal management.
-
@johnpoz that's actually a very good point. I too was still on 22 and just upgraded to 23. Let's see if that changes anything without the fan script
-
Update: after the update to the latest version the 7100 didn't find any boot partition anymore.. so I'm done with this and I'm going to virtualize from now on.
-
@geek_at Okay one more update.
I opened the 7100 up and found out that the heatsinks were installed in the wrong direction
They should align with the airflow but are rotated 90°
This doesn't seem to be normal as in the official documentation of the 7100 the heatsinks are aligned correctly
https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100-1u/m-2-sata-installation.html
It seems there was a mistake in the production process around the time my clients bought their Firewalls. This explains the overheating and random crashes my clients were experiencing.
-
It still seems as though something else is happening there because 50C really isn't that hit for that CPU.
But if that lowers the temps and prevents it....
-
@stephenw10 but temps are where the measurement are taken, I would think it possible without proper cooling that while the measurement point might only show 50, that its hotter at some other spot.. And maybe that is where the problem is?
It is a good finding.. Is this a known issue where the heat sink does seems to be installed in the wrong orientation?
Simple google for heat sinks and air flow does point to the wrong orientation being problematic for proper cooling.
-
Well the heatsink only cools the CPU in the 7100. The measurement is using the on-die sensors in the CPU package. Having the heatsink oriented incorrectly causes it to run hotter and hence the fans run faster It probably actually lowers the temps of everything else.
