CPU jumps to 100% every night
-
I am running 2 Netgate SG-4860 in High Availability. Every night between midnight and 3 AM at a random time the CPU goes from an average of 55% to 100% and starts dropping connections. It lasts for about 20-30 minutes. The amount of traffic passing through doesn't change. There's nothing in the logs. We don't have any special processes that we know of that run between those hours. The only reason we noticed the issue is we use DYN active failover and we failover to our DR facility every night when the pings start getting dropped. Datacenter monitoring doesn't see any increase in traffic or type of attack (and I assume an attack would leave some trace in the logs). We used ping plotter and can see when it hits 100% it starts dropping out.
I am out of ideas! Any suggestions are greatly appreciated.
Gary
-
What packages do you have installed? Nothing in the pfSense base would do that.
-
Is it 100% of the total CPU or one core? Can you run top in CLI and check what process is taking so much CPU?
The only time I had 100% CPU on one core was when the pfSense update servers were down and that was causing pkg-static to consume 100% of one CPU core until the server was back up. Maybe there is half a chance your firewalls are having problems reaching the pfSense update servers every night for some reason?
Here is the related thread: https://forum.netgate.com/topic/139903/available-packages-is-empty-in-package-manager/19
-
Only package installed is pfBlockerNG.
-
@gcjh01 said in CPU jumps to 100% every night:
Only package installed is pfBlockerNG.
The one that refreshes its rather big lists every x hours (24 ?) ?
Or
De activate pfBlockerNG - and see what happens ^^
-
@Gertjan The refresh was set to once an hour so that doesn't really explain every night between 12-3 AM. I am however going to disable it tonight and see if it changes anything... Thanks!
-
@gcjh01 said in CPU jumps to 100% every night:
@Gertjan The refresh was set to once an hour so that doesn't really explain every night between 12-3 AM. I am however going to disable it tonight and see if it changes anything... Thanks!
There can be a "check for updates" each hour, but if the posted file has not changed nothing might happen on the pfSense side until the file actually is updated on the server. That might happen only once per 24 hours, for example. That's how the Snort and Suricata IDS/IPS packages I maintain work. They check for updates frequently, but if the MD5 hash of the posted file is the same as that of the last downloaded version of the file, then nothing is actually updated and downloaded. If the MD5 hash has changed, then the new file is downloaded, unpacked and processed.
-
@bmeeks Good point...
-
Did you inspect the pfblockerng.log to see what is done during that period?
