Syslogd Spiking CPU

phospher

Hi, I know that this issue has cropped up from time to time but I've never been able to find a good answer or fix on here.

I just had a production firewall running 1.2.3-Release have it's cpu spike to 100%. The firewall has been up without reboot for 186 days and has not missed a beat until this issue. Can anyone provide more info into this?

Hardware is a 2.8 Ghz P4 1 Gig ram and Intel Pro 1000 nics.

Thanks,

jimp

Have you looked at all of the logs to see if anything is being repeatedly written to a log file?

And what does the output of "top -SH" look like from the shell?

And the CPU graph?

phospher

Last night I killed the syslogd process and it resolved the issue. The graph shows that the cpu(system i think) has been at or near 100% for the past 72hrs. Suprisingly, I didn't see any performance degredation. I think top -SH is pointless at this point since it's running normal again(?).

I do see this in the logs, I believe that at these particular time stamps the issue was still occuring.

Jun 24 00:10:57 fw-1 dnsmasq[30923]: overflow: 155 log entries lost
Jun 24 00:10:57 fw-1 dnsmasq[30923]: overflow: 155 log entries lost

Not sure if that is a cause of the syslogd at 100% or the effect of it being at 100%.

Thanks,

jimp

You are correct that top -SH would only be useful when the problem happens.

I'm not sure if killing syslogd actually fixed the problem or just stopped the log entries from being handled. The "real" failing process may still be generating log messages but if syslogd isn't running, they aren't getting logged.

phospher

It is in-fact logging again. syslogd seems to have automatically restarted itself right after I killed the process. I'm using clog to look a the logs and they seem to be working correctly based on the few files I looked at(system.log,filter.log). Dunno.

jimp

It's possible that something made it go crazy temporarily, but without more info while it's happening "live" it's really hard to speculate as to what it may have been.

phospher

Yeah, I'll try and gather more info if it happens again though It may be another 6 months :) Thanks