Rate taking 100% of CPU

Ocid · Aug 4, 2013, 3:43 AM

Lately I have noticed that my pfSense's temperature is over 90C and CPU usage 100%. System activity page shows that rate is taking 100% of CPU.

After reboot CPU load stays low for few days but then the rate once again starts to take everything my poor CPU has to offer.

What might be causing this problem?

kejianshi · Aug 4, 2013, 6:46 AM

What version pfsense?

Ocid · Aug 4, 2013, 7:06 AM

2.1-RC0 (i386)
built on Thu Jul 25 03:45:59 EDT 2013

This has happened with some previous versions (backed up on July 18 and July 12).

w0w · Aug 5, 2013, 7:24 AM

Do you have device poling enabled, any packages installed?

Ocid · Aug 5, 2013, 7:41 AM

Device polling is not enabled and the following packages are installed:

File manager
iftop
nmap
OpenVPN Client Export Utility

w0w · Aug 5, 2013, 8:28 AM

I've asked my friend who is freebsd guru and he told me that it could be anything that uses /usr/local/bin/rate. For 2.x versions of pfsense it is builtin rate package, that is used for Status > Traffic Graph
It could be also anything that uses rate.

Ocid · Aug 5, 2013, 8:51 AM

Well, I have been running Status: Traffic Graph page almost continuosly, maybe that has triggered rate's high CPU usage.

I have now updated to 2.1-RC1 (i386) built on Thu Aug 1 19:03:40 EDT 2013, current uptime ~22 hours and CPU usage is 5%. I keep running that Traffic Graph to see if I can reproduce that effect. If I can then I reboot and do not run Traffic Graph to see if it makes any difference.

jimp · Aug 6, 2013, 6:24 PM

That seems to be par for the course with rate. Sometimes it will just go nuts unpredictably. It will also occasionally just stop working and block a GUI thread when it gets stuck. It used to mess up the whole GUI on 2.0.2 and before when that happened.

Enrica_CH · Aug 13, 2013, 7:40 AM

I have a similar problem. After about one week of running the CPU usage goes to almost 100% but I can't open the console. All is frozen (console, web gui, interfaces). In my case the slave system (CARP) takes over.

I use 2.1-RC0 (amd64) built on Fri Jul 12 01:01:27 EDT 2013 on a KVM virtual instance. The processor is QEMU64. The old version 2.0.x never had a hanger. As NIC I use the e1000.

Is there a possibility to log the processes? I have no chance to see which process uses 100% of cpu.

wallabybob · Aug 13, 2013, 9:26 AM

@Erik_CH:

Is there a possibility to log the processes?

Leave the command```
top -S -H

Enrica_CH · Aug 13, 2013, 9:29 AM

This command I know but the console is frozen also. You can't type anything anymore.

I should have something that writes the same information to a file permanently and which is still available after a reboot.

wallabybob · Aug 13, 2013, 10:01 AM

@Erik_CH:

This command I know but the console is frozen also. You can't type anything anymore.

I should have something that writes the same information to a file permanently and which is still available after a reboot.

Once the system gets into that state it is pretty difficult to start something that will log to a file. If you log something to a file there is no guarantee you will be able to correctly close the file to make all its contents available after the reboot.

If the system genuinely freezes (in the sense of stopping running normal user processes) then leaving the suggested top command running on the console should give you an indication of heavy CPU users just before the freeze.

Enrica_CH · Aug 13, 2013, 10:07 AM

Thank you for your tip. I entered the top command let it running until it freezes again. On this frozen command shell I should see the last action. On my virtual instance I saw that just before freezing the CPU load increases to 80 % and then to 100%.

I will report my result as soon it happens again.

eri-- · Aug 13, 2013, 1:22 PM

How long do you leave rate running to have this behavior?
Probably even a libpcap bug as well!