2.0 RC1 CPU at 100% after 1-4 days
-
Try updating pfSense, you're using an older build, pfSense 2.0-RC1 is constantly updated sometimes more than once a day. I have an amd athlon k7 1.3Ghz with 512MB of ram from 2001 and I've never seen this problem and I've been using 2.0RC1 for just under a month and a half.
-
I'll try updating and keep an eye on it. Since this is newer than 1.5 months, I can't help but suspect some hardware configuration issue.
(I'd feel more confident about it if this was a known issue before that was resolved; the closest thing I found was a bug related to rate service (for traffic graphs) causing high CPU usage after a few days.) -
Top reports
674 processes, 262 running, 377 zombie
That you have so many inetd processes and zombie processes suggests there might be an issue with the interaction between inetd and a child process.
On my system
more /var/etc/inetd.conf
tftp-proxy dgram udp wait root /usr/libexec/tftp-proxy tftp-proxy -v
suggesting tftp is the only thing inetd is likely to start.
What uses inetd on your system? The pfSense shell command
clog /var/log/system.log | grep inetd
MIGHT provide some hints.
-
Thanks for the suggestions.
It happened again overnight, so only about 8 hours of uptime before it happened again. Another thing of note is that there are a ton of "nc" (netcat) processes along with all the zombie inetd processes.
I grepped the system log for inetd and didn't see any messages containing it (prior to rebooting.) Unfortunately I didn't realize the system.log was entirely wiped during a reboot, so I'll make sure to scp it over beforehand when it happens again.The only "non-standard" packages I have running are snort and openVPN.Something weird I noticed in the system.log is that each of the snort log entries is duplicated, such as
Mar 24 09:10:04 snort[52667]: --== Initialization Complete ==-- Mar 24 09:10:04 snort[52667]: --== Initialization Complete ==--I disabled snort from the webconfigurator and the system didn't recover, but I have disabled it for the time being to assist with the troubleshooting.
When (if) it happens again, I'll make sure I get the system.log to try and correlate and messages with when the system freaks out. -
What is in your /var/etc/inetd.conf?
Do you have anything attempting to use tftp or any other service in /var/etc/inetd.conf?
-
Looks like tftp and the firewall rules (first three digits edited to xxx) for a few of my external IP's
tftp-proxy dgram udp wait root /usr/libexec/tftp-proxy tftp-proxy -v 19000 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.26 xxx.xxx.xxx.86 25 19001 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.26 xxx.xxx.xxx.86 25 19002 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.26 xxx.xxx.xxx.86 25 19003 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.26 xxx.xxx.xxx.86 25 19004 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.4 xxx.xxx.xxx.85 443 19005 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.4 xxx.xxx.xxx.85 222 19006 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.26 xxx.xxx.xxx.86 222 19007 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19007 dgram udp nowait/0 nobody /usr/bin/nc nc -u -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19008 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19008 dgram udp nowait/0 nobody /usr/bin/nc nc -u -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19009 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19009 dgram udp nowait/0 nobody /usr/bin/nc nc -u -w 2000 10.0.0.4 xxx.xxx.xxx.85 54 19010 stream tcp nowait/0 nobody /usr/bin/nc nc -w 2000 10.0.0.115 xxx.xxx.xxx.91 80 -
On my pfSense:
/usr/bin/nc -w 2000 10.0.0.26 205.126.89.86 25
nc: port number invalid: 205.126.89.86
Also when I tried to match up the nc command in inetd.conf against the FreeBSD man page for nc it seemed to me that the command didn't match the template in the man page.
I'm running 2.0-RC1-IPv6 (i386)
built on Sun Mar 20 02:20:38 EDT 2011 -
Thanks for investigating; I'll go ahead with the upgrade tonight and see if that changes anything. I haven't updated it yet, so hopefully it'll go smoothly.
-
Thanks for investigating; I'll go ahead with the upgrade tonight
Its probably a good thing to upgrade snapshot builds from time to time, espeially when you come across problems. I just want to be clear that I wasn't suggesting you upgrade. AFter the investigation I recently reported I fully expect the version I'm running would display similar symptoms to your system if I had a similar inetd.conf and I had traffic activating the nc entries in inetd.conf.
Do you have any idea what parts of your configuration are responsible for those nc entries in inetd.conf?
-
Right, I figured I'd upgrade anyway, no huge hopes that it'll solve this issue, but perhaps the something with nc changed?
I haven't done anything exotic, just set up some NAT port forwarding via Web Configurator, which I assume was what added those nc lines to inetd.conf.
-
Upgraded to
2.0-RC1 (i386)
built on Thu Mar 24 13:58:11 EDT 2011and disabled Snort for the time being. Thanks for the input so far; if it happens again I'll be sure to copy down the logs for more info before rebooting it.
-
I haven't done anything exotic, just set up some NAT port forwarding via Web Configurator, which I assume was what added those nc lines to inetd.conf.
I have a number of port forward rules defined in Firewall -> NAT, click on Port Forward tab and I don't have any nc entries in /etc/inetd.conf. Do you have a different type of port forward?
-
Can you tell me if you have any aliases referenced on port forward rules?
-
Yes, I have aliases defined for most of my firewall rules. For some aliases, I specified both the internal and external IP's.
Under Firewall->Aliases, I have a few entries similar to
Name | Values
mailserver | 10.0.0.4, xxx.xxx.xxx.85Then in Firewall->NAT I created rule(s) using the aliases, like:
WAN TCP * * xxx.xxx.xxx.85 25 (SMTP) mailserver 25 (SMTP) Mail Server -
Ok try with latest snapshot. I fixed an issue on the generated configs in the backend.
If you do not want to wait for next snapshot the change is https://rcs.pfsense.org/projects/pfsense/repos/mainline/commits/650b573bd8a435449178385a2d132f7f0002d309 -
OK, thanks! Other than upgrading my snapshot, should I remove/re-add my firewall rules? Delete the nc entries from inetd.conf?
-
For the time being, I've removed the aliases from my setup; once things are stable I'll turn them back on.
-
You should not do any changes to your firewal other than upgrade.
It would be good to give feedback if it solves your issues since its better fix it now rather than go through the hoops again after 2.0
-
OK. Updated, with aliases enabled, so far so good over the weekend. Out of town this week so hopefully it will behave; I'll report on hopefully successful results then.
-
Also of note, after the upgrade, the only line in /etc/inetd.conf is for tftp-proxy.
[edit: never mind, the file of interest is /var/etc/inetd.conf, which does contain the nc lines.]