A few days of uptime then slow to a halt..
-
Hello pfsense dev team.
I'm the admin for a small network of around 15-30 simultaneous clients and we have a little problem with the firewall. The machine has an uptime of about 3-4 days, then suddenly the system slows to a halt. Currently, I'm at work so I can't post it, but I have a copy of the RRD graph for this problem. I'll update this post with the image once I get home.
Anyhow, network traffic works okay as long as you don't try to surf to any web pages. I can still ping, I can send IM messages, and such without any greater latency, but when I try to surf to the firewall for settings, it times out and is completely unable to serve my web request. It should be noted that the ping response from the pfsense server is <1ms during this time.
The only chance to solve this problem would be a hard reboot and after that it works smoothly.
Any idea what might be causing this problem?
Henrik
-
Hardware releated, I bet.
-
Unfortunately, that might be the case… I'm thinking a lack of memory, it has reported that it is low some times, and it's always in the upper regions of the memory use. However, I've removed as much overhead as possible by turning off packages that may cause the problems. I'm quite dependant on ntop though, which I'd suppose is one of the bigger memory hogs in the whole system.
Henrik
-
Also make sure that you are not going over your max states allocation.
-
Hmm, I currently have it set to 10000 states, and I have seen it go above that before. Would increasing this possibly help me out, or is that also cause of a big memory usage? It looks like my CPU usage is at 100-ish percent when this problem occurs…
-
If you see that you are hitting the limit then the firewall will stop accepting new connections until a state is removed. Increasing the state table will help this but you are looking at around 1KB per state if my memory serves me correctly.
-
Okay, then I suppose adding a couple more states will not be that much a memory drain. Just to be sure though, if I do hit the roof of the states, will it stop accepting inbound connections to the pfsense webgui as well? It's quite painful trying to admin the whole thing when I can't get through to the webgui.
I'm inclined to think it's a CPU problem, but I'm hoping not… I don't see why it'd suddenly spike for no apparent reason after a couple of days.
Thanks for the quick and great reply, by the way. Better than most tech supports I've been in contact with :)
-
Yes, I imagine it will stop the GUI as well. Everything will stop working somewhat.
Might want to try 15,000 or 20,000 states if you have enough memory.
-
Btw, newer snapshots have a states rrd graph. It should log if you reach the limit. Maybe view that graph to see if this really is the case when the problems occur. You even can view it after reboot (if you are not running the embedded version or livecd as you have to do a clean reboot from gui or shell to write the rrds to disk).