Memory Leak
-
States almost never go above 2000. They're pretty consistent too. There aren't spikes in the state table size when the problem occurs.
-
It's also interesting to note that when the firewall starts having issues, neither the processor usage nor the memory usage is above 50%.
-
kejianshi,
The dansguardian processes were in a post from timthetortoise who says he's not having issues. I (Jason, the OP) am not running dansguardian or ClamAV, but I am having a memory leak issue. My process lists is in the original post.
Thanks for your help though!
-
HAHA - Yea. I saw that after I posted and realized my mistake and deleted it… I was hoping no one saw it. You caught me!
Question. Is this system upgraded or is it a fresh clean install?
I've been told repeatedly that that shouldn't matter, but I'm not sure how true that is.
-
The system was upgraded to 2.1–not a clean install. The problem existed both on 2.0.2 and continues on 2.1.
-
Is a new clean install out of the question?
-
At this point, nothing is out of the question. Of course a new install is a pain, but we have to solve the problem. The last thing we did was turn off the traffic shaper all together. The folks behind this firewall don't seem to consume their Internet pipe (10Mb symmetrical), so the traffic shaper isn't that big of a deal. Now I'm just waiting for the memory leak to start again. Sometimes it takes a few days; others it takes a month or more.
We're in a virtualized environment, so I've considered setting up a separate gateway in a high-availability configuration. It'd give me somewhere to move traffic when the problem occurs assuming the memory leak doesn't show up on both firewalls at the same time. That's a few hours work and I have to make sure I have extra Internet IPs available, but it would reduce the risk of having to incur a middle-of-the-workday outage.
-
"traffic shaper" - Is that no longer running?
-
The traffic shaper is no longer running.
-
I'd wipe the drive and install clean. Unless your rules are special in some way, I would not even do a restore of settings. I'd redo them by hand to be sure you didn't import trouble. Is this a 64 or 32 bit install? Because I'd also prefer 32bit.
-
For your information, I think this problem has followed us through several iterations of pfSense from 1.2.3 (i think) on an ALIX board to 2.0.1 on a PC to 2.0.2 on a virtual server to 2.1 on the same server. I do think the same XML config has been ported through all of those environments–both 32- and 64-bit. I guess I'm in agreement that a scratch rebuild might be the next good thing to do.
It's an amd64 install today. I'm curious, why do you prefer 32-bit? Does anyone else have thoughts on 32-bit vs 64-bit?
-
Because with less than 4GB ram you don't NEED 64bit version and I've seen more issues with 64 than 32. Some would say its my imagination, but I don't think so. I think the 32bit version is just working better under VMs. I've experienced that anyway. Seems others have also, but thats more of something I've noticed than a scientific study of the issue.
-
I have a customer on a 64 bit version and it behaves stupendously. Not one issue! :)
-
We also have at least six other 64-bit installs that are working flawlessly. A couple of them have fairly complicated configurations with ~10 interfaces, ~20 IPsec tunnels, 100s of firewall & NAT rules, traffic shaping, etc. With all of that, they're still able to sustain 100Mb/s throughput easily. I'm curious if there would be performance degradation moving from 64-bit to 32-bit.
-
The problems I personally experienced were on 64bit version on ESXi 5.
Same install, nothing else modified and it worked fine. Tried several times.
I couldn't solve it, so I just stayed with 32bit version.
Sounds like your hypervisor isn't same, so probably depends.
Also, the problems I had were mainly either with NTP or with nothing working well with 5 or more interfaces.
There are just too many variables in hardware and network to say anthing for sure for everybody, but as a rule, if I don't need 64bit, I use 32bit where I can. As for it being a point of contention for this or that person, I say do whatever floats your boat and works for you.
There won't be any performance losses. 64bit is just able to address more RAM. Thats all.
-
Jason: Did you ever solve this?
I ended up just re-installing the firmware over the original install via the firmware page and that seems to have solved it for me here.
My other watchguard "customer" had his unit do the same thing starting last Wednesday and build up to over 30% before we caught it. Restarted Apinger and it has been normal since…
-
Since we've been through at least four firmware updates on this firewall and the problem still persists, I'm doubting a firmware reinstall will fix the problem. I'm also pretty sure apinger is not the problem as I believe that would cause memory usage to spike in user space. I'm seeing wired kernel memory balloon.
My plan at this point is to build another near-identical firewall through the web gui, download the XML config from the new and old ones, and compare. Maybe that will reveal some anomaly. This XML has been pulled up through the 1.2 versions to 2.1. Maybe something bad slipped through.
-
My memory does exactly the same thing. Looks like utilization climbs and climbs and then it hits about 90%, teeters there for a while and then will drop back down to about 30% all by its self, with no reset, reboot or anything and no ill effects and then the cycle starts again. I don't worry about it.
-
Im actually still seeing this intermittently on a couple of boxes. Though last time it happened to me here it actually used up all memory before it reset itself.
But since Ive reset the webgui from a console it has not occurred. Re-booting the box never solved it. In fact it would simply do the same right after the re-boot.
-
When the memory usage is high, have you checked the process list to see what is using the RAM?
-
I'll chuck this in this mix as its running in a VM.
Brute force
http://www.run-virtual.com/?p=695then how to reset the VM password which might be sat in a data centre and doesnt get checked often
http://blog.geekdiary.me/?p=21because it might be running auto failover type of software like this:
http://www.veeam.com/virtual-machine-backup-solution-free.html -
When the memory usage is high, have you checked the process list to see what is using the RAM?
Just this- https://forum.pfsense.org/index.php/topic,67703.msg370496.html#msg370496
It hasn't happened since the webgui reset from the console so I cant look now. Im watching for it.
-
After some testing it seems the LCDProc package is causing this for me.
When I do a webgui reset the display goes to a screen that shows Cli: 0 Scr: 0 with no backlight. I didn't notice it till days after my initial webgui reset. Once I restarted the LCD package the memory leak started again. Reset tonight and Im going to leave the package in the inop state for a few weeks.
Running the dev package.