pfSense running out of memory and locking up
-
Interestingly, the log messages are slightly different from last time. @stephenw10 , would these indicate that your suggested fix might indeed be the problem:
Jul 5 01:14:36 pfSense upsmon[90689]: Communications with UPS ups established Jul 5 01:14:36 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:14:36 pfSense upsmon[16358]: Can't invoke wall: Cannot allocate memory Jul 5 01:14:44 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:15:30 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:17:08 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:18:45 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:19:11 pfSense kernel: vm_thread_new: kstack allocation failed Jul 5 01:19:44 pfSense upsd[93206]: Data for UPS [ups] is stale - check driver Jul 5 01:19:46 pfSense upsd[93206]: UPS [ups] data is no longer stale Jul 5 01:20:31 pfSense upsd[93206]: Data for UPS [ups] is stale - check driverAnd when I was trying to reboot from the web GUI:
Jul 6 11:08:57 pfSense php-fpm[365]: /diag_reboot.php: Stopping all packages. Jul 6 11:08:57 pfSense kernel: vm_thread_new: kstack allocation failed Jul 6 11:08:57 pfSense php-fpm[365]: /diag_reboot.php: The command '/usr/local/etc/rc.d/nut.sh stop' returned exit code '2', the output was 'stopping NUT /usr/local/etc/rc.d/nut.sh: Cannot fork: Cannot a llocate memory' Jul 6 11:08:57 pfSense kernel: vm_thread_new: kstack allocation failed Jul 6 11:08:58 pfSense kernel: vm_thread_new: kstack allocation failed Jul 6 11:08:58 pfSense php-fpm[365]: /diag_reboot.php: The command 'nohup /etc/rc.reboot > /dev/null 2>&1 &' returned exit code '-1', the output was ''No logs are present from when I was trying to reboot from the console, but I was seeing the same kind of messages echoed. It couldn't spawn any of the commands I was issuing.
~Dan
-
My bet is the upsd driver. The message says your system is running out of kstack memory. This is special, reserved memory for the kernel stack. I think that is a fixed allocated chunk of memory, so it very well may not show up as being "consumed" in the Dashboard memory consumption indicator. Or stated another way, it is probably part of the base system memory area and the entire block is accounted for once and processes use small bits of that preallocated chunk when running.
From what I understand from the limited research I did, that block is not expandable. So when a process consumes enough of it, that will kill the kernel because other processes that need some space can't get it.
My guess is the upsd driver is crashing in some fashion (or some portion of it is crashing) and leaking kstack memory each time it crashes. After enough days of crashing, all of the kstack memory is consumed via those "leaks".
I know it can be dangerous, especially if power at your location is flaky, but I would test with nut removed and the UPS unplugged from the USB port to see if the kstack errors go away. It will take several days to know.
-
@DannyBoy2k said in pfSense running out of memory and locking up:
from the command line as it appears you did
I have to deceive you here.
I copied with my mouse the follow part of the GUI dashboard :
I copied the text, and used the format tool
to make it readable for humans.
( and thus using 190 bytes storage in stead of several Kilo of bytes for the image)Btw : command line version :
ls -al /usr/local/pkgwill list you what you have on your pfSense.
It's a bit messy, but can be useful.The directory list doesn't show what is up to date, actually activated etc.
-
@Gertjan:
Are you running nut on an SG-3100? I tried a number of times to get it running on an SG-3100 with a CyberPower UPS and was never successful in getting the UPS to be recognized. -
-
@Gertjan said in pfSense running out of memory and locking up:
@bmeeks said in pfSense running out of memory and locking up:
You mean @DannyBoy2k
No, I was asking you since you mentioned nut running okay. Not trying to change the thread topic, but wondering if the SG-3100 due to its ARM architecture acts weird with some peripherals.
@DannyBoy2k has it sort of running, but with the serious issue he posted about.
-
@bmeeks , yes, I was able to get nut running with the CyperPower just using the usb driver. It's just that is seems to occasionally (maybe once a day) need to restart/reconnect to it.
~Dan
-
@bmeeks said in pfSense running out of memory and locking up:
No, I was asking you
I'm using NUT (pfSense) and a bare bone Intel PC's from the last decade - APC UPS's only using their "USB" ports.
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@DannyBoy2k said in pfSense running out of memory and locking up:
@bmeeks , yes, I was able to get nut running with the CyperPower just using the usb driver. It's just that is seems to occasionally (maybe once a day) need to restart/reconnect to it.
~Dan
Okay, but it appears to not be running well. Should not disconnect. I was never able to get it to work, so have that firewall for now running on the UPS but "blind" to battery exhaustion. Not ideal!
Mentioned this in your thread to say perhaps there are issues with the USB driver for UPS/nut that manifest themselves in various ways.
-
@Gertjan said in pfSense running out of memory and locking up:
@bmeeks said in pfSense running out of memory and locking up:
No, I was asking you
I'm using NUT (pfSense) and a bare bone Intel PC's from the last decade - APC UPS's only using their "USB" ports.
Ah! I've never had issues with my Intel-based firewalls and have used both APC and other UPS boxes. The SG-3100 was the first one to ever kick my butt! It's also the first ARM architecture firewall I've encountered.
-
@bmeeks , thank you for the thoughts. I posted a message in the pfsense packages Category to see if it leads anywhere:
https://forum.netgate.com/topic/155094/possible-memory-leak-in-nut-package~Dan
-
Review my first post in this thread where I mention the kstack memory allocation error. My bet is still on the USB driver for the UPS being the problem. If you can, disable that driver completely and see if stability returns. Might take a month to be sure since you went as far as 28 or 29 days between lockups.
-
@bmeeks said in pfSense running out of memory and locking up:
My guess is the upsd driver is crashing in some fashion (or some portion of it is crashing) and leaking kstack memory each time it crashes. After enough days of crashing, all of the kstack memory is consumed via those "leaks".
I know it can be dangerous, especially if power at your location is flaky, but I would test with nut removed and the UPS unplugged from the USB port to see if the kstack errors go away. It will take several days to know.
Hello!
I use a pi running upsd and netgates attaching to it with upsmon (Remote NUT Server). This could be a workaround while exploring local upsd issues.
John
-
@bmeeks said in pfSense running out of memory and locking up:
The SG-3100 was the first one to ever kick my butt!
Hello!
With a sample size of one...
https://forum.netgate.com/topic/154674/nut-and-apc-smart-ups-750-rm-usb
John
-
@serbus said in pfSense running out of memory and locking up:
@bmeeks said in pfSense running out of memory and locking up:
The SG-3100 was the first one to ever kick my butt!
Hello!
With a sample size of one...
https://forum.netgate.com/topic/154674/nut-and-apc-smart-ups-750-rm-usb
John
I tried a number of things with that SG-3100, and never did get the UPS properly recognized. It is something to do with device file permissions I suspect. I did not want to dive into a bunch of repetitive reboots and tinkering with the base OS at the time. I've never had any issues at all with either nut or apcupsd on several iterations of Intel-based hardware with pfSense. That particular SG-3100 is currently serving duty as a church firewall.
I found another post or two here in the past about assigning specific permissions to one or more of the /dev psuedo files/directories that get created for peripherals, but as I said above I did not want to get off into those weeds.
The ARM architecture of the SG-1000, SG-1100 and SG-3100 appliances has turned out to be shall we just say "interesting" ...
. Lots ofSome legacy C source code programs that run fine on Intel hardware will crash on the ARM stuff due to memory alignment errors. Other subtle differences in the internal architecture can also contribute to "weirdness" with some software on the ARM devices. -
If there is some memory leak I would expect to be able to see it somewhere before it actually locks up.
The first place I've look in the Monitoring Graphs for System - Memory. When we have seen bugs like that before you can see the usage ramp up there.
Steve
-
@stephenw10 , I am almost always at
7% of 2020 MiBwhen I log into the web GUI. I'm barely using the features of this box. A couple of VLANs, DNS, DHCP. That's about it. The only indication I've ever found of something being wrong are the systems logs I've pasted above.~Dan
-
Hmm. Well I guess if it is kernel memory that's harder to see.... try checking the output of
sysctl vm.kmem_map_free.First thing you will see is that on the 32bit arm system that is much smaller than other architectures so far easier to hit an issue. See if hat value decreases over time.
Steve
-
@stephenw10 , I explored the web GUI a bit more and found the Status: Monitoring section. This seems interesting, but I have no idea what it means. I mean, I can see that all free memory suddenly became unavailable, but no idea why.
I ran your suggested command:
[2.4.5-RELEASE][admin@pfSense.localdomain]/root: sysctl vm.kmem_map_free vm.kmem_map_free: 141295616I'll try to log in every now and again and continue to monitor.
~Dan
-
Here is a higher fidelity snapshot around the period of interest. It appears it just happened very suddenly, not ramping up over time.
~Dan
