Out of swap space error and Blinking Power LED on Alix2d13

cyboc

Yesterday I upgraded 5 identical Alix2d13 boxes to 2.0-RC1 (i386) built on Thu Mar 17 08:29:17 EDT 2011. On one of them LED #1 (the left one) kept blinking, even after rebooting. I finally fixed the problem by power cycling it. All of them have blinkled installed and enabled. I don't see any weird errors messages in the logs. There's no info in the Alix manual either. What could be the problem?

jimp

If that LED is blinking on 2.0 it's because there is a system notification, which should have been shown in the GUI up at the top of the screen. Could have been anything from generating SSH keys to a failed ruleset load.

cyboc

@jimp:

If that LED is blinking on 2.0 it's because there is a system notification, which should have been shown in the GUI up at the top of the screen. Could have been anything from generating SSH keys to a failed ruleset load.

I don't recall seeing the notification at the time but I'll remember to check next time I see the blinking LED.

cyboc

This happened again when I upgraded to yesterday's firmware (April 12). Upon rebooting the Alix, I got the flashing power LED.

There was no yellow ticker tape error message thingy in the web console but I did notice a couple of errors similar to this in the system log:

kernel: pid 252 (php), uid 0, was killed: out of swap space

Could this be the cause of the flashing LED?

Also, the "out of swap space" message is probably not good. When the Alix is idling, memory usage is below 50%. However, is it possible that during boot up the memory usage temporarily balloons to 100%? Is there any way to avoid this without removing rules and disabling services?

I have removed one unnecessary OpenVPN tunnel, disabled an unnecessary service and deleted 5 unnecessary firewall rules. Upon reboot, I don't see the "out of swap space" message and the power LED is now solid green.

cyboc

Okay, I'm pretty sure the "out of swap space" error on reboot is causing the blinking LED on my Alix. I've rebooted several times and the blinking LED only occurs when I see "out of swap space" in the system log.

I noticed that the vm.kmem_size setting in /boot/loader.conf is larger than the physical memory of 256 MB:

hw.ata.atapi_dma="0"
hw.ata.ata_dma="0"
loader_color="NO"
console=comconsole
autoboot_delay="5"
hw.ata.wc="0"
kern.ipc.nmbclusters="0"
beastie_disable="YES"
vm.kmem_size="435544320"
vm.kmem_size_max="535544320"

I tried overriding vm.kmem_size and vm.kmem_size_max in /boot/loader.conf.local to use a setting lower than 256 MB. With sysctl, I verified that the new settings survived. However, I still get the out of swap space error on some reboots.

Besides turning off some services and removing some firewall rules, is there anything I can do about this out of swap space error on a Alix during boot time or am I SOL?

cyboc

The tipping point seems to be having too many OpenVPN servers. With 5, I don't get the out of swap space error but when I add a sixth and then reboot, I will often see the error. :(

Any suggestions other than "upgrade your hardware"? I mean, is there any setting I can tune to cause less memory to be used at boot time?

cyboc

I forgot to mention, I am NOT running snort or any other optional packages, except for blinkled.

wallabybob

@cyboc:

Any suggestions other than "upgrade your hardware"? I mean, is there any setting I can tune to cause less memory to be used at boot time?

Since the problem seems to be related to the number of OpenVPN servers you have configured and the servers presumably are configured at boot time I suspect there is no easy answer. Perhaps some reordering of the startup might help but there is probably no easy way to come up with an optimum solution.

eri--

THere was a issue fixed lately which could cause memory starvation if you had dns hostnames in aliases.
Latest snapshots should have that fixed.

Otherwise check with others suggestions.