128MB RAM no longer enough?

irvingpop

I have a mix of ALIX-based pfSense firewalls all over the US. 75% are the older ALIX model (2D3 I think) which have 128MB RAM, the rest are newer 2D13 w/ 256MB RAM.

The newer models have been upgrading to pfSense 2.0 (and now 2.0.1) with no problem, performance is good and the systems are stable.

The older ones are having lots of problems since upgrading to pfSense 2.0. Memory usage is typically 75%, with regular log entries about php being killed due to "out of swap space" (pretty sure there's no swap on these systems). Occasionally something more critical will get killed, like bsnmpd.

Systems aren't heavily loaded or doing anything extraordinary, just basic firewalling and PtP VPN tunnels (services running: bsnmpd, dhcpd, miniupnpd, ntpd, racoon - all lightly loaded).

The 2D3 systems have also exhibited other strangeness, even as of 2.0.1:
* Issue where the gettytab isn't being properly generated: http://forum.pfsense.org/index.php?topic=21765.0
* Issue where dhcpleases pidfile doesn't exist: http://forum.pfsense.org/index.php/topic,33240.msg219511.html#msg219511

So, what's the (nearly) 2012 consensus of the group? Is it time to toss out the older (ALIX 2D3 and equivalent) systems, or is there any ways to squeeze a few more years out of them?

podilarius

Anything (1.2.3 and on) that had only 128MB of memory, I always had problems with aside from just basic routing.

I would start a hardware refresh … even if there is something that can be done, it will only temporary as 2.1 will hit by 2nd quarter of next year. It will hopefully be running FreeBSD 9.

You could probably load 1.2.3 back and restore an old config.

Up to you though.

wallabybob

I have a couple of pfSense firewalls with 256MB RAM. The more lightly loaded one has two active interfaces (one wired, one wireless) and was recently successfully upgraded from 2.0 to 2.0.1. Top shows:```

last pid: 44659; load averages: 0.09, 0.09, 0.08 up 0+22:25:01 19:32:11
47 processes: 1 running, 46 sleeping
CPU: % user, % nice, % system, % interrupt, % idle
Mem: 45M Active, 51M Inact, 23M Wired, 32M Buf, 95M Free
Swap: 260M Total, 260M Free

I suspect I might be able to run this configuration on 128MB RAM but haven't tried it.

If you are running pfSense embedded then there is no swap space so all applications have to fit into memory together. You might be able to reduce the memory requirements by reducing the number of applications. I had the bandwidthd package installed on this system a while ago then removed it well before I upgraded. But I noticed the bandwidthd program is still running even though I removed the package some number of reboots ago. You may have some programs you don't need that could be removed to free up some memory space.

@irvingpop:

> Systems aren't heavily loaded or doing anything extraordinary, just basic firewalling and PtP VPN tunnels  (services running:  bsnmpd, dhcpd, miniupnpd, ntpd, racoon - all lightly loaded).

On systems with no swap the memory requirement can be a much more significant factor than CPU use. If you don't need much swap you might be able to change to use the "full embedded" version of pfSense. People wouldn't normally configure a swap file on a compact flash card but if it gets light intermittent use it might not do too much damage. (Compact Flash cards usually have a fairly limited number of write cycles. It might be possible to get a "disk module" in compact flash card size with considerably more write cycles than Compact Flash cards really designed for digital cameras.) Depending on the number of 128MB Alix boards you have it may not be worth the time investment to make them work reliably.

As already suggested 128MB is pretty much a bare minimum for a useful pfSense and future releases are very unlikely to reduce that requirement.

irvingpop

Guys, thanks for the feedback so far.

Looking at my memory usage on a 128MB ALIX running pfSense 2.0.1, processes sorted by resident memory (res):

last pid: 50789; load averages: 0.22, 0.08, 0.03 up 1+00:52:15 07:08:44
92 processes: 2 running, 77 sleeping, 13 waiting
CPU: 0.8% user, 1.5% nice, 0.0% system, 1.6% interrupt, 97.7% idle
Mem: 54M Active, 10M Inact, 30M Wired, 1112K Cache, 21M Buf, 12M Free
Swap:

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
31165 root 1 76 0 27996K 13176K accept 0:01 0.00% php
45369 root 1 76 0 29020K 12464K accept 0:03 0.00% php
20677 root 1 76 0 29020K 12456K accept 0:03 0.00% php
50043 root 1 45 0 26972K 6108K accept 0:00 0.00% php
5056 root 1 44 0 8488K 5944K select 0:31 0.00% bsnmpd
33451 root 1 45 0 26972K 5908K wait 0:01 0.00% php
33875 root 1 51 0 26972K 5900K wait 0:00 0.00% php
56716 dhcpd 1 44 0 8436K 5020K select 0:00 0.00% dhcpd
58698 root 1 64 20 6588K 4008K kqread 0:02 0.00% lighttpd
19333 root 1 44 0 7992K 2984K select 0:00 0.00% sshd
52706 root 1 44 0 5116K 2596K select 0:05 0.00% openvpn
63994 root 1 44 0 5988K 2296K select 0:34 0.00% racoon
25823 root 1 50 0 3672K 2000K pause 0:00 0.00% tcsh
27008 root 1 44 0 5272K 1812K select 0:00 0.00% sshd
20491 root 1 44 0 5912K 1656K bpf 0:05 0.00% tcpdump
26670 root 1 44 0 3712K 1508K RUN 0:00 0.00% top
7594 root 1 44 0 4948K 1324K select 0:08 0.00% syslogd
48373 root 1 44 0 4496K 1152K piperd 0:00 0.00% rrdtool
20410 root 1 76 0 3656K 1012K wait 0:00 0.00% sh
19534 root 1 76 0 3656K 1008K wait 0:00 0.00% sh
60604 root 1 76 20 3656K 1004K wait 0:01 0.00% sh
24600 root 1 44 0 3436K 932K select 0:00 0.00% inetd

According to this, PHP is using an aggregate of 56MB of RAM which seems a bit disproportionate to me.

Compare to identical hardware running pfSense 1.2.2:

last pid: 50176; load averages: 0.05, 0.03, 0.05 up 245+03:09:03 15:14:11
77 processes: 2 running, 59 sleeping, 2 zombie, 14 waiting
CPU states: 0.0% user, 0.0% nice, 0.4% system, 0.8% interrupt, 98.8% idle
Mem: 41M Active, 4900K Inact, 32M Wired, 52K Cache, 22M Buf, 38M Free
Swap:

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
533 root 1 4 0 41692K 14004K accept 0:16 0.00% php
348 root 1 -58 0 15936K 12456K bpf 9:46 0.00% tcpdump
63567 root 1 44 0 7116K 5448K select 213:33 0.00% bsnmpd
531 root 1 8 0 39644K 3440K wait 0:00 0.00% php
529 root 1 4 0 6124K 3340K kqread 15:33 0.00% lighttpd
4370 root 1 44 0 4892K 3148K select 0:24 0.00% openvpn
656 root 1 44 0 5720K 2828K select 679:31 0.00% racoon
50144 root 1 44 0 7736K 2776K select 0:00 0.00% sshd
50169 root 1 20 0 3500K 1940K pause 0:00 0.00% tcsh
300 root 1 44 0 5020K 1876K select 18:20 0.00% sshd
873 dhcpd 1 44 0 3132K 1548K select 3:38 0.00% dhcpd
50172 root 1 44 0 3516K 1456K RUN 0:00 0.00% top
21115 root 1 8 0 3488K 1216K wait 0:00 0.00% login
50150 root 1 8 0 3484K 1128K wait 0:00 0.00% sh
21117 root 1 5 0 3484K 1124K ttyin 0:00 0.00% sh
50149 root 1 8 0 3484K 1088K wait 0:00 0.00% sh
21116 root 1 8 0 3484K 1088K wait 0:00 0.00% sh
595 nobody 1 44 0 3132K 1028K select 34:55 0.00% dnsmasq
481 proxy 1 4 0 3236K 988K kqread 0:02 0.00% pftpx
8462 root 1 -8 0 3212K 984K piperd 0:00 0.00% cron
902 root 1 8 0 3212K 920K nanslp 2:36 0.00% cron
63396 root 1 44 0 3236K 912K select 1:23 0.00% syslogd
391 _ntp 1 44 0 3132K 896K select 3:26 0.00% ntpd
402 root 1 44 0 3132K 872K select 0:43 0.00% ntpd

So how did PHP become so piggy in pfSense 2? Why is there 6 PHP processes running when previously there was 2? Any ways to reduce the memory footprint of PHP?

wallabybob

It might help to reduce the number of web configurator processes: System -> Advanced, Admin Access tab, Max processes entry

irvingpop

I had tried that earlier (changing web configurator processes to 1) but it didn't seem to help. Apparently the extra PHP processes were just hanging around. After a reboot (or killall php) I'm down to 2 PHP processes, just like the 1.2.2 boxes. Memory down to 63% on the 128MB box.

It's still not clear how I got to 6 PHP processes . I'll continue to monitor.

wallabybob

@irvingpop:

After a reboot (or killall php) I'm down to 2 PHP processes, just like the 1.2.2 boxes.

On the reboot did you get any "out of swap" reports on startup?

irvingpop

Haven't tried reboot yet (just killall php with very promising results). I'll do that tomorrow when the unit isn't handling traffic and let you know.

Thanks!

irvingpop

@wallabybob:

On the reboot did you get any "out of swap" reports on startup?

Yes, I'm still getting one PHP process killed during bootup on the 128MB system, but not on the 256MB box:

Trying to mount root from ufs:/dev/ufs/pfsense1
glxsb0: <amd geode="" lx="" security="" block="" (aes-128-cbc,="" rng)="">mem 0xefff4000-0xefff7fff irq 9 at device 1.2 on pci0
pid 260 (php), uid 0, was killed: out of swap space
vr2: link state changed to DOWN

This would explain why the 128MB box shows various oddities on bootup (gettytab gets regenerated incompletely, miniupnpd.conf doesn't get generated). But after it boots and settles down, memory usage is approx 50% (plenty of room to spare, just 2 PHP processes running).

I'm getting close to a decent bug report here. Any way to debug the bootup process in more detail?</amd>

irvingpop

Update:

Disabling APC allows the system to successfully boot without killing any processes.

I changed the RAM threshold from 96 to 135 MB on line:249 of /etc/rc.php_ini_setup:

if [  "$RAM" -gt 135 ]; then

        /bin/cat >>/usr/local/lib/php.ini < <eof<br>; APC Settings
apc.enabled="1"
apc.enable_cli="0"
apc.shm_size="${APCSHMEMSIZE}"

EOF

else
        LOWMEM="TRUE"
        echo ">>> WARNING!  under 128 megabytes of ram detected.  Not enabling APC."
        echo ">>> WARNING!  under 128 megabytes of ram detected.  Not enabling APC." | /usr/bin/logger -p daemon.info -i -t r
c.php_ini_setup
fi

        /bin/cat >>/usr/local/lib/php.ini <</eof<br>

Anyone else with 128MB pfSense 2 boxes: Can you please reproduce and test this fix?

Thanks!

pfsense_fan009

I have an 2d13 and I have also those errors:

"out of swap space"

caused bij php, any solution to this ?

irvingpop

@pfsense_fan009:

I have an 2d13 and I have also those errors:

"out of swap space"

caused bij php, any solution to this ?

Your issues are most likely software/configuration and not related to a hardware limitation, because ALIX 2D13 has 256 MB of RAM, which is more than enough for normal pfSense operation. I suggest you start a new topic in the General support forum and post your details there.

cheekymonkey

Thanks very much for posting this, I am using a WRAP and having all sorts of problems till I found this.