128MB RAM no longer enough?



  • I have a mix of ALIX-based pfSense firewalls all over the US.   75% are the older ALIX model (2D3 I think) which have 128MB RAM,  the rest are newer 2D13 w/ 256MB RAM.

    The newer models have been upgrading to pfSense 2.0 (and now 2.0.1) with no problem,  performance is good and the systems are stable.

    The older ones are having lots of problems since upgrading to pfSense 2.0.   Memory usage is typically 75%, with regular log entries about php being killed due to "out of swap space"  (pretty sure there's no swap on these systems).  Occasionally something more critical will get killed, like bsnmpd.

    Systems aren't heavily loaded or doing anything extraordinary, just basic firewalling and PtP VPN tunnels  (services running:  bsnmpd, dhcpd, miniupnpd, ntpd, racoon - all lightly loaded).

    The 2D3 systems have also exhibited other strangeness, even as of 2.0.1:
     *  Issue where the gettytab isn't being properly generated:  http://forum.pfsense.org/index.php?topic=21765.0
     *  Issue where dhcpleases pidfile doesn't exist: http://forum.pfsense.org/index.php/topic,33240.msg219511.html#msg219511

    So, what's the (nearly) 2012 consensus of the group?   Is it time to toss out the older (ALIX 2D3 and equivalent) systems,  or is there any ways to squeeze a few more years out of them?



  • Anything (1.2.3 and on) that had only 128MB of memory, I always had problems with aside from just basic routing.

    I would start a hardware refresh … even if there is something that can be done, it will only temporary as 2.1 will hit by 2nd quarter of next year. It will hopefully be running FreeBSD 9.

    You could probably load 1.2.3 back and restore an old config.

    Up to you though.



  • I have a couple of pfSense firewalls with 256MB RAM. The more lightly loaded one has two active interfaces (one wired, one wireless) and was recently successfully upgraded from 2.0 to 2.0.1. Top shows:```

    last pid: 44659;  load averages:  0.09,  0.09,  0.08                                                up 0+22:25:01  19:32:11
    47 processes:  1 running, 46 sleeping
    CPU:    % user,    % nice,    % system,    % interrupt,    % idle
    Mem: 45M Active, 51M Inact, 23M Wired, 32M Buf, 95M Free
    Swap: 260M Total, 260M Free

    I suspect I might be able to run this configuration on 128MB RAM but haven't tried it.
    
    If you are running pfSense embedded then there is no swap space so all applications have to fit into memory together. You might be able to reduce the memory requirements by reducing the number of applications. I had the bandwidthd package installed on this system a while ago then removed it well before I upgraded. But I noticed the bandwidthd program is still running even though I removed the package some number of reboots ago. You may have some programs you don't need that could be removed to free up some memory space.
    
    @irvingpop:
    
    > Systems aren't heavily loaded or doing anything extraordinary, just basic firewalling and PtP VPN tunnels  (services running:  bsnmpd, dhcpd, miniupnpd, ntpd, racoon - all lightly loaded).
    
    On systems with no swap the memory requirement can be a much more significant factor than CPU use. If you don't need much swap you might be able to change to use the "full embedded" version of pfSense. People wouldn't normally configure a swap file on a compact flash card but if it gets light intermittent use it might not do too much damage. (Compact Flash cards usually have a fairly limited number of write cycles. It might be possible to get a "disk module" in compact flash card size with considerably more write cycles than Compact Flash cards really designed for digital cameras.) Depending on the number of 128MB Alix boards you have it may not be worth the time investment to make them work reliably.
    
    As already suggested 128MB is pretty much a bare minimum for a useful pfSense and future releases are very unlikely to reduce that requirement.


  • Guys, thanks for the feedback so far.

    Looking at my memory usage on a 128MB ALIX running pfSense 2.0.1,  processes sorted by resident memory (res):

    last pid: 50789;  load averages:  0.22,  0.08,  0.03                                                 up 1+00:52:15  07:08:44
    92 processes:  2 running, 77 sleeping, 13 waiting
    CPU:  0.8% user,  1.5% nice,  0.0% system,  1.6% interrupt, 97.7% idle
    Mem: 54M Active, 10M Inact, 30M Wired, 1112K Cache, 21M Buf, 12M Free
    Swap:

    PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME    CPU COMMAND
    31165 root        1  76    0 27996K 13176K accept   0:01  0.00% php
    45369 root        1  76    0 29020K 12464K accept   0:03  0.00% php
    20677 root        1  76    0 29020K 12456K accept   0:03  0.00% php
    50043 root        1  45    0 26972K  6108K accept   0:00  0.00% php
    5056 root        1  44    0  8488K  5944K select   0:31  0.00% bsnmpd
    33451 root        1  45    0 26972K  5908K wait     0:01  0.00% php
    33875 root        1  51    0 26972K  5900K wait     0:00  0.00% php
    56716 dhcpd       1  44    0  8436K  5020K select   0:00  0.00% dhcpd
    58698 root        1  64   20  6588K  4008K kqread   0:02  0.00% lighttpd
    19333 root        1  44    0  7992K  2984K select   0:00  0.00% sshd
    52706 root        1  44    0  5116K  2596K select   0:05  0.00% openvpn
    63994 root        1  44    0  5988K  2296K select   0:34  0.00% racoon
    25823 root        1  50    0  3672K  2000K pause    0:00  0.00% tcsh
    27008 root        1  44    0  5272K  1812K select   0:00  0.00% sshd
    20491 root        1  44    0  5912K  1656K bpf      0:05  0.00% tcpdump
    26670 root        1  44    0  3712K  1508K RUN      0:00  0.00% top
    7594 root        1  44    0  4948K  1324K select   0:08  0.00% syslogd
    48373 root        1  44    0  4496K  1152K piperd   0:00  0.00% rrdtool
    20410 root        1  76    0  3656K  1012K wait     0:00  0.00% sh
    19534 root        1  76    0  3656K  1008K wait     0:00  0.00% sh
    60604 root        1  76   20  3656K  1004K wait     0:01  0.00% sh
    24600 root        1  44    0  3436K   932K select   0:00  0.00% inetd

    According to this,  PHP is using an aggregate of 56MB of RAM  which seems a bit disproportionate to me.

    Compare to identical hardware running pfSense 1.2.2:

    last pid: 50176;  load averages:  0.05,  0.03,  0.05                                               up 245+03:09:03 15:14:11
    77 processes:  2 running, 59 sleeping, 2 zombie, 14 waiting
    CPU states:  0.0% user,  0.0% nice,  0.4% system,  0.8% interrupt, 98.8% idle
    Mem: 41M Active, 4900K Inact, 32M Wired, 52K Cache, 22M Buf, 38M Free
    Swap:

    PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME    CPU COMMAND
     533 root        1   4    0 41692K 14004K accept   0:16  0.00% php
     348 root        1 -58    0 15936K 12456K bpf      9:46  0.00% tcpdump
    63567 root        1  44    0  7116K  5448K select 213:33  0.00% bsnmpd
     531 root        1   8    0 39644K  3440K wait     0:00  0.00% php
     529 root        1   4    0  6124K  3340K kqread  15:33  0.00% lighttpd
    4370 root        1  44    0  4892K  3148K select   0:24  0.00% openvpn
     656 root        1  44    0  5720K  2828K select 679:31  0.00% racoon
    50144 root        1  44    0  7736K  2776K select   0:00  0.00% sshd
    50169 root        1  20    0  3500K  1940K pause    0:00  0.00% tcsh
     300 root        1  44    0  5020K  1876K select  18:20  0.00% sshd
     873 dhcpd       1  44    0  3132K  1548K select   3:38  0.00% dhcpd
    50172 root        1  44    0  3516K  1456K RUN      0:00  0.00% top
    21115 root        1   8    0  3488K  1216K wait     0:00  0.00% login
    50150 root        1   8    0  3484K  1128K wait     0:00  0.00% sh
    21117 root        1   5    0  3484K  1124K ttyin    0:00  0.00% sh
    50149 root        1   8    0  3484K  1088K wait     0:00  0.00% sh
    21116 root        1   8    0  3484K  1088K wait     0:00  0.00% sh
     595 nobody      1  44    0  3132K  1028K select  34:55  0.00% dnsmasq
     481 proxy       1   4    0  3236K   988K kqread   0:02  0.00% pftpx
    8462 root        1  -8    0  3212K   984K piperd   0:00  0.00% cron
     902 root        1   8    0  3212K   920K nanslp   2:36  0.00% cron
    63396 root        1  44    0  3236K   912K select   1:23  0.00% syslogd
     391 _ntp        1  44    0  3132K   896K select   3:26  0.00% ntpd
     402 root        1  44    0  3132K   872K select   0:43  0.00% ntpd

    So how did PHP become so piggy in pfSense 2?  Why is there 6 PHP processes running when previously there was 2?  Any ways to reduce the memory footprint of PHP?



  • It might help to reduce the number of web configurator processes: System -> Advanced, Admin Access tab, Max processes entry



  • I had tried that earlier (changing web configurator processes to 1)  but it didn't seem to help.  Apparently the extra PHP processes were just hanging around.  After a reboot (or killall php) I'm down to 2 PHP processes,  just like the 1.2.2 boxes.  Memory down to 63% on the 128MB box.

    It's still not clear how I got to 6 PHP processes .    I'll continue to monitor.



  • @irvingpop:

    After a reboot (or killall php) I'm down to 2 PHP processes,  just like the 1.2.2 boxes.

    On the reboot did you get any "out of swap" reports on startup?



  • Haven't tried reboot yet (just killall php with very promising results).    I'll do that tomorrow when the unit isn't handling traffic and let you know.

    Thanks!



  • @wallabybob:

    On the reboot did you get any "out of swap" reports on startup?

    Yes,  I'm still getting one PHP process killed during bootup on the 128MB system,  but not on the 256MB box:

    Trying to mount root from ufs:/dev/ufs/pfsense1
    glxsb0: <amd geode="" lx="" security="" block="" (aes-128-cbc,="" rng)="">mem 0xefff4000-0xefff7fff irq 9 at device 1.2 on pci0
    pid 260 (php), uid 0, was killed: out of swap space
    vr2: link state changed to DOWN

    This would explain why the 128MB box shows various oddities on bootup (gettytab gets regenerated incompletely,  miniupnpd.conf doesn't get generated).  But after it boots and settles down,  memory usage is approx 50%  (plenty of room to spare,  just 2 PHP processes running).

    I'm getting close to a decent bug report here.  Any way to debug the bootup process in more detail?</amd>



  • Update:

    Disabling APC allows the system to successfully boot without killing any processes.

    I changed the RAM threshold from 96 to 135 MB on line:249 of /etc/rc.php_ini_setup:

    if [  "$RAM" -gt 135 ]; then
    
            /bin/cat >>/usr/local/lib/php.ini < <eof<br>; APC Settings
    apc.enabled="1"
    apc.enable_cli="0"
    apc.shm_size="${APCSHMEMSIZE}"
    
    EOF
    
    else
            LOWMEM="TRUE"
            echo ">>> WARNING!  under 128 megabytes of ram detected.  Not enabling APC."
            echo ">>> WARNING!  under 128 megabytes of ram detected.  Not enabling APC." | /usr/bin/logger -p daemon.info -i -t r
    c.php_ini_setup
    fi
    
            /bin/cat >>/usr/local/lib/php.ini <</eof<br>
    

    Anyone else with 128MB pfSense 2 boxes:    Can you please reproduce and test this fix?

    Thanks!



  • I have an 2d13 and I have also those errors:

    "out of swap space"

    caused bij php, any solution to this ?



  • @pfsense_fan009:

    I have an 2d13 and I have also those errors:

    "out of swap space"

    caused bij php, any solution to this ?

    Your issues are most likely software/configuration and not related to a hardware limitation, because ALIX 2D13 has 256 MB of RAM, which is more than enough for normal pfSense operation.  I suggest you start a new topic in the General support forum and post your details there.



  • Thanks very much for posting this, I am using a WRAP and having all sorts of problems till I found this.


Log in to reply