Virtualized pfsense 2.1.5 - out of swap space



  • Hi,

    Im getting strange behaviour on my virtualized pfsense 2.1.5 (qemu 2.0.0+dfsg-2ubuntu1.9 and libvirt 1.2.2-0ubuntu13.1.7).

    I'm getting those messages:

    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(16): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(16): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(16): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(16): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(12): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(16): failed
    Jan  5 19:55:14 pfsense kernel: swap_pager_getswapspace(12): failed
    Jan  5 19:55:14 pfsense kernel: pid 22576 (php), uid 0, was killed: out of swap space
    
    

    Sometimes my radius seems to get shot in the head by this.

    But my machine has 1 GB of RAM and as normally it's using hardly any RAM, but look for yourself. I have no idea why the gaps.
    I experimented with squid, but it's not even installed any more.

    There also seem to be a lot of php processes:

    root    1792  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root    2763  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root    4023  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root    4056  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root    4457  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root    5368  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root    6851  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root    8260  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   10798  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   12744  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   14130  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   14656  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   16917  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   22529  0.0  1.5 147776 19848  ??  Is    7:55PM   0:00.06 /usr/local/bin/php
    root   24054  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   25582  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   27940  0.0  3.1 156224 39672  ??  I     7:55PM   0:01.52 /usr/local/bin/php
    root   31934  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   32892  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   34593  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   34632  0.0  1.6 147776 20176  ??  Is    8:16PM   0:00.06 /usr/local/bin/php
    root   34654  0.0  3.1 151872 39856  ??  I     8:16PM   0:01.25 /usr/local/bin/php
    root   36972  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   43906  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   46206  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   48809  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   49553  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   50333  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   53829  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   54447  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   54938  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   55750  0.0  2.9 160064 38020  ??  IN    4:30PM   0:00.16 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   57091  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   58447  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   58782  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   59429  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   59467  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   61061  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   62477  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   64928  0.0  2.9 160064 38116  ??  IN    7:45PM   0:00.13 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   65593  0.0  2.9 160064 38116  ??  IN    7:45PM   0:00.13 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   67485  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   73511  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   74311  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   74356  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   81334  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   81873  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   82196  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   82861  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   88261  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   89119  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   89412  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn HE_IPV6_TUNNELV6
    root   95295  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   97302  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    root   99899  0.0  0.0 160064     0  ??  IWN  -         0:00.00 /usr/local/bin/php -f /etc/rc.openvpn WAN_PPPOE
    




  • Swap space is usually on the hard drive and not in memory, right?



  • What makes you think I don't know the difference between physical and virtual memory?
    My experience actually shows me that swap is only getting rare, when phyical memory is getting rare, too.



  • how much swap space is available when you are getting these messages. (is it a bad error message / or is swap really full ?)

    it appears you are using 70% of RAM at the end of the RRD-graph. can you find out what processes are using it ? (using top or something)



  • That's current top:

    last pid: 98703;  load averages:  0.05,  0.17,  0.99                                                                                                                                                                  up 0+01:51:55  09:50:35
    45 processes:  1 running, 44 sleeping
    CPU:  0.0% user,  0.0% nice,  3.6% system,  0.4% interrupt, 96.1% idle
    Mem: 35M Active, 19M Inact, 128M Wired, 1856K Cache, 40M Buf, 1993M Free
    Swap: 512M Total, 81M Used, 431M Free, 15% Inuse

    Edit:
    I just gave the virtual machine 2 gig of RAM because I'm desperate.



  • Strange.  I run 2.1.5 in production under ESXi 5.5 w/2GB and 2.2 in my home lab under VirtualBox w/512MB.  Never had a swap error, and that's with multiple packages installed and acrobats on the trapeze and everything.



  • Something threw openvpn's reload script into a loop and consumed all your RAM. Does it recur after a reboot? The reboot (or just a "killall php") will clean up the mess it's gotten itself into. It's not clear what or why, can't say I've seen anything like that.


Log in to reply