Backup of RRD fills /var on shutdown



  • Alix nanoBSD 256MB memory
    This is not a 2.2-specific problem, but putting here to see if anyone thinks a solution like this is worth having.
    I have a system with a bunch of VLANs (7 of them) and 2 WANs. The RRD data in /var/db/rrd is about 12MB. When the backup is happening at shutdown/reboot these files get converted into *.xml, which is much bigger - it generates 35MB of xml files. Then all the *.xml are put into a single tar in /cf/conf - that file is less than 2MB.
    During the creation of the *.xml there needs to be 35MB somewhere.
    /var memory disk runs out of space. I increased it to 80MB, but then my whole system runs out of real memory during shutdown while trying to write all the *.xml in /var/db/rrd
    It would be nice if there was a way to be generating each *.xml and piping it directly into the tar (not going through /var filesystem) - but I cannot see how to achieve this in /etc/rc.backup_rrd.sh
    Another way is to put the *.xml in a dir on a real file system that will have plenty of space. e.g. I made /backuptmp (in "/" file system - the real CF card). That means doing some extra writes to the CF card to manage the interim data, but it works with this code in /etc/rc.backup_rrd.sh

    #!/bin/sh
    
    : ${RRDDBPATH:=/var/db/rrd}
    : ${CF_CONF_PATH:=/cf/conf}
    : ${BACKUPTMP_PATH:=/backuptmp}
    
    # Save the rrd databases to the config path.
    if [ -d "${RRDDBPATH}" ]; then
    	[ -z "$NO_REMOUNT" ] && /etc/rc.conf_mount_rw
    	for rrdfile in "${RRDDBPATH}"/*.rrd ; do
    		xmlfiletmp="${rrdfile%.rrd}.xml"
    		xmlfile="${BACKUPTMP_PATH}/${xmlfiletmp##*/}"
    		/usr/bin/nice -n20 /usr/local/bin/rrdtool dump "$rrdfile" "$xmlfile"
    	done
    	cd / && tar -czf "${CF_CONF_PATH}"/rrd.tgz -C / "${BACKUPTMP_PATH#/}"/*.xml
    	rm "${BACKUPTMP_PATH}"/*.xml
    	[ -z "$NO_REMOUNT" ] && /etc/rc.conf_mount_ro
    fi
    
    

    Then I expect there would need to be similar code when unpacking the tar file at bootup to restore the rrd files via /backuptmp

    I have hard-coded BACKUPTMP_PATH in this example, but in a real solution it would need to be passed in as an option.

    Something could be added in System:Advanced Miscellaneous in the RAM Disk Settings area to give the user the option to use "/" filesystem as temporary storage for RRD backup XML.

    Is this something that people would find useful as an option on low-memory systems?



  • Actually it was not as difficult as I thought to code a possible solution: https://github.com/pfsense/pfsense/pull/1353
    If anyone would find this useful or has comments on my solution, go ahead…



  • A bit of back and forth with Ermal and I doing bits of code. One last fix to actually delete the transitory XML files on bootup/restore:
    https://github.com/pfsense/pfsense/pull/1356

    This is working very nicely for me now. I have some RRD history piling up over the last days and many boots and upgrades on my little Alix with a list of VLANs…

    Thanks Ermal for working this along.


Log in to reply