Backup of RRD fills /var on shutdown

phil.davis

Alix nanoBSD 256MB memory
This is not a 2.2-specific problem, but putting here to see if anyone thinks a solution like this is worth having.
I have a system with a bunch of VLANs (7 of them) and 2 WANs. The RRD data in /var/db/rrd is about 12MB. When the backup is happening at shutdown/reboot these files get converted into *.xml, which is much bigger - it generates 35MB of xml files. Then all the *.xml are put into a single tar in /cf/conf - that file is less than 2MB.
During the creation of the *.xml there needs to be 35MB somewhere.
/var memory disk runs out of space. I increased it to 80MB, but then my whole system runs out of real memory during shutdown while trying to write all the *.xml in /var/db/rrd
It would be nice if there was a way to be generating each *.xml and piping it directly into the tar (not going through /var filesystem) - but I cannot see how to achieve this in /etc/rc.backup_rrd.sh
Another way is to put the *.xml in a dir on a real file system that will have plenty of space. e.g. I made /backuptmp (in "/" file system - the real CF card). That means doing some extra writes to the CF card to manage the interim data, but it works with this code in /etc/rc.backup_rrd.sh

#!/bin/sh

: ${RRDDBPATH:=/var/db/rrd}
: ${CF_CONF_PATH:=/cf/conf}
: ${BACKUPTMP_PATH:=/backuptmp}

# Save the rrd databases to the config path.
if [ -d "${RRDDBPATH}" ]; then
	[ -z "$NO_REMOUNT" ] && /etc/rc.conf_mount_rw
	for rrdfile in "${RRDDBPATH}"/*.rrd ; do
		xmlfiletmp="${rrdfile%.rrd}.xml"
		xmlfile="${BACKUPTMP_PATH}/${xmlfiletmp##*/}"
		/usr/bin/nice -n20 /usr/local/bin/rrdtool dump "$rrdfile" "$xmlfile"
	done
	cd / && tar -czf "${CF_CONF_PATH}"/rrd.tgz -C / "${BACKUPTMP_PATH#/}"/*.xml
	rm "${BACKUPTMP_PATH}"/*.xml
	[ -z "$NO_REMOUNT" ] && /etc/rc.conf_mount_ro
fi

Then I expect there would need to be similar code when unpacking the tar file at bootup to restore the rrd files via /backuptmp

I have hard-coded BACKUPTMP_PATH in this example, but in a real solution it would need to be passed in as an option.

Something could be added in System:Advanced Miscellaneous in the RAM Disk Settings area to give the user the option to use "/" filesystem as temporary storage for RRD backup XML.

Is this something that people would find useful as an option on low-memory systems?

phil.davis

Actually it was not as difficult as I thought to code a possible solution: https://github.com/pfsense/pfsense/pull/1353
If anyone would find this useful or has comments on my solution, go ahead…

phil.davis

A bit of back and forth with Ermal and I doing bits of code. One last fix to actually delete the transitory XML files on bootup/restore:
https://github.com/pfsense/pfsense/pull/1356

This is working very nicely for me now. I have some RRD history piling up over the last days and many boots and upgrades on my little Alix with a list of VLANs…

Thanks Ermal for working this along.