RRD Graphs on Embedded - CF Issue?

[MindTwist]

I would have actually replied to this 2 year old thread, but since old threads get locked after one year of inactivity I will open a new one:

http://forum.pfsense.org/index.php/topic,1818.0.html

My issue is that I have just moved from having my pfSense on an old normal PC (Athlon 1333, 512Mb, HD80Gb, 4 NICs 3Com-905C) to a Fabiatech FX5620 (VIA 1Ghz, 256Mb, Compact Flash, 6 NICs). I have also done the switch to pfSense embedded version, which is the first time I have used. I understand that the main differences between normal and embedded versions are that the latest should have minimal disk access.

My problem is that for whatever reason (unknown at this moment) yesterday my pfSense system stoped working, it didn't even respond to ping requests, so I had to do a cold reboot (off/on switch on its back). After it came back up, everything was up and running again, but I had lost the RRD graph information on about the last 48h (that might be since I last restarted the system, I do not know for sure). I was wondering if this is normal behaviour? I understand that to minimize writes on the CF card, maybe the RRD information is not written every minute, the cache is not flushed, or whatever. But 48h of information lost?

I have only been running embedded for about 7 days now, so I have info from maybe the first 4 days, then I have a 2 days white gap, and again info for the last 48h. I was wondering if it is normal behaviour to loose that much information if the system is restarted abnormally for whatever reason (power goes out, etc.)

Thanks!

[databeestje]

The rrd graphs are only saved to the config area during a clean reboot or shutdown from the console or webui.
They are restored from here when you bootup the machine.

This machines that if your machine crashes after a year you lose all graphs since the last clean shutdown.

Kind regards,

Seth

[MindTwist]

So here I am back after the power went out another time :D

Since restarting by hand daily would be a bit boring, isn't there some command that can be issued with some automated way to save the RRD data by hand maybe once a day at least? Like, a way of running daily pfsense/exec.php with some parameters that will make pfSense flush the data to the CF?

Thanks!

[MindTwist]

Hopefully this might help me, I'll give it a try.

http://forum.pfsense.org/index.php/topic,10992.0.html
http://forum.pfsense.org/index.php/topic,4786.0.html

[databeestje]

Add a cron entry in the config.xml to run tar. That is the same way we do it during reboots.

I believe the exact tar command is listed in the rc.shutdown or rc.reboot script

[MindTwist]

That did it, it is in both rc.shutdown and rc.reboot. The exact command is:

cd / && tar -czf /cf/conf/rrd.tgz -C / var/db/rrd/*.rrd

I tried doing it from Diagnostics / Command promt, then just turned off and on pfSense without shutting down correctly, and now I still have my RRD data. I just configured a cron job to run that command daily at 11:59 and 23:59, so hopefully I will never lose more than 12h of RRD data if the power gouts out.

I do not think that writing 2 times daily will wear out too much my CF card.

Thanks! :D

[fabifri]

I have also the issue, that sometimes some hours of rrd-graphs are lost…due to power losses...

So I have tried to install the PC-Edition on the CF, but it won't boot in my Alix - not only for the savings of the RRD-Databases, i also liked it to have packages-support on my Alix. I know it's not supported to run the PC-Build on Embedded platforms, but is there any guide for a workaround to get the PC-Edition running on my Alix-Board?

So, as a temporal solution i would like to make this cron-job to save the data, but i first tried to issue the command manually on the shell. I got the following error:

tar: Failed to open '/cf/conf/rrd.tgz': Read-only file system

so i think, that the cron-job also would fail - i am running 1.2.1-RC2 on my Alix now.

Thanks for any tipps and help

regards
Fabian

[MindTwist]

It does work ok for me, but it ain't working when doing a cron job.

http://forum.pfsense.org/index.php/topic,11275.msg62500.html#msg62500

Note that the command I am using is exactly the following, so it will be one single line:
/usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd

When done from Diagnostics -> Command prompt, it does work. I can then do a ls -la /cf/conf and a new rrd.tgz has been created. I was unable to get it to work as a cron job like you can see on thread I link to above (haven't done any more trying after that)

[databeestje]

the easiest way to get a full install onto a CF is using vmware server.

I create a new VM in vmware server with a disk size of a little less then the CF card.
I then install pfSense from the cdrom, select GRUB as the boot loader, and select the embedded kernel.

Then I reboot pfsense after the install and power down the VM during the reboot when it hits the bios.

I then use physdiskwrite to write the FreeBSD-flat.vmdk to the CF card.

That works for me.

[MindTwist]

Hi databeestje,

The issue is not about installing pfSense on a CF card, but on what to do so the RRD graph information won't get lost when a cold reboot/power out happens! :)

[databeestje]

fabifri asked for a solution to get the full install on a CF. So it's perfectly valid.

[GruensFroeschli]

I think a friend run into a similar problem recently.
I think he solved it by temporarily mounting the filesystem as read/write.

I'll ask him the next time i see him if he could post his solution.

[MindTwist]

databeestje, fabifri was trying to install the full version on a CF card because he was also having lost RRD information due to power losses.

I really do not know how many read/write cycles a CF card can sustain before passing out, but I do not think installing the full version on a CF card is a good solution for that. The RRD data on a normal installation is probably written to disk at least once a minute, which would ammount to at least 525600 extra writes every year on the card. CF cards are cheap, yeah, so it probably is not such a big deal to just throw an extra CF card every once in a while before your current CF dies, but I am sure the embedded version must exist for a reason.

By saving the RRD data with a cron job you can choose how often you want to do it, so if the power goes out you won't lose much data. Once a day? Every 12h? Hourly? It will probably be just easier than installing the full version on a CF card.

Thx!

[GruensFroeschli]

mount -o rw /cf/ ; /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd ; mount -o ro /cf

[MindTwist]

Thanks, I will give that a try. Maybe the command I am running /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd from the WebGUI is working ok but not working from a cron job for whatever permissions reason. I will try changing the cron job to mount -o rw /cf/ ; /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd ; mount -o ro /cf and see how that works out! :D

[MindTwist]

I think that didn't do the trick for me. Not only the cron job still is doing nothing, but now I can not even do it by hand like I used to, dang… I created the cron job with mount -o rw /cf/ ; /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd ; mount -o ro /cf yesterday at 17:23, restarted pfSense, and now my rrd.tgz has timestamp 17:23, the cron job has done nothing.

And doing a /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd now does nothing at all either, I guess the directory/file is readonly… Not even if I do a mount -o rw /cf/ by hand before. Oh well :(

		 <minute>0</minute>
			<hour>1,7,13,19</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>mount -o rw /cf/ ; /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/*.rrd ; mount -o ro /cf 

[MindTwist]

Ok, that sucked. I would advise anyone to NOT try the command I tried above as a cron job, lol…

You end up with everything being readonly on /conf , so the RRD info does not get saved at all (not even on reboot), and I was not even able to restore one of the old config.xml I kept around, since it was giving me the following error:

Warning: copy(/conf/config.xml): failed to open stream: Read-only file system in /etc/inc/config.inc on line 1172 

Now, what I end up doing was doing a mount -o rw /cf/ from the command prompt, and just deleting the /conf/config.xml. Luclily, pfSense is smarter than me, and it detected that the config file was gone, and restored one of the old ones. After rebooting, I was then able to restore a config.xml from the WebGUI.

So, a new gap on my RRD graphs due to lost data, and I still do not have a way of saving the RRD data at least once a day, except doing it once in a while from the command prompt… :(

[spock521]

I was having similar issues some time ago and I never found any working solution on these forums but after some experimentation I figured out a way to make this work (atleast for me). So in case someone else is trying to do the same I thought I'd share my solution (hopefully it will help someone).

Add this cron-entry to the configuration file:

<minute>0</minute>
  <hour>3,15</hour>
  <mday></mday>
  <month></month>
  <wday></wday>
  <who>root</who>
  <command></command>echo "RRD Graphs: Backup started." | logger; /etc/rc.conf_mount_rw ; /usr/bin/tar -czf /cf/conf/rrd.tgz -C /var/db/rrd/.rrd ; /etc/rc.conf_mount_ro ; echo "RRD Graphs: Backup complete." | logger

I don't know if this is a proper way to do it, but it's the only way I found to work (manually remounting the CF read-write complains about permissions etc.). And it havent caused me any problems or side-effects so far.

[littlekp]

I'm just struggling with the lost RRD data on my 1.2 version.
Thanks for the info about the reason of lost data.
Next Time I will do the clean boot rather than just turn it off by the power button. :P

[MindTwist]

I was out all movember and hadn't been able to try it, check out the results, and post back until now, but the cron job as above by spock521 works great for me too.
Thanks!