[solved] RRD graphs stop logging after reboot of VM

lockheed

It used to work splendidly, but since yesterdays reboot, Status: RRD Graphs show nothing. The graphs stopped upon reboot, the historical graph lines are still there, but now it registers absolutely nothing. No Throughput, no Memory, no Processor.

I am on pfSense 2.2.1 and the only installed extra package is RRD Summary.

EDIT:
I followed those advices: https://forum.pfsense.org/index.php?topic=57312.0
and it started working after reboot.

lockheed

Scratch that. Turns out it only works until next reboot. Then the graphs stop once more, and I have to delete all old data by running

rm /var/db/rrd/*.rrd

to get them start logging again. But again, this is good only until next reboot.

I thought it might be because the timestamps, but I made sure system time is correct inside the pfSense VM by adding working NTP servers, and did the same thing for the host machine.

charliem

@lockheed:

I am on pfSense 2.2.1 and the only installed extra package is RRD Summary.

~~Have you tried removing that package?~~

I installed the RRD Summary package, and it works OK. Looking at it, the package only deals with RRD data that's already collected, so this package is not related to your issue.

Can you get to a command prompt and see if 'updatedrrd.sh' is running? That's the process that populates the RRD databases; it normally runs continuously, sleeping 60 seconds then waking up to take a set of readings, and repeating. The file is generated and started at each boot, with a note in the syslog file:


[2.2.2-DEVELOPMENT][root@pfsense.localdomain]/root: ps axfw | grep updaterrd
34555  -  SN     0:04.29 /bin/sh /var/db/rrd/updaterrd.sh
44960  0  S+     0:00.00 grep updaterrd
[2.2.2-DEVELOPMENT][root@pfsense.localdomain]/root: clog /var/log/system.log | grep rrd
Apr  1 08:34:55 pfsense php: rc.bootup: Creating rrd update script
Apr  1 08:35:01 pfsense php-fpm[94438]: /rc.newwanip: Creating rrd update script

Hmm, seems it's being created twice: once at boot and once when rrd.inc is included in rc.newwanip? Probably not related to your issue though.

lockheed

Deleting RDD Summary didn't help. Here's the output of the command you suggested:

$ ps axfw | grep updaterrd
27699  -  S     0:00.00 sh -c ps axfw | grep updaterrd 2>&1
28057  -  R     0:00.00 grep updaterrd
31035 v0- IN    0:00.28 /bin/sh /var/db/rrd/updaterrd.sh

I reinstalled pfSense like 7 times already. Every time its the same problem.

charliem

Any rrd stuff in the logs?

Are the rrd files actually being updated?

[2.2.2-DEVELOPMENT][root@pfsense.localdomain]/root: date
Thu Apr  2 22:36:10 EDT 2015
[2.2.2-DEVELOPMENT][root@pfsense.localdomain]/root: ls -lt /var/db/rrd | head -10
total 11340
-rw-r--r--  1 nobody  wheel    98784 Apr  2 22:35 WAN_DHCP-quality.rrd
-rw-r--r--  1 nobody  wheel    98784 Apr  2 22:35 WAN_DHCP6-quality.rrd
-rw-r--r--  1 nobody  wheel   882048 Apr  2 22:35 ntpd.rrd
-rw-r--r--  1 nobody  wheel   588592 Apr  2 22:35 system-mbuf.rrd
-rw-r--r--  1 nobody  wheel   735320 Apr  2 22:35 system-memory.rrd
-rw-r--r--  1 nobody  wheel   245976 Apr  2 22:35 system-processor.rrd
-rw-r--r--  1 nobody  wheel    49720 Apr  2 22:35 ovpns2-vpnusers.rrd
-rw-r--r--  1 nobody  wheel   245976 Apr  2 22:35 system-states.rrd
-rw-r--r--  1 nobody  wheel   393168 Apr  2 22:35 ipsec-packets.rrd

@lockheed:

I reinstalled pfSense like 7 times already. Every time its the same problem.

Fresh install? Or importing an old config file?

Have you tried clearing the checkbox 'Enable RRD graphing' (status, rrd graphs, settings)? Hit save. Then tick the box and hit save again. That should restart the data collection. Not a solution, but another data point.

lockheed

I appreciate your help, charliem!

Playing around with 'Enable RRD graphing' switch did not work and in most cases, I was just restoring old config.

Anyway, I migrated my firewall from VirtualBox to KVM and now, even though I restored the exact same config, graphs seem to be working. I will report back if something changes.

lockheed

Ok, I had to come back to VirtualBox because KVM is a total disaster in terms of virtual networking which makes it impossible to set up.

Anyhow, I followed your suggestion and - lo and behold - the cause: The time on pfSense is 13:30 but the timestamp on rrd files is 15:30, which means they are two hours ahead.

This does not happen in KVM. I do not know why it is so in VBox. The time on pfSense box is correct because it's synced with an NTP server. Perhaps the box starts with wrong time and only afterwards it corrects it, but the .rrd files are already created?

EDIT:
I fixed it by setting the time to UTC in VBox guest settings.