Vnstat to retain more than 30 days daily stats
-
Is there any way for me to tell vnstat to retain more than 30 days of daily traffic totals stats? When I need to challenge my ISP it always happens after the billing cycle when I lost the first couple of days of daily stats, so I only have the monthly to go on. If I could retain 60 days of daily totals that would be awesome. Any ideas?
-
Simple solution would be just to export it as csv.. Under the display advanced there is an export button
-
I’ll script it, thx.
-
If anyone else needs this - it is really a quick hack I put together last night after midnight so please ignore the horrible code. It is a python script that should be created somewhere on pfSense (mine is in /root), then called from cron once a day - I recommend something like 00:30 each day but it is not important. It will append the daily totals for all interfaces on the cmdline to txt files located at /var/spool/vnstat_daily_<ifacename>.txt. It is idempotent and will only append those lines not yet present and only up to the previous day (as that is the last complete day of stats vnstat has). I am very busy at the moment and might improve it later to output proper CSV - for now I just need the data. Call me lazy. Use and modify as you wish:
#!/usr/local/bin/python2.7 import os import re import sys, getopt from datetime import datetime, timedelta def present(dte, filename): with open(filename) as infile: for line in infile: m = re.match(r'^' + dte + ' .*$', line.strip()) if not m: continue return True return False def usage(): print("Usage: python parsevnstat.py -i ifacelist") sys.exit(2) def main(argv): try: opts, args = getopt.getopt(argv, 'hi:') except: usage() ifaces = [] for opt, arg in opts: if opt == "-h": usage() elif opt == "-i": ifaces = arg.split(",") if len(ifaces) <= 0: usage() for iface in ifaces: filename = '/var/spool/vnstat_daily_'+iface+'.txt' header = False if not os.path.isfile(filename): header = True with open(filename, 'a') as outfile: if header: outfile.write(' day rx | tx | total | avg. rate\n') outfile.write('-----------------------+-------------+-------------+---------------\n') with os.popen('/usr/local/bin/vnstat -d -i '+iface) as pipe: for line in pipe: m = re.match(r'^(\d\d/\d\d/\d\d) .*$', line.strip()) if not m: continue dte = datetime.strptime(m.group(1), '%m/%d/%y').date() if dte <= (datetime.today() - timedelta(days=1)).date() and not present(m.group(1), filename): outfile.write(line.strip() + '\n' ) main(sys.argv[1:])Cron entry:
30 0 * * * root /usr/local/bin/python2.7 /root/parsevnstat.py -i em0,em1 -
Starting from vnStat version 2.0, it's possible to freely configure the data retention periods without discarding already existing data. I don't however know which version of vnStat is currently provided by pfSense.
Regarding that quick hack, it's a good practice to avoid parsing the formatted output as the layout is somewhat version dependent and the date string depends on configuration settings and system locale. Parsing the --json output would be a more fail proof solution. There's also --xml and --oneline available as alternatives.
-
[2.4.4-RELEASE][root@fw-pwn.local]/root: vnstat --version vnStat 1.15 by Teemu Toivola <tst at iki dot fi>I agree with the json / XML comment, as I said, this was a very quick thing done for me to preserve my history. My region will not change for me (for now) so it will work. But sure - that change will make it much better. Just no time...
-
Slightly improved version - this uses the raw export format as that is much easier consumed by spreadsheets and bypasses the regional issues.
#!/usr/local/bin/python2.7 import os import re import sys, getopt from datetime import datetime, timedelta def format_date(dte): return "{0:04d}-{1:02d}-{2:02d}".format(dte.year, dte.month, dte.day) def present(dte, filename): formatted_dte = format_date(dte) with open(filename) as infile: for line in infile: m = re.match(r'^' + formatted_dte + ',.*$', line.strip()) if not m: continue return True return False def format_line(fields,dte): rx = int(fields[3]) + int(fields[5])/1024.0 tx = int(fields[4]) + int(fields[6])/1024.0 return format_date(dte) + \ ',{0:.3f}'.format(rx) + \ ',{0:.3f}'.format(tx) + \ ',{0:.3f}'.format(rx+tx) + \ ',{0:.3f}'.format(rx / 1024.0) + \ ',{0:.3f}'.format(tx / 1024.0) + \ ',{0:.3f}'.format((rx+tx) / 1024.0), return "{0:04d}-{1:02d}-{2:02d}".format(dte.year, dte.month, dte.day) def usage(): print("Usage: python parsevnstat2.py -i ifacelist") sys.exit(2) def main(argv): try: opts, args = getopt.getopt(argv, 'hi:') except: usage() ifaces = [] for opt, arg in opts: if opt == "-h": usage() elif opt == "-i": ifaces = arg.split(",") if len(ifaces) <= 0: usage() for iface in ifaces: filename = '/var/spool/vnstat_daily_'+iface+'.csv' header = False if not os.path.isfile(filename): header = True with open(filename, 'a') as outfile: if header: outfile.write('date,rx_MiB,tx_MiB,tot_MiB,rx_GiB,tx_GiB,tot_GiB\n') lines = [] with os.popen('/usr/local/bin/vnstat --exportdb -i '+iface) as pipe: for line in pipe: m = re.match(r'^d;.*$', line.strip()) if not m: continue fields = line.strip().split(";") if len(fields) != 8: continue dte = datetime.fromtimestamp(int(fields[2])).date() if dte <= (datetime.today() - timedelta(days=1)).date() and not present(dte, filename): lines.append(format_line(fields,dte)) for line in sorted(lines): outfile.write(line + '\n' ) main(sys.argv[1:]) -
Thanks for this, and also thanks for the other help you gave, I will see if I can make a patch for the GUI to support 60 days worth of data.
