Vnstat to retain more than 30 days daily stats



  • Is there any way for me to tell vnstat to retain more than 30 days of daily traffic totals stats? When I need to challenge my ISP it always happens after the billing cycle when I lost the first couple of days of daily stats, so I only have the monthly to go on. If I could retain 60 days of daily totals that would be awesome. Any ideas?


  • LAYER 8 Global Moderator

    Simple solution would be just to export it as csv.. Under the display advanced there is an export button



  • I’ll script it, thx.



  • If anyone else needs this - it is really a quick hack I put together last night after midnight so please ignore the horrible code. It is a python script that should be created somewhere on pfSense (mine is in /root), then called from cron once a day - I recommend something like 00:30 each day but it is not important. It will append the daily totals for all interfaces on the cmdline to txt files located at /var/spool/vnstat_daily_<ifacename>.txt. It is idempotent and will only append those lines not yet present and only up to the previous day (as that is the last complete day of stats vnstat has). I am very busy at the moment and might improve it later to output proper CSV - for now I just need the data. Call me lazy. Use and modify as you wish:

    #!/usr/local/bin/python2.7
    
    import os
    import re
    import sys, getopt
    from datetime import datetime, timedelta
    
    
    def present(dte, filename):
        with open(filename) as infile:
            for line in infile:
                m = re.match(r'^' + dte + ' .*$', line.strip())
                if not m:
                    continue
                return True
        return False
    
    
    def usage():
        print("Usage: python parsevnstat.py -i ifacelist")
        sys.exit(2)
    
    def main(argv):
        try:
            opts, args = getopt.getopt(argv, 'hi:')
        except:
          usage()
    
        ifaces = []
        for opt, arg in opts:
            if opt == "-h":
                usage()
            elif opt == "-i":
                ifaces = arg.split(",")
    
        if len(ifaces) <= 0:
            usage()
    
        for iface in ifaces:
            filename = '/var/spool/vnstat_daily_'+iface+'.txt'
    
            header = False
            if not os.path.isfile(filename):
                header = True
            with open(filename, 'a') as outfile:
                if header:
                    outfile.write('   day         rx      |     tx      |    total    |   avg. rate\n')
                    outfile.write('-----------------------+-------------+-------------+---------------\n')
                with os.popen('/usr/local/bin/vnstat -d -i '+iface) as pipe:
                    for line in pipe:
                        m = re.match(r'^(\d\d/\d\d/\d\d) .*$', line.strip())
                        if not m:
                            continue
                        dte = datetime.strptime(m.group(1), '%m/%d/%y').date()
                        if dte <= (datetime.today() - timedelta(days=1)).date() and not present(m.group(1), filename):
                            outfile.write(line.strip() + '\n' )
    
    main(sys.argv[1:])
    

    Cron entry:

    30	0	*	*	*	root	/usr/local/bin/python2.7 /root/parsevnstat.py -i em0,em1
    


  • Starting from vnStat version 2.0, it's possible to freely configure the data retention periods without discarding already existing data. I don't however know which version of vnStat is currently provided by pfSense.

    Regarding that quick hack, it's a good practice to avoid parsing the formatted output as the layout is somewhat version dependent and the date string depends on configuration settings and system locale. Parsing the --json output would be a more fail proof solution. There's also --xml and --oneline available as alternatives.



  • [2.4.4-RELEASE][root@fw-pwn.local]/root: vnstat --version
    vnStat 1.15 by Teemu Toivola <tst at iki dot fi>
    

    I agree with the json / XML comment, as I said, this was a very quick thing done for me to preserve my history. My region will not change for me (for now) so it will work. But sure - that change will make it much better. Just no time...



  • Slightly improved version - this uses the raw export format as that is much easier consumed by spreadsheets and bypasses the regional issues.

    #!/usr/local/bin/python2.7
    
    import os
    import re
    import sys, getopt
    from datetime import datetime, timedelta
    
    
    def format_date(dte):
        return "{0:04d}-{1:02d}-{2:02d}".format(dte.year, dte.month, dte.day)
    
    def present(dte, filename):
        formatted_dte = format_date(dte)
        with open(filename) as infile:
            for line in infile:
                m = re.match(r'^' + formatted_dte + ',.*$', line.strip())
                if not m:
                    continue
                return True
        return False
    
    def format_line(fields,dte):
        rx = int(fields[3]) + int(fields[5])/1024.0
        tx = int(fields[4]) + int(fields[6])/1024.0
        return format_date(dte) + \
            ',{0:.3f}'.format(rx) + \
            ',{0:.3f}'.format(tx) + \
            ',{0:.3f}'.format(rx+tx) + \
            ',{0:.3f}'.format(rx / 1024.0) + \
            ',{0:.3f}'.format(tx / 1024.0) + \
            ',{0:.3f}'.format((rx+tx) / 1024.0),
        return "{0:04d}-{1:02d}-{2:02d}".format(dte.year, dte.month, dte.day)
    
    
    def usage():
        print("Usage: python parsevnstat2.py -i ifacelist")
        sys.exit(2)
    
    def main(argv):
        try:
            opts, args = getopt.getopt(argv, 'hi:')
        except:
          usage()
    
        ifaces = []
        for opt, arg in opts:
            if opt == "-h":
                usage()
            elif opt == "-i":
                ifaces = arg.split(",")
    
        if len(ifaces) <= 0:
            usage()
    
        for iface in ifaces:
            filename = '/var/spool/vnstat_daily_'+iface+'.csv'
    
            header = False
            if not os.path.isfile(filename):
                header = True
            with open(filename, 'a') as outfile:
                if header:
                    outfile.write('date,rx_MiB,tx_MiB,tot_MiB,rx_GiB,tx_GiB,tot_GiB\n')
                lines = []
                with os.popen('/usr/local/bin/vnstat --exportdb -i '+iface) as pipe:
                    for line in pipe:
                        m = re.match(r'^d;.*$', line.strip())
                        if not m:
                            continue
    		    fields = line.strip().split(";")
    		    if len(fields) != 8:
    			continue
                        dte = datetime.fromtimestamp(int(fields[2])).date()
                        if dte <= (datetime.today() - timedelta(days=1)).date() and not present(dte, filename):
                            lines.append(format_line(fields,dte))
    
    	    for line in sorted(lines):
                	outfile.write(line + '\n' )
    
    main(sys.argv[1:])
    


  • Thanks for this, and also thanks for the other help you gave, I will see if I can make a patch for the GUI to support 60 days worth of data.


Log in to reply