Bandwidthd graphs no longer displaying after updating to the latest version



  • Hi All

    I recently update bandwidthd to the latest available version. After the update I was no longer able to view the graphs.

    Findings:
    1. The settings are configured to enable showing graphs. (I have checked in the bandwidthd.conf file, /usr/local/bandwidthd/etc/)
    2. All the .png graph files located in (/usr/local/bandwidthd/htdocs/) have a zero file size.
    3. Running the "/usr/local/bandwidthd/bandwidthd" process application does not yield any different results or produce any errors.
    4. Uninstalling and Re-installing bandwidthd does not resolve the issue.
    5. All other graphs (RRD, ntop, lightsquid) are working/displaying correctly.

    ???

    Any suggestions would be very welcome.

    My System:
    2.0.2-RELEASE (i386)
    built on Fri Dec 7 16:30:38 EST 2012
    FreeBSD 8.1-RELEASE-p13

    Bandwidthd version 2.0.1.4

    Thanks



  • I am experiencing the same issues.

    2.0.2-RELEASE (i386)
    built on Fri Dec 7 16:30:38 EST 2012
    FreeBSD 8.1-RELEASE-p13

    Bandwidthd version 2.0.1.4



  • same here.

    im using:
    2.0.2-RELEASE (i386)
    built on Fri Dec 7 16:30:38 EST 2012
    FreeBSD 8.1-RELEASE-p13

    bandwidthd 2.0.1.4



  • From a non-nanoBSD system, can someone post a copy of /usr/local/etc/rc.d/bandwidthd.sh
    and the output of

    ls -l /usr/local/bandwidthd/bandwidthd
    

    The code now makes a custom bandwidthd.sh for nanoBSD that puts the data files into /var/bandwidthd (memory disk)
    But for systems with a real disk:
    bandwidthd.sh should be the same as before;
    /usr/local/bandwidthd/bandwidthd should be a sym link to where the program is (different for pfSense 2.0.n and 2.1 due to pbi on 2.1); and
    the data files on the real disk.
    Obviously I broke something for the non-nanoBSD case, but I can't see what and don't have access to a non-nanoBSD system right now.



  • Hi Phil

    Thanks for the fast response.
    Here are the outputs you requested:

    
    cat /usr/local/etc/rc.d/bandwidthd.sh
    
    

    Output

    
    #!/bin/sh
    # This file was automatically generated
    # by the pfSense service handler.
    
    rc_start() {
            /usr/local/bandwidthd/bandwidthd
    }
    
    rc_stop() {
            /usr/bin/killall bandwidthd
    }
    
    case $1 in
            start)
                    rc_start
                    ;;
            stop)
                    rc_stop
                    ;;
            restart)
                    rc_stop
                    rc_start
                    ;;
    esac
    
    
    
    ls -l /usr/local/bandwidthd/bandwidthd
    
    

    Output

    
    -rwxr-xr-x  1 root  wheel  59776 Jun 11  2012 /usr/local/bandwidthd/bandwidthd
    
    

    Thanks



  • Your bandwidthd.sh and /usr/local/bandwidthd/bandwidthd program look fine.
    I installed a VM running:
    2.0.2-RELEASE (i386)
    built on Fri Dec 7 16:30:14 EST 2012
    FreeBSD 8.1-RELEASE-p13

    Installed bandwidthd package (2.0.1.4) and put another VM behind it to generate some traffic.
    It runs fine for me:

    [2.0.2-RELEASE][admin@pfSense.localdomain]/usr/local/bandwidthd(79): ls -l
    total 64
    -rwxr-xr-x  1 root  wheel  59776 Jun 11  2012 bandwidthd
    drwxr-xr-x  2 root  wheel    512 Mar  6 06:31 etc
    drwxr-xr-x  2 root  wheel    512 Mar  6 11:17 htdocs
    [2.0.2-RELEASE][admin@pfSense.localdomain]/usr/local/bandwidthd(80): ls -l htdocs/
    total 30
    -rw-r--r--  1 root  wheel  2175 Mar  6 11:20 192.168.42.208-1-R.png
    -rw-r--r--  1 root  wheel  2164 Mar  6 11:20 192.168.42.208-1-S.png
    -rw-r--r--  1 root  wheel  2264 Mar  6 11:20 Subnet-1-192.168.42.0.html
    -rw-r--r--  1 root  wheel  2175 Mar  6 11:20 Total-1-R.png
    -rw-r--r--  1 root  wheel  2164 Mar  6 11:20 Total-1-S.png
    -rw-r--r--  1 root  wheel  2949 Mar  6 11:20 index.html
    -rw-r--r--  1 root  wheel   653 Mar  6 11:14 index2.html
    -rw-r--r--  1 root  wheel   653 Mar  6 11:14 index3.html
    -rw-r--r--  1 root  wheel   653 Mar  6 11:14 index4.html
    

    When I click the "Access Bandwidthd" tab, it opens in a new browser tab. I need to refresh that browser tab once bandwidthd has started making graphs, then it displays them.
    I tried the default settings (LAN and Draw Graphs on, other settings blank/off), and with output_cdf and recover_cdf selected. Both worked.
    Did you update the whole package, or just the XML?
    Can you give any clues about what is interesting/different about your setup?
    Then I can try and replicate it.



  • I'm not sure it's helpful or not, but I don't think it can hurt

    I just switched to pfSense over the weekend. Running the 2.0.2 vmware image I noticed the graphs were not displaying sometimes. It took me a few minutes, but I did realize that for me, it seemed to be if I accessed it with internet explorer 9. Chrome and Opera seem to work fine for me.



  • Hi Phil

    There is nothing special about my configurations. I am currently running two firewalls both having the same issue.
    One is a simple single WAN, single LAN with packages:
    squid3, lightsquid, bandwidthd, ntop, and backup

    The second is also very basic but has a single LAN and multiple WANs. With traffic shaping and routing over the multiple WANs.
    This configuration has the same packages installed.

    I have done an uninstall and re-install of bandwidthd on both servers.

    It appears though that the graph .png images are not being generated when I run /usr/local/bandwidthd/bandwidthd.
    So using different browsers does not resolve the issue.

    
    /usr/local/bandwidthd(13): ls -lst htdocs/ | less
    total 234
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-1-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-1-S.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-2-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-2-S.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-3-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.1-3-S.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-1-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-1-S.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-2-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-2-S.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-3-R.png
      0 -rw-r--r--  1 root  wheel      0 Mar  7 08:43 192.168.99.10-3-S.png
    
    

    Bandwidth Config:

    
    /usr/local/bandwidthd(15): cat etc/bandwidthd.conf
    #
    # This file was automatically generated by the pfSense
    # package management system.  Changing this file
    # will lead to it being overwritten again when
    # the package manager resyncs.
    #
    ####################################################
    # Bandwidthd.conf
    #
    # Commented out options are here to provide
    # documentation and represent defaults
    
    # Subnets to collect statistics on.  Traffic that
    # matches none of these subnets will be ignored.
    # Syntax is either IP Subnet Mask or CIDR
    subnet 192.168.99.0/24
    
    # Device to listen on
    # Bandwidthd listens on the first device it detects
    # by default.  Run "bandwidthd -l" for a list of
    # devices.
    dev "re0"
    
    ###################################################
    # Options that don't usually get changed
    
    # An interval is 2.5 minutes, this is how many
    # intervals to skip before doing a graphing run
    
    # Graph cutoff is how many k must be transfered by an
    # ip before we bother to graph it
    
    #Put interface in promiscuous mode to score to traffic
    #that may not be routing through the host machine.
    promiscuous false
    
    #Log data to cdf file htdocs/log.cdf
    output_cdf true
    
    #Read back the cdf file on startup
    recover_cdf true
    
    #Libpcap format filter string used to control what bandwidthd sees
    #Please always include "ip" in the string to avoid strange problems
    
    #Draw Graphs - This default to true to graph the traffic bandwidthd is recording
    #Usually set this to false if you only want cdf output or
    #you are using the database output option.  Bandwidthd will use very little
    #ram and cpu if this is set to false.
    graph true
    
    #Set META REFRESH seconds (default 150, use 0 to disable).
    
    

    I wish I could provide you with more information but I am not sure where else to look that may be causing the issue. There are not error messages appearing in my log files either.



  • I did another uninstall and re-install of bandwidthd. After the uninstall I noticed the file "bandwidthd.core" remains in the /usr/local/bandwidthd directory along with the cdf files.
    I moved the bandwidthd.core file to another location and ran the install. After the install there is no new bandwidthd.core file created in the /usr/local/bandwidthd folder.
    Is this file no longer required or has it been moved to another location?

    After uninstall "/usr/local/bandwidthd"

    
    /usr/local/bandwidthd(97): ls -lst
    total 189190
         2 drwxr-xr-x  2 root  wheel       512 Mar  7 09:01 etc
     63424 -rw-r--r--  1 root  wheel  64893216 Mar  7 09:00 log.1.0.cdf
     58080 -rw-r--r--  1 root  wheel  59413778 Mar  7 08:58 log.2.0.cdf
     10960 -rw-r--r--  1 root  wheel  11191933 Mar  7 08:40 log.3.0.cdf
      1376 -rw-r--r--  1 root  wheel   1386281 Mar  7 08:39 log.4.0.cdf
         0 -rw-r--r--  1 root  wheel         0 Jan 13 19:44 log.1.1.cdf.bak
     45760 -rw-r--r--  1 root  wheel  46807500 Jan 13 19:44 log.1.2.cdf
         2 -rw-r--r--  1 root  wheel       660 Nov 19 07:51 log.1.3.cdf
       112 -rw-r--r--  1 root  wheel    113848 Nov 19 07:51 log.1.4.cdf
         2 -rw-r--r--  1 root  wheel       432 Nov 18 22:56 log.1.5.cdf
      9472 -rw-------  1 root  wheel   9678848 Oct 18 16:16 bandwidthd.core
    
    

    After Re-install having moved the bandwidthd.core file before the install.

    
    /usr/local/bandwidthd(103): ls -lst
    total 179790
        12 drwxr-xr-x  2 root  wheel     11264 Mar  7 09:03 htdocs
         2 drwxr-xr-x  2 root  wheel       512 Mar  7 09:02 etc
     63424 -rw-r--r--  1 root  wheel  64893216 Mar  7 09:00 log.1.0.cdf
     58080 -rw-r--r--  1 root  wheel  59413778 Mar  7 08:58 log.2.0.cdf
     10960 -rw-r--r--  1 root  wheel  11191933 Mar  7 08:40 log.3.0.cdf
      1376 -rw-r--r--  1 root  wheel   1386281 Mar  7 08:39 log.4.0.cdf
         0 -rw-r--r--  1 root  wheel         0 Jan 13 19:44 log.1.1.cdf.bak
     45760 -rw-r--r--  1 root  wheel  46807500 Jan 13 19:44 log.1.2.cdf
         2 -rw-r--r--  1 root  wheel       660 Nov 19 07:51 log.1.3.cdf
       112 -rw-r--r--  1 root  wheel    113848 Nov 19 07:51 log.1.4.cdf
         2 -rw-r--r--  1 root  wheel       432 Nov 18 22:56 log.1.5.cdf
        60 -rwxr-xr-x  1 root  wheel     59776 Jun 11  2012 bandwidthd
    
    


  • It might turn out to be some file/dir protection issue? Or disk that has no space left? (and can add new filenames to an exist dir, but the files cannot get bigger than zero bytes) or something in bandwidthd ends up not being root and thus has not enough priv to write? or ?
    Do you have anything interesting about your disk partitions?
    As soon as I start bandwidthd, it produces 4 index html files wherever htdocs is (In /usr/local/bandwidthd (i386) or /var/bandwidth (nanobsd):

    -rw-r--r--  1 root  wheel  14735 Mar  7 12:46 index.html
    -rw-r--r--  1 root  wheel  13966 Mar  7 12:42 index2.html
    -rw-r--r--  1 root  wheel    653 Mar  7 11:52 index3.html
    -rw-r--r--  1 root  wheel    653 Mar  7 11:52 index4.html
    

    I only get the png files after data has been collected for the first period (2.5 minutes) (or if recover_cdf is on, then the first action of bandwidthd is to regenerate all the index*.html and *.png).
    Thinking…



  • My hdd is only at 11% capacity and there are no special permissions or anything like that required.
    All the HTML files (index and subnet) are being generated and the usage is definitely being recorded as the table at the top is displaying the usage per IP correctly.
    It only appears to be the graphs that have broken.
    Are there an specific libraries required to generate the .png graph files?



  • Having the same Problem here with 1.3 everything was fine 1.4 just not working. try´d reinstall. manually delete…
    MFG Alex



  • Hi Phil

    I have noticed that on my system I have two png libraries installed.

    
    png-1.4.8           Library for manipulating PNG images
    png-1.5.14          Library for manipulating PNG images
    
    

    The 1.4.8 library is installed by Bandwidthd and the 1.5.14 is installed by Lightsquid. Could it be possible that the method required by bandwidthd to produce the .png files is no longer available in the 1.5 version and is therefore failing to produce the files?

    Would you be able to test this by installing Lightsquid on the test instance you setup? And if this is the case how would one go about resolving the issue?

    Sorry for all the questions.

    Thanks



  • I also have lightsquid installed, but i already testet it with all packages uninstalled except for bandwithd.
    This was making no difference on my system.



  • I reproduced the symptoms. Installed 2.0.2 on a VM, then bandwidthd, enable bandwidthd defaults plus output_cdf and recover_cdf, reboot, all is good.
    Installed squid, enable transparent proxy and all other defaults, reboot, all is good.
    Installed lightsquid, did not much with it (pressed the "Refresh Now" button), rebooted.
    The packages all start fine. bandwidthd produces numeric output, but does not make the graphs.
    Here is a dir listing:

    [2.0.2-RELEASE][admin@front.localdomain]/usr/local/bandwidthd(14): ls -l
    total 82
    -rwxr-xr-x  1 root  wheel  59776 Jun 11  2012 bandwidthd
    drwxr-xr-x  2 root  wheel    512 Mar  8 15:27 etc
    drwxr-xr-x  2 root  wheel    512 Mar  8 16:05 htdocs
    -rw-r--r--  1 root  wheel    493 Mar  8 17:40 log.1.0.cdf
    -rw-r--r--  1 root  wheel    264 Mar  8 17:37 log.1.1.cdf
    -rw-r--r--  1 root  wheel   1671 Mar  8 17:37 log.1.2.cdf
    -rw-r--r--  1 root  wheel    254 Mar  8 17:22 log.1.3.cdf
    -rw-r--r--  1 root  wheel   2566 Mar  8 17:22 log.1.4.cdf
    -rw-r--r--  1 root  wheel    248 Mar  8 16:57 log.1.5.cdf
    -rw-r--r--  1 root  wheel   2654 Mar  8 17:28 log.2.0.cdf
    [2.0.2-RELEASE][admin@front.localdomain]/usr/local/bandwidthd(15): ls -l htdocs/
    total 30
    -rw-r--r--  1 root  wheel     0 Mar  8 17:40 10.99.1.42-1-R.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:40 10.99.1.42-1-S.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:33 10.99.1.42-2-R.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:33 10.99.1.42-2-S.png
    -rw-r--r--  1 root  wheel  2222 Mar  8 17:40 Subnet-1-10.99.1.0.html
    -rw-r--r--  1 root  wheel  2224 Mar  8 17:33 Subnet-2-10.99.1.0.html
    -rw-r--r--  1 root  wheel     0 Mar  8 17:40 Total-1-R.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:40 Total-1-S.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:33 Total-2-R.png
    -rw-r--r--  1 root  wheel     0 Mar  8 17:33 Total-2-S.png
    -rw-r--r--  1 root  wheel  2914 Mar  8 17:40 index.html
    -rw-r--r--  1 root  wheel  2916 Mar  8 17:33 index2.html
    -rw-r--r--  1 root  wheel   653 Mar  8 17:33 index3.html
    -rw-r--r--  1 root  wheel   653 Mar  8 17:33 index4.html
    -rw-r--r--  1 root  wheel  1649 Jun 11  2012 legend.gif
    -rw-r--r--  1 root  wheel  7573 Jun 11  2012 logo.gif
    

    The "png" files are all zero bytes!
    And the bandwidthd report looks like the screenshot below.
    While lightsquid was installing, I noticed that it had different versions of the same tbz files that bandwidthd had. e.g.
    bandwidthd - lightsquid
    png-1.4.8.tbz - png-1.5.14.tbz

    I am guessing that one of these messes up the report generation?
    And I guess uninstalling lightsquid does not remove whatever has gone wrong. The lightsquid versions were all later, so maybe they get left in place and then even a reinstall of bandwidthd does not overwrite them with the old version??? I am guessing here!

    I only changed the bandwidthd package xml and inc file. I didn't actually build any new binaries for this. I incremented the version number from 2.0.1.3 to 2.0.1.4 to flag a change to the package xml/inc and because actually the tbz file was already bandwidthd-2.0.1_4.tbz 2011-Oct-15 00:56:07 - so that tbz has been around for a long while also.

    It's well past bedtime in my time zone! I will sleep on it - any further ideas are most welcome.




  • The following differences in dependent packages seemed to be breaking the graphing:

    gd-2.0.35_7,1       A graphics library for fast creation of images
    gd-2.0.35_8,1       A graphics library for fast creation of images
    png-1.4.8           Library for manipulating PNG images
    png-1.5.14          Library for manipulating PNG images
    

    The earlier versions get installed with bandwidthd, then LightSquid installs the later ones - presumably overwriting various so files.
    I found ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/bandwidthd-2.0.1_5.tbz
    That has later dependencies.
    I did the following to use 2.0.1_5 bandwidthd:
    a) Remove LightSquid package, using the GUI
    b) Stop bandwidthd from Services->Status
    c) At a command prompt:

    pkg_delete bandwidthd-2.0.1_4
    pkg_delete gd-2.0.35_7,1
    pkg_delete png-1.4.8
    pkg_add -r ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/bandwidthd-2.0.1_5.tbz
    
    

    The addition emits a bunch of warnings about various other dependency issues:

    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/bandwidthd-2.0.1_5.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/png-1.5.12.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/gd-2.0.35_8,1.tbz... Done.
    pkg_add: warning: package 'gd-2.0.35_8,1' requires 'jpeg-8_3', but 'jpeg-8_4' is installed
    pkg_add: warning: package 'gd-2.0.35_8,1' requires 'pkgconf-0.8.8', but 'pkgconf-0.8.9' is installed
    pkg_add: warning: package 'gd-2.0.35_8,1' requires 'freetype2-2.4.9_1', but 'freetype2-2.4.11' is installed
    pkg_add: warning: package 'gd-2.0.35_8,1' requires 'freetype2-2.4.9_1', but 'freetype2-2.4.7' is installed
    pkg_add: warning: package 'bandwidthd-2.0.1_5' requires 'jpeg-8_3', but 'jpeg-8_4' is installed
    pkg_add: warning: package 'bandwidthd-2.0.1_5' requires 'pkgconf-0.8.8', but 'pkgconf-0.8.9' is installed
    pkg_add: warning: package 'bandwidthd-2.0.1_5' requires 'freetype2-2.4.9_1', but 'freetype2-2.4.11' is installed
    pkg_add: warning: package 'bandwidthd-2.0.1_5' requires 'freetype2-2.4.9_1', but 'freetype2-2.4.7' is installed
    pkg_add: warning: package 'bandwidthd-2.0.1_5' requires 'libiconv-1.14', but 'libiconv-1.13.1_1' is installed
    
    

    d) Install LightSquid from the pfSense GUI

    2.0.1_5 pbi is used by 2.1-BETA1 - maybe the tbz will work with 2.0.2.
    Can others try this?
    If successful then we can update the pfSense package version to 2.0.1_5



  • Hi Phil

    Thank you for your solution. I have updated my Bandwidthd package following your instructions and it is now working and displaying graphs correctly again.
    :D

    I did encounter an issue with lightsquid once I had re-installed it. I experienced a "Server 500 - Internal Error", this is easily fixed however using my previous resolution for the issue:
    http://forum.pfsense.org/index.php/topic,57505.0.html

    I have also check the RRD graphs and ntop Graphs and they all display correctly.

    I would say you may go ahead and update the bandwidthd package to version 2.0.1_5 for pfSense 2.0.2

    Thank you for your time.




  • I have submitted a pull request to make the GitHub changes: https://github.com/pfsense/pfsense-packages/pull/405
    Someone with privs also needs time to put the 2.0.1_5 file into http://files.pfsense.org/packages/8/All/ - things look rather busy right now, as they have changed the GitHub repo addresses, so it might take some time to be looked at.



  • Jimp has put bandwidthd-2.0.1_5.tbz into http://files.pfsense.org/packages/8/All/ and committed the version change - thanks.
    I installed 2.0.2 to a fresh VM, added bandwidthd 2.0.1_5 from the packages menu. It works. Installed Squid and then Lightsquid. Bandwidthd still works. Rebooted - bandwidthd comes up and generates and updates its report and graphs. Seems fixed.
    Can others on pfSense 2.0.n try this latest bandwidthd 2.0.1_5 and confirm it is OK.



  • Hi Phil

    Firstly, thank you to both you and Jimp for resolving this issue.

    Just a quick note on the upgrading to the new bandwidthd 2.0.1_5 from previous versions.
    I would advise that a full uninstall and re-install is done when upgrading. When running just the upgrade I experienced issues with the logo no longer being present.
    Also it is not necessary to uninstall and re-install any other packages as previously stated when we were testing. (eg. Lightsquid)

    Thanks again for the resolution gents. Much appreciated.



  • Uninstalled 1.4, installed 1.5, pressed Reload on Graph Site and voila there they are.
    Works fine.
    THX Alex


Log in to reply