Bounty $200: Monitor bandwidth use on IP adresses. NOW $250

[morbus]

I have had this problem too is seems to take a moment after install to generate the first graph. This looks ok after the first graph generation. Then after maybe one more cycle it fills index.html with nothing (if you look at the raw file via ssh it is empty) sometimes index2.html has some content for longer but leave it for a night and they are all blank.

If it helps I have a CARP setup that may be confusing it. I also have the snort package installed so are they both trying to use the same interface and that is confusing it.

If you restart the service the whole process starts again.

I am not hugely into using this package so I have removed it now but I will test any changes

As an aside we use Argus and rrd-tool on our current firewall to generate data like this. I installed argus from BSD (pkg_add etc) and it works fine on Pfsense. I started working on a package but haven't had any time to really work on it. Maybe someone else would do better.

for info on argus see http://www.qosient.com/argus/

Thanks for a great project

[MJK]

Trivial updates…

I just installed NTOP and DarkStat, and they are collecting and displaying data with no obvious problems. Which suggests that the PC itself, pfSense, etc, etc, is probably running correctly, and that the "blank-page" is an issue with BandWidthD.

Also, any time today that I tried to access the graphs in BandWidthD, I always got the "blank-screen". Previously, I was randomly getting either the totally-blank-screen, or the logo and a note that there was no data to graph, but the latter has not appeared for some time.

Also, could someone please suggest what value I should use in each field in the Options page? I want to monitor "LAN" traffic, per internal IP.

Thank you,
  - Mike

[sullrich]

By default you do not need to enable any options besides picking the LAN interface.

Sorry but I am having trouble reproducing the issue you outline.  But I can say that running NTOP and BandwidthD at the same time is not a good idea…

[Perry]

@MJK:

Also, could someone please suggest what value I should use in each field in the Options page? I want to monitor "LAN" traffic, per internal IP.

Your may have unsupported network cards (bandwidthd issue not pfsense). So if you ain't got errors in system logs try with another network adapter using default settings.

/Perry

[sullrich]

@Perry:

@MJK:

Also, could someone please suggest what value I should use in each field in the Options page? I want to monitor "LAN" traffic, per internal IP.

Your may have unsupported network cards (bandwidthd issue not pfsense). So if you ain't got errors in system logs try with another network adapter using default settings.

/Perry

That is very true.  We discussed this in the beginning of the thread if someone has overlooked it.

[MJK]

Thanks to Scott and Perry for responses.

I'll gladly uninstall NTOP - I was just wondering…..

I'm using the same card for the WAN and LAN sides - an old "standard" 10/100 SMC card; using the 83C171A2QF chip. I think this is very common NIC. Is there some list of supported cards that I can check, and I could then try one that's definitely supported.

It might be useful if BandWidthD could check for compatible NICs, or indicate if the NIC is not responding properly. Also, would an incompatible card be more likely to produce no traffic, or crazy traffic, rather than upset the reporting functions? IOW, are the reporting functions working on the data that has been previously collected, rather than accessing the NICs directly?

Also, the Router itself has been in serious use over recent days, with no comms problems whatsoever.

Many thanks again,
  - Mike

[MJK]

Slight update on the NICs…

SMC NIC, IC is 83c17x, also coded SMC 9432TX, also known as the EPIC version...

Seems it's supported wit the TX(4) drivers in FreeBSD/i386 6.1 - apologies if this info is useless...

http://www.freebsd.org/cgi/man.cgi?query=tx&sektion=4&manpath=FreeBSD+6.1-RELEASE

- Mike

[MJK]

I've re-tried accessing the stats on BandWidthD, but I'm still getting a totally blank screen.

I looked in the HTDOCS folder; it contains:
  - about 20 PNG files, typically about 2KB to 3KB in size
  - a few "subnet"… HTML files, also about 2KB in size.
  - 4 INDEX.HTML files (Index, Index2, Index3, and Index4), each has a size of 0.
  - A few other misc files.

Anyone know if all these are Ok, especially the "0" size in the Index files?

Thank you,
  - Mike

[sullrich]

Most likely another bug in bandwidthd.

[MJK]

WOW, that was fast! Thank you.

I presume you're referring to the "0" files? Do the 2KB/3KB files suggest that the SMC NICs are supported properly?

Seems many others have had similar experiences - ie, empty reports, and, perhaps, with similar causes.

Anything I can do to help investigate - let me know, but you'll have to give me "stooopid" instructions! I may have to install the Router at it's targeted site over the next few days, so we may not have unlimited time to crack this one.
  - Mike

[sullrich]

I don't trust any nics except intel or 3com under FreeBSD.  That very well could be the problem.

[MJK]

OK.

I'll try to locate some Intel or 3com NICs, and let you know the results.

Thanks again,
  - Mike

[MJK]

Some more significant feedback:

Logging:
I tried with "Log to CDF" switched on; some large CDF files are being created/updated in /usr/local/bandwidthd/,  one is 250KB, another 110KB, etc. These files have just integer values inside, as well as IP addresses, etc, and the layout seems exactly as expected, based on some websearches I ran. While there's active traffic on the LAN NIC, these files are being updated, which indicates that BandWidthD is monitoring the traffic…

Analysing:
The PNG files, etc, in /usr/local/bandwidthd/htdocs/ are also being refreshed frequently, from every few minutes to every 10/15/20/30 mins, depending on traffic. Also as expected - these are being updated from the logged data. The 4 x INDEX.HTML files are also being refreshed every xx minutes, BUT, they are always totally empty. When I manually put some data into these files, that content appeared on the BWD Access screen.

So... BandWidthD is simply not creating the INDEX files. Maybe we're getting real close! Anyone any ideas on where their contents comes from - from bandwidthd code, or from some other apps, or copied out of some other master files, or...

Seems many others have had similar experiences, including:
http://sourceforge.net/tracker/index.php?func=detail&aid=1365577&group_id=89685&atid=591011

...and it "got resolved"... somehow...
http://sourceforge.net/forum/forum.php?thread_id=986168&forum_id=308609
This user just re-cycled stopping and starting BWD, and, magically, the INDEX files were built again. I've tried that sequence a little also, with kill or killall, and with/without "-HUP", but, here, the INDEX files have not kicked into life.

All suggestions welcome! I'm now guessing there's no need to change the NICs.
  - Mike

[imoex2]

i will contrib $25 to this bounty

[MJK]

Comment for Scott, in case it's of any interest:

While still messing with the non-creation of the INDEX files, I sometimes tweaked that actual BANDWIDTHD.CONF file manually. If this file is updated via the GUI, I noticed the following:
  - If the LAN/WAN option is changed, then internal subnet is automatically inserted (LAN), or removed (WAN). Nice!
  - The SKIP_INTERVALS, GRAPH (draw graphs), and META_REFRESH parameters are not inserted to the CONF file. However, they are retained (somewhere?) for the GUI!
  - All other parameters are retained in the CONF file, as per the GUI.

Re the persistent non-updating of the INDEX files:
  - I looked, briefly, at the source.
  - There's one proc to create "empty" INDEX files, with a message that there's no data to graph - I'm assuming that proc is not being used (though ASS-U-ME could be tragic!!).
  - The proc "MakeIndexPages()" is defined in GRAPH.C, and called from BANDWIDTHD.C. It refreshes the INDEX files, and the SUBNET-xxx-xxx.HTML files. If it has trouble opening the INDEX files, a message should appear in the LOG files.

In my case:
  - I get no messages in the LOG file. Maybe "logging" is broken?
  - The INDEX.HTML files are created/overwritten - based on the time-stamp. If I delete the files, they're re-created. But, no data is retained in them. Perhaps some data is written, but the "closing" is broken?
  - The SUBNET-xx-xx.HTML files are NOT refreshed, through the code should attempt to re-build these also.
  - …suggesting that the proc blows up at an early stage??
  - File-Permissions look OK to me.
  - Inserting some traces, and re-compiling the package is probably beyond me...

...just in case anything here rings bells with others who have had similar experiences...

- Mike

[dakiller]

Hi, thanks for this great package, I have it up running just fine but have a little problem

I got 2 WAN's and a wired LAN and a wireless LAN that is bridged to the wired and as such with monitoring on the LAN I am seeing the local traffic between the wired and wireless. I see that there is a field to enter in filter rules but I got no idea how to format them to ignore local traffic. Googling hasn't turned up anything that I can follow, so has anyone got a link to a page that shows all the options and how to use them or can even just give me the command needed to filter local traffic.

Thanks,
Dan.

[sullrich]

Thats a good question.  I reinstalled the FreeBSD package and it did not contain a man page.  I'll poke around and see if I can locate something.

[sullrich]

Here is the contents of the readme file.

Programmed by David Hinkle, Commissioned by DerbyTech of Illinois. 
Special thanks goes to Brice Beaman at brice@beamans.org for releasing
the software, testing and debugging, Blaze at ts@spective.net for his
excellent logo, Andreas Henriksson for polishing, testing, fixing, and
all the guys at havok for distributing clue.

LISCENCE

You may use this software under any version of the GPL that is current
as of your download.  For exact terms and conditions please see www.gnu.org.

WHAT IT IS

Bandwidthd is a UNIX daemon/Windows service for graphing the traffic
generated by each machine on several configurable subnets.  It is much
easier to configure than MRTG, and provides significantly more useful
information.  MRTG only tells you how much bandwidth you are using,
Bandwidthd tells you that, and who is using it.

Each IP address that has moved any significant volume of traffic has its
own graph.  The graphs are color coded to help you figure out at a glance
if your user is surfing the web, or surfing Kazaa.

Bandwidthd is targeted to run on my routing platforms.  It is very low
overhead.  Easily graphing small business traffic on a 133Mhz Elan 486
every 2.5 minutes. My entire ISP (2000-3000 IP addresses across 4 states)
is graphed on a Celeron 450 every 10 minutes.

PORTABILITY

Bandwidthd compiles clean on:

ix86 Solaris 9
Debian 2.2
Fedora Core 2
OpenBSD 3.4
FreeBSD 4.8
NetBSD 1.6.1
AMD64 Fedora Core 3
PPC G4 MacOSX 10.2

Thanks goes to SourceForge for providing the test platforms.

CONFIGURATION INSTRUCTIONS

There are now two ways to install Bandwidthd.  The fast easy way, which uses
the built in Bandwidthd graphing system to generate static HTML pages and graphs,
and the much more complicated way that supports multiple sensors, stores it's
data in a back end database, and generates reports and graphs with easily
customized php scripts.

If you are new to Bandwidthd I would recommend just installing it the following the
instructions in the bandwidthd.conf file.  If you are interested in customizing
your output or you need a more scalable solution, you can always come back and
jump through the database hoops later.

See "DATABASE SUPPORT" for information on Bandwidthd's advanced configuration.

GRAPHING INTERVAL

Bandwidthd defaults to graph up to 4000 local IPs every 200 seconds.  If you need
to track more IPs, change IP_NUM in bandwidthd.h.

The weekly graph updates every 10 minutes, monthly every hour, and yearly every
12 hours.

A graphing run will automatically be "skipped" if that last one isn't finished
before the new one would begin.

CDF LOGGING

Bandwidthd can be made to log to CDF by setting "output_cdf" to true. This will
now log out each interval's traffic, so you can import them into a database and
use a tool like access to create your own graphs, or implement 95 percentile
billing, for example.  Sending Bandwidthd a HUP will cause it to rotate it's logs.
It will rotate out 5 times before deleting the oldest log file.

These logs are log.1.0.cdf-log.1.5.cdf for daily, log.2.0.cdf-log.2.5.cdf for
weekly, etc, etc.

If you are upgrading from an older version of Bandwidthd from before all 4 logs
rotated you must rename your log files for the new Bandwidthd to find them:

mv log.cdf log.1.0.cdf
mv log.1.cdf log.1.1.cdf
mv log.2.cdf log.1.2.cdf
mv log.3.cdf log.1.3.cdf
mv log.4.cdf log.1.4.cdf
mv log.5.cdf log.1.5.cdf
mv log2.cdf log.2.1.cdf
mv log3.cdf log.3.1.cdf
mv log4.cdf log.4.1.cdf

The log format is best documented in the "StoreIPDataInCDF" function in
bandwidthd.c.  As of this writing, it consists of one line for each  IP address
for each interval.  The line contains only data for the previous interval.

Fields:
IP Address,Timestamp,Total Sent,Icmp Sent,Udp Sent,Tcp Sent,Ftp Sent,Http Sent, P2P Sent,Total Received,Icmp Received,Udp Received,Tcp Received,Ftp Received,Http Received, P2P Received

HOW TO KEEP YOUR GRAPHS BETWEEN REBOOTS

Following is the easy way to keep your graphs between reboots.  You can also opt
to build and use bandwidthd with database support, as described in "DATABASE SUPPORT"
below.

In the bandwidthd.conf file set:
output_cdf true
recover_cdf true

output_cdf will cause Bandwidthd to log all of it's data to the log.cdf file
in it's directory.  recover_cdf will cause Bandwidthd to load that file when
it starts.  You will also want to make a crontab entry like so:

0 0 * * * * /bin/kill -HUP cat /var/run/bandwidthd.pid

This will send Bandwidthd a HUP every night at midnight.  When Bandwidthd
receives a HUP it schedules a rotation of it's log files during the next
graphing run.  Daily log files rotate each HUP.  Weekly/Monthly/Yearly log
files rotate every so many HUPs.  Log files get rotated out 5 times before
deletion.

Fyi, if you use killall instead of kill, each of the children will receive
the HUP command twice, causing them to rotate their log files twice as
often as they should.

GRAPHING

Also note that Bandwidthd does not bother to graph an IP that has transmitted
less than 1Mbit of data.  It does however, include that IP in the table of IPs
at the top of the page, so it's traffic can still be viewed.  This cutoff can be
changed by modifying graph_cutoff in bandwidthd.conf.  "graph_cutoff" is in
kilobytes.

Graphing can be disabled by setting "graph" to false.  This will still log, but
will use almost no ram or CPU cycles.

COLOR CODES

RED ICMP
BROWN UDP
YELLOW IP ENCAPSULATED (IP over IP, IPSEC, most VPN's)
BLUE HTTP (Port 80 TCP, actually)
PURPLE Peer2Peer (Lots of TCP ports generally used by P2P software)
GREEN TCP

SPECIFYING THE LIBPCAP FILTER

if you'd like more control over what traffic is counted, you can specify a Manuel
filter to be passed to libpcap.  You can use this to remove certain IPs or only
graph certain IPs, or limit the graphs to certain kinds of traffic.  You should
always specify "ip" in the filter.  For example:

filter "ip and host 64.215.40.1"

HOW TO IMPROVE PERFORANCE

Bandwidthd's primary bottleneck in static HTML mode is the drawing of graphs for
IP addresses.  To improve bandwidthd's performance, therefore, the only thing you
can really do is reduce the number of graphs it has to draw in any given run. 
Adjust graph_cutoff in the bandwidthd.conf file in order to tune out the IP addresses
that don't use so much bandwidth.  These IP addresses will still have their data
displayed in the table at the top of the page, so you can still see what's going
on with them.

Alternatively, you can choose to graph less often.  Bandwidthd automatically skips
a graphing run if the last one is still going when the new one is scheduled to start. 
If you'd like to graph less often than your server is capeable of, you can edit
skip_intervals in bandwidthd.conf.  A value of 1means to skip every other interval,
3 would mean to skip three intervals between each run.  This doesn't disable
Bandwidthd's automatic skipping.

Also, you can choose to deploy Bandwidthd with database support, which provides
significant performance gains.

DATABASE SUPPORT

Since version 2.0, Bandwidthd now has support for external databases.  This system
consists of 3 major parts:

1. The Bandwidthd binary which acts as a sensor, recording traffic information and
storing it in a database across the network or on the local host.  In this mode
Bandwidthd uses very little ram and CPU. In addition, multiple sensors can record
to the same database.

2. The database system.  Currently Bandwidthd only supports Postgresql.

3. The webserver and php application.  Bundled with Bandwidthd in the "phphtdocs"
directory is a php application that reports on and graphs the contents of the database. 
This has been designed to be easy to customize.  Everything is passed around on the urls,
just tinker with it a little and you'll see how to generate custom graphs pretty easy.

Using Bandwidthd with a database has many advantages, such as much lower overhead, because
graphs are only graphed on demand.  And much more flexibility, SQL makes building new
reports easy, and php+sql greatly improves the interactivity of the reports.

My ISP has now switched over to the database driven version of bandwidthd entirely, we
have half a dozen sensors sprinkled around the country, writing millions of data points a
day on our customers into the system.

INSTRUCTIONS

As a prerequisite for these instructions, you must have Postgresql installed and working,
as well as a web server that supports php.

Database Setup:
1. Create a database for Bandwidthd.  You will need to create users that can access the
database remotely if you want remote sensors.

2. Bandwidthd's schema is in "schema.postgresql".  "psql mydb username < schema.postgresql"
should load it and create the 2 tables and 4 indexes.

Bandwidthd Setup:
1. Add the following lines to your bandwidthd.conf file:

Standard postgres connect string, just like php, see postgres docs for

details

pgsql_connect_string "user = someuser dbname = mydb host = databaseserver.com"

Arbitrary sensor name, I recommend the sensors fully qualified domain

name

sensor_id "sensor1.mycompany.com"

Tells Bandwidthd to keep no data and preform no graphing locally

graph false

If this is set to true Bandwidthd will try to recover the daily log

into the database.  If you set this true on purpose only do it once.

Bandwidthd does not track the fact that it has already transferred

certain records into the database.

recover_cdf false

4. Simply start bandwidthd, and after a few minutes data should start appearing in
your database.  If not, check syslog for error messages.

Web Server Setup:
1. Copy the contents of phphtdocs into your web tree some where.
2. Edit config.conf to set your db connect string

You should now be able to access the web application and see you graphs.  All graphing
is done by graph.php,  all parameters are passed to it in it's url.  You can create
custom urls to pull custom graphs from your own index pages, or use the canned
reporting system.

In addition, you should schedule bd_pgsql_purge.sh to run every so often.  I recomend
running it weekly.  This script outputs sql statements that aggregate the older
data points in your database in order to reduce the amount of data that needs to
be slogged through in order to generate yearly, monthly, and weekly graphs.

Example:
bd_pgsql_purge.sh | psql bandwidthd postgres

Will connect to the bandwidthd database on local host as the user postgres and summarize
the data.

KNOWN BUGS AND TROUBLESHOOTING

If Bandwidthd shows you nothing but a message saying "Bandwidthd has nothing to graph",
that means it currently has recorded no data.  The cause is most likely one of these:
1. It's first interval hasn't expired yet (2.5 minutes).
2. It is still chewing through large CDF logs.
3. Bandwidthd's host machine is on a switch and therefor isn't seeing any traffic not
  destined to or from or going through it.
4. You don't have a subnet line.
5. You have a subnet line but it doesn't match any of the packets Bandwidthd is seeing.
6. You have a filter line that is filtering out all the traffic Bandwidthd could be seeing.

Bandwidthd doesn't do a very good job of tracking ftp.  This is because only some ftp
server software follows the rfc standard of sourceing all ftp transfers from port 20. 
Surprisingly, Microsofts ftp daemon is compliant in this regard, whereas most open source
daemons are not.

Bandwidthd forks to perform it's graphing functions.  After this fork finishes it remains
as a zombie until the next interval, at which time it is reaped by the main process. 
This zombie is nothing to worry about, it's just an entry in the process table waiting
to be deleted.  With the new weekly, monthly, yearly graphs you'll have up to 4 zombies now.

By default, Bandwidthd now runs in promiscuous mode.  This means it can be used to monitor
traffic not routing through it's host.  Just make sure that the host and the actual router
are on the same hub (Not switch) and everything will be ok. Under some circumstances, traffic
may get counted twice.  If traffic routes to the other router, then routes back out along
the same wire, it may get counted twice by Bandwidthd.  This is much less of a problem than
you might think, due to a little known packet called an "icmp redirect" that causes all packets
after the first to go directly to it's target.  If you find that traffic looks like it's 
getting counted twice, make sure you're not firewalling off the icmp redirects. If you find that
you actually see none of this traffic, it may be because the icmp redirects cause it to ultimately
end up going from one port on a switch to another, never touching your hub. If you want
Bandwidthd to only see traffic actually going into and out of the host set "promiscuous" to
false in bandwidthd.conf.

Bandwidthd supports ethernet, Linux cooked sockets, raw, and token ring, and most ppp packet
encapsulation.  If you get an "Unknown Datalink Type" error, Bandwidthd has not been programed
to handle the physical encapsulation of the ip packets on that interface.  If you send me a
sample capture and a copy paste of the error message I'll see if I can make bandwidthd work for you.

[dakiller]

Thanks, but that doesn't explain how to write the filters

I need something that is going to ignore all traffic from my lan subnet to my lan subnet

[sullrich]

@dakiller:

Thanks, but that doesn't explain how to write the filters

I need something that is going to ignore all traffic from my lan subnet to my lan subnet

Sure it does…

SPECIFYING THE LIBPCAP FILTER

if you'd like more control over what traffic is counted, you can specify a Manuel
filter to be passed to libpcap.  You can use this to remove certain IPs or only
graph certain IPs, or limit the graphs to certain kinds of traffic.  You should
always specify "ip" in the filter.  For example:

filter "ip and host 64.215.40.1"