Bandwidth Usage/Statistics Question

[cmbaker82]

Hello,

I've read through some posts in the forum and elsewhere about bandwidth monitoring and have tried the Traffic Graphs, RRD Graphs, ntop, and bandwidthd, but none seem to get me to where i want to go.

I run a small hosting company and need to track bandwidth usage.  I have a pfsense 2.0 (upgrading to 2.0.1) firewall that has public IPs on a single interface.  all servers are connected to the LAN interface.  all traffic is natted.
I have several services:
DNS
Exchange Email
HMail
IIS
Terminal Server

Also have backup software that offloads data elsewhere using http, and ftp servers

What I would like to see is something that can monitor all bandwidth usage and give me a break down.  Some of the tools i've looked at provide a break down by IP address and Port/protocol, but for the traffic coming to and from my web server and backup servers I need to be able to break down the traffic by URL as several sites share a common IP address.

For example I would like to specify a date range, say 6/1/12 to 7/1/12 and get a listing of highest bandwidth consumers (in GB and out GB reported separately)
Then lets say that reports that my internal server at 192.168.1.10 is using the most bandwidth I would like to break it down to protocol,
That would then tell me that HTTP sent out 200GB and received 100GB, and HTTPS received 300GB and sent out 200GB.
I then need to be able to break down the HTTP and HTTPS so I can see which urls are using the most.
For example I host the website for a local fair, once a year their bandwidth goes up considerably, so I would like to see how much they are using relative to my other sites, and the backup service that usings web services to send out mass quantities of data.

Does anyone know of any tool that I can use with PFSense to accomplish all of this?  I would prefer free if available, but not opposed to reasonably priced commercial solutions.

Thank you for your time in reading this and making suggestions.

[Nachtfalke]

Hi,

I am probably not the expert for that but I know that SQUID proxy can log and Lightsquid or Sarg can analyze the squid log for http and https access. And you can see how much traffic was transferred by URL/IP.

When squid is running in transparent mode it just filters http access but no need to configure anything on the clients. Perhaps you can start with that a see if the log analyzer do what you want. If it is ok and you need https then you need to configure squid in non-transparent mode and you need to configure the hosts to use squid as your proxy.

PS: Try with squid2 first - this is probably more stable than squid3 at the moment.

[cmbaker82]

Setting up squid as a transparent proxy will monitor both incoming and outgoing http traffic?

[cmb]

As a hosting provider, you should be using Netflow. I like nfsen as a free option for a netflow collector which has all the reporting capabilities you mentioned.

[cmbaker82]

Thank you Netflow seems like a good option.  What is the recommended way of exporting the netflow info from pfsense? I've seen pfflowd and softflowd but I am unsure of the differences

[cmbaker82]

So i tried pfflowd with Manageengines Netflow Analyzer.  This did not work as it displayed thousands of ifindex#### interfaces, and showed multiple terabytes of traffic in less then 10 minutes.

I uninstalled pfflowd and installed softflowd which seems to be working.  Started it with this command:
./softflowd -i em1 -v 9 -n 192.168.2.10:9996

em1 is my LAN interface, em0 is the wan interface
however in ManageEngines Analyzer it only shows inbound traffic
I tried starting a second copy with  ./softflowd -i em0 -v 9 -n 192.168.2.10:9996
but that seemed to just add the wan traffic to the incoming traffic section in manage engine.  it shows all traffic ifindex-1

Any ideas?

[Alan87i]

I gave up on managed engines and went with PRTG network monitor and softflowd .
I remember having issues with PFflowd.

[cmbaker82]

I've installed PRTG and it seems to be working, but It also seems to only show total bandwidth for the interface.
I added pfsense and setup a custom netflow v9 sensor.
I still can't figure out how to break down traffic by URL

[Alan87i]

In prtg you have to add a sensor for each IP
so one device ( the PF box
and sensors for this box
Ie in the Include filter you put    IP[192.168.25.41]  The Ip you want.

Or let it auto create sensors

[cmbaker82]

Pfsense is 192.168.2.1
Ok, so i have my webservers, internal ip of 192.168.2.40 and .46
both webservers serve several domains on each IP address, and I need to find out how much traffic each site is using, is what you are suggesting going to give me that kind of detail?

[Alan87i]

Yes
You can let it auto create sensors ant it should probe and find the servers and make snmp sensors . I think the newest version lets you add credentials so it can probe deeper and bring back more info such as cpu load disk status as well as traffic.

Or you can manually add a sensor too a device (pf box) Need to add the device first. and add a filter so so each sensor watches for only 1 IP
A simple filter is IP[192.168.2.40]
set flow time out too 6 or 10 minutes.

But try the auto create wizard first. If you put in the user/pw for the servers you might get all the info you need and more from that.