Vnstat "like" package to monitor bandwidth usage PER LAN IP {NOW $280USD}

[Guest]

I'm using squid with a non-transparent proxy in place, in combination with vnstat, works great. Very little details are left out. I read somewhere that you can use a transparent proxy similar to a non by having a file (pac or something) with the configuration pre-defined. I think internet explorer's "automatically detect settings" looks for this file.

[Alan87i]

Now at $230.00

[romulous]

I've been working commercially with netflow for some years now and have implemented systems which cope with billions of flows per day, calculating hundreds of customers data plans. While those systems run from cisco routers, this can be done under FreeBSD fairly easily and can monitor all the interfaces individually and then filter/aggregate them based on subnets, IP's, ports etc. The only failing currently with flow tools is no IPv6 support. I have built netflow v9 gernerator/capture software (which supports IPv6) in php to gain an understanding of it. Based on all of this knowledge, I am sure I can contribute something to the pfSense project, if so desired.

[Alan87i]

I've been running PRTG on a windows XP box and have had poor results with softflowd V9 data. PRTG displays a small percentage of the actual data used.
I have switched back to the PFsense package Pfflowd. Witch seems to send data V9 that PRTG can interpret correctly. But PRTG has issues with the flow time outs when using Pfflowd.
I also have an issue with filtering local traffic where a PC running pf sense has 2 lans. One being a static route too another network with it's own WAN gateway.
If a package that exported flow data could be configured too only export flows between X and Y interfaces leaving Z interface out of the picture I would love to try it!

It seems that PRTG can not filter an interface with Letters in it's name. Mine for example is (EM1) and (EM2) and (BGE0).

[wallabybob]

If you have a Linux or Unix system available you could run flow-tools (home page http://code.google.com/p/flow-tools) to analyse your flow records. There is a variety of filter and report options. There is a reasonable writeup in Network Flow Analysis by Michael W Lucas, ISBN 978-1-59327-203-6

I run pfflowd on pfSense, direct the flow records to a collector program (flow-capture) on a Linux system and use flow-report and flow-nfilter on the Linux system to generate reports.

[RedRep]

I would kick in $20.

[serialdie]

Ill pitch in another $20.00

[slth]

Any updates on this matter? I find it hard to believe something as simple as bandwidth usage statistics per IP would be so hard to develop..

I'll trow in another 10$

[xbipin]

that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that

[Alan87i]

@xbipin:

that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that

Yes please someone give us an idea of what it would take to get this rolling. I think a package like this should be a part of PFsense.

[slth]

Yesterday, I had a chat with the vnStat author. I asked him if he had any plans to implement this feature directly into vnStat, here are some excerpts from our conversation:

<vergo>that would require a complete rewrite since the linux kernel doesn't provide that information directly. I wouldn't integrate that sort of feature into vnStat
<vergo>the thing is, the kernel provides the information about traffic per interface directly so vnStat can just query it and sleep between the queries
<vergo>filtering traffic per ip would require inspecting every packet and that's a totally different thing
<vergo>it might be possible to cheat a little bit and use iptables for getting the data but the end result wouldn't work in anything else than linux and even that would have some restrictions</vergo></vergo></vergo></vergo>

I asked if he had any idea of something we are looking for already exists for FreeBSD:

<vergo>I've had some plans for writing at least some kind of proof of concept program for doing per ip stats with a console based program but haven't so far found time to start it
<vergo>darkstats is the closest there currently is and it isn't exactly what you are searching since it's also filtering target ips, doesn't provide simple stats from console and can't survive a restart</vergo></vergo>

So I guess we are pretty much out of luck with this bounty, as far as my understanding goes, an entirely new package is needed to accomplish the listing of usage per IP..  :(

[Alan87i]

Thanks for the Info.!
I'll update the topic subject and revise this bounty for those keeping track.

A vnstat "like" package to monitor bandwidth usage PER LAN IP

[Jimmy_uk]

I will post a further $20.00 for the development of this feature/package.

[wallabybob]

I was asked to elaborate on my earlier post about flow tools to get per IP usage stats.

pfSense needs a flow collector installed. I used pfflowd. pfflowd sends flow records to a collector. I used flow-capture from the flow-tools package which I installed on a Linux system. flow-capture stores its flow records in directories, one for each day.  Mostly I'm interested in finding out who has used the most data during a day so I can take appropriate action if the monthly download quota looks like being exceeded. My ISP makes available daily usage stats and from them I can see about 9GB was downloaded on 19 Nov. So to see who was downloading and from where on the Linux system holding my flow records I can:

[root@sme ~]# pushd /var/db/flows/2011/2011-11/2011-11-19/
/var/db/flows/2011/2011-11/2011-11-19 ~
[root@sme 2011-11-19]# flow-cat * | flow-report -v TYPE=ip-source/destination-address/ip-source/destination-port -v SORT=+octets | more
#  –- ---- ---- Report Information --- --- ---

build-version:        flow-tools 0.68

name:                default

type:                ip-source/destination-address/ip-source/destination-port

options:              +header,+xheader,+totals

ip-src-addr-type:    address

ip-dst-addr-type:    address

sort_field:          +octets

fields:              +key1,+key2,+key3,+key4,+flows,+octets,+packets,+duration,+other

records:              165068

first-flow:          1321624808 Sat Nov 19 00:00:08 2011

last-flow:            1321711187 Sat Nov 19 23:59:47 2011

now:                  1322602258 Wed Nov 30 07:30:58 2011

mode:                streaming

compress:            off

byte order:          little

stream version:      3

export version:      5

#  ['/usr/bin/flow-rptfmt', '-f', 'ascii']
ip-source-address ip-destination-address ip-source-port ip-destination-port flows octets    packets duration
64.188.166.206    192.168.211.244        6881          6881                4    282428402 205832  4594000
173.194.28.84    192.168.211.244        80            51905              2    52110568  35902  468000 
58.174.20.228    192.168.211.244        25565          58525              2    38788562  61226  2782000
12.129.255.100    192.168.211.244        3724          56975              2    31560842  357082  7366000
74.125.109.182    192.168.211.244        80            52042              2    25368268  17660  376000 
125.252.225.176  192.168.211.244        80            58396              2    25020948  17238  978000 
117.121.249.80    192.168.211.244        80            52876              2    23684584  16464  522000 
195.8.214.79      192.168.211.244        80            50283              2    21343766  14708  578000 
12.120.15.208    192.168.211.244        80            52877              2    21147556  14578  520000 
125.252.225.176  192.168.211.244        80            58423              2    18952452  13060  2258000
125.252.225.176  192.168.211.244        80            58380              2    18219946  12566  866000 
117.121.249.75    192.168.211.244        80            57241              2    17291682  11948  462000 
173.194.28.106    192.168.211.244        80            51947              2    16064040  11196  392000 
195.8.214.37      192.168.211.244        80            58489              2    15804278  10890  445000 
117.121.249.81    192.168.211.244        80            52620              2    15645356  10894  386000 
125.252.225.151  192.168.211.244        80            52377              2    14250122  9828    354000 
12.120.14.206    192.168.211.244        80            52606              2    14162172  9764    422000 
125.252.225.152  192.168.211.244        80            52431              2    13742162  9576    466000 
125.252.225.152  192.168.211.244        80            52432              2    13539082  9430    466000 
74.125.109.143    192.168.211.244        80            49399              2    13522672  9392    298000 
125.252.225.151  192.168.211.244        80            52874              2    11739240  8098    526000 
74.125.10.15      192.168.211.244        80            49470              2    11368880  7976    278000 
125.252.225.152  192.168.211.244        80            52600              2    11335216  7818    450000 
192.168.211.244  222.154.97.65          6881          6881                22    10993548  13430  4250000
64.233.183.132    192.168.211.216        443            42135              2    10846542  7822    1804000
192.168.211.244  64.188.166.206        6881          6881                4    10631398  207608  4594000
121.223.82.76    192.168.211.244        6881          6881                2    10495864  9388    3062000
125.252.225.151  192.168.211.244        80            52397              2    9478840  6536    360000 
12.129.255.91    192.168.211.244        3724          57334              2    9141684  103534  2410000
192.168.211.244  12.129.255.100        56975          3724                2    8131908  178936  7366000
222.154.97.65    192.168.211.244        6881          6881                20    7941028  13162  4184000
195.8.214.22      192.168.211.244        80            58467              2    6694386  4618    440000 
173.194.28.113    192.168.211.244        80            51989              2    6375462  4456    344000 
125.252.225.151  192.168.211.244        80            53094              2    5851168  4186    1024000

flow-cat reads a bunch of flow files and removes headers and writes a stream of flow records to stdout.  There is a flow-nfilter program which can strip specified flow records from the stream (e.g. flows between LAN and OPT1, flows over specified time intervals). flow-report has a number of reporting and sorting options. There are more advanced reporting options discussed in the book including graphing options.

I would like to take a look at software described in http://www.manageengine.com/products/netflow/ which seems to have much more extensive reporting capability. It is commercial software but there is a free edition which handles a limited number of interfaces.

[Alan87i]

If you sort this forum but Most views count this topic is at the top of the list under the sticky's .
Any response from the pfsense team?
Is there a chance you could work this feature into the OS. I'm sick of running 2 computers just to monitor bandwidth. I just want a list per month of all lan too wan traffic sorted by lan IP.
If you commit I'll send the coin asap!

[xbipin]

im ready to pledge some more coins if some1 seriously is willing to complete this and im sure others will add more of the coins once we know what it would take to complete it

[akghetto]

As I understand it, said package would monitor total bandwidth by IP across multiple NICs, so if I have multiple internal subnets routing through pfsense, I can monitor how much IPs on the multiple LANs are transferring in and out across my WAN link(s).  If this understanding is accurate, count me in for $50 if said package would also support IPv6/pfsense 2.1.  I need IPv6 accounting as well.

[Alan87i]

Well It's been just over 1 year and over 7000 views on this topic . And not one hint of anything from the pf team.

[marsboer]

To achieve this I use pfSense as a netflow collector (using softflowd) exporting the data to nfsen (running on another machine). This gives me full analyzing capabilities using a web GUI. This gives you all the capabilities that is asked for here in a free open source way.

But the goal is perhaps to use pfSense only.

[Alan87i]

@marsboer:

But the goal is perhaps to use pfSense only.

Exactly! Why do we need two power sucking devices for this 1 job