Vnstat "like" package to monitor bandwidth usage PER LAN IP {NOW $280USD}

Alan87i

I've been running PRTG on a windows XP box and have had poor results with softflowd V9 data. PRTG displays a small percentage of the actual data used.
I have switched back to the PFsense package Pfflowd. Witch seems to send data V9 that PRTG can interpret correctly. But PRTG has issues with the flow time outs when using Pfflowd.
I also have an issue with filtering local traffic where a PC running pf sense has 2 lans. One being a static route too another network with it's own WAN gateway.
If a package that exported flow data could be configured too only export flows between X and Y interfaces leaving Z interface out of the picture I would love to try it!

It seems that PRTG can not filter an interface with Letters in it's name. Mine for example is (EM1) and (EM2) and (BGE0).

wallabybob

If you have a Linux or Unix system available you could run flow-tools (home page http://code.google.com/p/flow-tools) to analyse your flow records. There is a variety of filter and report options. There is a reasonable writeup in Network Flow Analysis by Michael W Lucas, ISBN 978-1-59327-203-6

I run pfflowd on pfSense, direct the flow records to a collector program (flow-capture) on a Linux system and use flow-report and flow-nfilter on the Linux system to generate reports.

RedRep

I would kick in $20.

serialdie

Ill pitch in another $20.00

slth

Any updates on this matter? I find it hard to believe something as simple as bandwidth usage statistics per IP would be so hard to develop..

I'll trow in another 10$

xbipin

that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that

Alan87i

@xbipin:

that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that

Yes please someone give us an idea of what it would take to get this rolling. I think a package like this should be a part of PFsense.

slth

Yesterday, I had a chat with the vnStat author. I asked him if he had any plans to implement this feature directly into vnStat, here are some excerpts from our conversation:

<vergo>that would require a complete rewrite since the linux kernel doesn't provide that information directly. I wouldn't integrate that sort of feature into vnStat
<vergo>the thing is, the kernel provides the information about traffic per interface directly so vnStat can just query it and sleep between the queries
<vergo>filtering traffic per ip would require inspecting every packet and that's a totally different thing
<vergo>it might be possible to cheat a little bit and use iptables for getting the data but the end result wouldn't work in anything else than linux and even that would have some restrictions</vergo></vergo></vergo></vergo>

I asked if he had any idea of something we are looking for already exists for FreeBSD:

<vergo>I've had some plans for writing at least some kind of proof of concept program for doing per ip stats with a console based program but haven't so far found time to start it
<vergo>darkstats is the closest there currently is and it isn't exactly what you are searching since it's also filtering target ips, doesn't provide simple stats from console and can't survive a restart</vergo></vergo>

So I guess we are pretty much out of luck with this bounty, as far as my understanding goes, an entirely new package is needed to accomplish the listing of usage per IP.. :(

Alan87i

Thanks for the Info.!
I'll update the topic subject and revise this bounty for those keeping track.

A vnstat "like" package to monitor bandwidth usage PER LAN IP

Jimmy_uk

I will post a further $20.00 for the development of this feature/package.

wallabybob

I was asked to elaborate on my earlier post about flow tools to get per IP usage stats.

pfSense needs a flow collector installed. I used pfflowd. pfflowd sends flow records to a collector. I used flow-capture from the flow-tools package which I installed on a Linux system. flow-capture stores its flow records in directories, one for each day. Mostly I'm interested in finding out who has used the most data during a day so I can take appropriate action if the monthly download quota looks like being exceeded. My ISP makes available daily usage stats and from them I can see about 9GB was downloaded on 19 Nov. So to see who was downloading and from where on the Linux system holding my flow records I can:

[root@sme ~]# pushd /var/db/flows/2011/2011-11/2011-11-19/
/var/db/flows/2011/2011-11/2011-11-19 ~
[root@sme 2011-11-19]# flow-cat * | flow-report -v TYPE=ip-source/destination-address/ip-source/destination-port -v SORT=+octets | more
# –- ---- ---- Report Information --- --- ---

build-version: flow-tools 0.68

name: default

type: ip-source/destination-address/ip-source/destination-port

options: +header,+xheader,+totals

ip-src-addr-type: address

ip-dst-addr-type: address

sort_field: +octets

fields: +key1,+key2,+key3,+key4,+flows,+octets,+packets,+duration,+other

records: 165068

first-flow: 1321624808 Sat Nov 19 00:00:08 2011

last-flow: 1321711187 Sat Nov 19 23:59:47 2011

now: 1322602258 Wed Nov 30 07:30:58 2011

mode: streaming

compress: off

byte order: little

stream version: 3

export version: 5

# ['/usr/bin/flow-rptfmt', '-f', 'ascii']
ip-source-address ip-destination-address ip-source-port ip-destination-port flows octets packets duration
64.188.166.206 192.168.211.244 6881 6881 4 282428402 205832 4594000
173.194.28.84 192.168.211.244 80 51905 2 52110568 35902 468000
58.174.20.228 192.168.211.244 25565 58525 2 38788562 61226 2782000
12.129.255.100 192.168.211.244 3724 56975 2 31560842 357082 7366000
74.125.109.182 192.168.211.244 80 52042 2 25368268 17660 376000
125.252.225.176 192.168.211.244 80 58396 2 25020948 17238 978000
117.121.249.80 192.168.211.244 80 52876 2 23684584 16464 522000
195.8.214.79 192.168.211.244 80 50283 2 21343766 14708 578000
12.120.15.208 192.168.211.244 80 52877 2 21147556 14578 520000
125.252.225.176 192.168.211.244 80 58423 2 18952452 13060 2258000
125.252.225.176 192.168.211.244 80 58380 2 18219946 12566 866000
117.121.249.75 192.168.211.244 80 57241 2 17291682 11948 462000
173.194.28.106 192.168.211.244 80 51947 2 16064040 11196 392000
195.8.214.37 192.168.211.244 80 58489 2 15804278 10890 445000
117.121.249.81 192.168.211.244 80 52620 2 15645356 10894 386000
125.252.225.151 192.168.211.244 80 52377 2 14250122 9828 354000
12.120.14.206 192.168.211.244 80 52606 2 14162172 9764 422000
125.252.225.152 192.168.211.244 80 52431 2 13742162 9576 466000
125.252.225.152 192.168.211.244 80 52432 2 13539082 9430 466000
74.125.109.143 192.168.211.244 80 49399 2 13522672 9392 298000
125.252.225.151 192.168.211.244 80 52874 2 11739240 8098 526000
74.125.10.15 192.168.211.244 80 49470 2 11368880 7976 278000
125.252.225.152 192.168.211.244 80 52600 2 11335216 7818 450000
192.168.211.244 222.154.97.65 6881 6881 22 10993548 13430 4250000
64.233.183.132 192.168.211.216 443 42135 2 10846542 7822 1804000
192.168.211.244 64.188.166.206 6881 6881 4 10631398 207608 4594000
121.223.82.76 192.168.211.244 6881 6881 2 10495864 9388 3062000
125.252.225.151 192.168.211.244 80 52397 2 9478840 6536 360000
12.129.255.91 192.168.211.244 3724 57334 2 9141684 103534 2410000
192.168.211.244 12.129.255.100 56975 3724 2 8131908 178936 7366000
222.154.97.65 192.168.211.244 6881 6881 20 7941028 13162 4184000
195.8.214.22 192.168.211.244 80 58467 2 6694386 4618 440000
173.194.28.113 192.168.211.244 80 51989 2 6375462 4456 344000
125.252.225.151 192.168.211.244 80 53094 2 5851168 4186 1024000

flow-cat reads a bunch of flow files and removes headers and writes a stream of flow records to stdout. There is a flow-nfilter program which can strip specified flow records from the stream (e.g. flows between LAN and OPT1, flows over specified time intervals). flow-report has a number of reporting and sorting options. There are more advanced reporting options discussed in the book including graphing options.

I would like to take a look at software described in http://www.manageengine.com/products/netflow/ which seems to have much more extensive reporting capability. It is commercial software but there is a free edition which handles a limited number of interfaces.

Alan87i

If you sort this forum but Most views count this topic is at the top of the list under the sticky's .
Any response from the pfsense team?
Is there a chance you could work this feature into the OS. I'm sick of running 2 computers just to monitor bandwidth. I just want a list per month of all lan too wan traffic sorted by lan IP.
If you commit I'll send the coin asap!

xbipin

im ready to pledge some more coins if some1 seriously is willing to complete this and im sure others will add more of the coins once we know what it would take to complete it

akghetto

As I understand it, said package would monitor total bandwidth by IP across multiple NICs, so if I have multiple internal subnets routing through pfsense, I can monitor how much IPs on the multiple LANs are transferring in and out across my WAN link(s). If this understanding is accurate, count me in for $50 if said package would also support IPv6/pfsense 2.1. I need IPv6 accounting as well.

Alan87i

Well It's been just over 1 year and over 7000 views on this topic . And not one hint of anything from the pf team.

marsboer

To achieve this I use pfSense as a netflow collector (using softflowd) exporting the data to nfsen (running on another machine). This gives me full analyzing capabilities using a web GUI. This gives you all the capabilities that is asked for here in a free open source way.

But the goal is perhaps to use pfSense only.

Alan87i

@marsboer:

But the goal is perhaps to use pfSense only.

Exactly! Why do we need two power sucking devices for this 1 job

Strider3000

I'd like to contribute $50 towards this project.
This would be very useful to me.

tester_02

@Alan87i:

@marsboer:

But the goal is perhaps to use pfSense only.

Exactly! Why do we need two power sucking devices for this 1 job

I thought the same thing, but then looked at the 2 suggestions (have to go back and see if supported in freebsd). It would work, and has many many features (maybe overkill).
Could we not get what we all want if these 2 were bundled up in a pfsense package with some default options setup?).
It may not be good for the embedded buys, but lately my hardware can more than handle this.
It could be the only way we get what we all want.

??

vitesse

Does anyone know what it would cost and how much time it would take to get this sort of thing built in to pfsense or for someone to do an addon package?

Maybe the reason there hasnt been much movement is the cost and timescales especially if it would mean having to rewrite a large part of pfsense's core?

I'd chip in a reasonable amount if need be, but some idea of costs & timescales would be useful to begin with imo.