Vnstat "like" package to monitor bandwidth usage PER LAN IP {NOW $280USD}


  • I will post 50.00 to start it off . I have read other posts with members looking for this as well.
    Please jump on board So someone with the skills to write this package has the motivation to try it.

    I would like a accurate monitoring per IP package . Bandwidthd does not cut the mustard at all .
    Vnstat works great for the total per month on the wan port. I would like the same output listed per lan IP by month updated at 24 hour intervals.

    Please note the bottom few posts on page 2.

    We are looking for a new package that works sort of the way vnstat does with a usuage table for lan users based on there lan IP.


  • $50 from me too for this


  • Do you mean source or destination ip, or both? Or one of the ips of the pfSense box, no matter if they are source or destination?


  • i would say the list based on all LAN ip and not the ips that they try to connect on the internet, so basically if i have 10 LAN than a list with those 10 and their usage.


  • @xbipin:

    i would say the list based on all LAN ip and not the ips that they try to connect on the internet, so basically if i have 10 LAN than a list with those 10 and their usage.

    That's my thinking just a simple list of total in/out combined from lan to wan traffic PER lan IP  , Not internal and not lan to opt1,2,or 3. Have it count by months or days with a starting day. Keeping last 30 days and current up to date data is plenty. I can grab the screen and save a jpeg or copy to a notepad document each month. Or open the file from the GUI.
    I have a lot of internal traffic from lan to opt1 on two networks with two PFsense PC routers.  We are getting fined for going over the cap every other month.

    I have tried 3 netflow analyzers and all of them count the local traffic , I have tried filtering it out with no luck.  Plus it takes an extra PC. I have filtered Bandwidthd and it still counts the lan/opt1 traffic and or it's way off track counting almost double of what the ISP claims we use.


  • I'll put in $30USD.  Pfsense needs a better bandwidth tracking (netop is broken, bandwidthd sucks and has not been worked on in years).  I can't code, but I can contribute a bit.


  • so we have $130 in total, any1 ready to code?

    mayb more would be willing to contribute if some1 was ready to do it and set a target price


  • I might chip in if there was a target Price


  • Why bandwidthd sucks? Doesn't it do bandwidth consumption per IP?
    EDIT scratch it.

  • Rebel Alliance Developer Netgate

    Not that it's exactly relevant, and it's a lot heavier, but ntop should be working now too. It's the 800 pound gorilla of network monitoring, but it can also have the temper of one.


  • @Alan87i:

    I will post 50.00 to start it off . I have read other posts with members looking for this as well.
    Please jump on board So someone with the skills to write this package has the motivation to try it.

    I would like a accurate monitoring per IP package . Bandwidthd does not cut the mustard at all .
    Vnstat works great for the total per month on the wan port. I would like the same output listed per lan IP by month updated at 24 hour intervals.

    Alright I pledge 100.00 instead of 50.00 !
    Could really use this.
    Allan


  • pretty new to these parts and to pfsense but if $50 would help the cause I'm in for $50.

    I would really like to be able to monitor per IP.

    I would add - this feature has the possibility to increase my income a bunch and I will be willing to donate a portion of that directly to an individual/group or this board. I would need minute or second breakdown (maybe 5-min is fine but that is max) per IP. The more data the better.

    So I'll post/pledge $50 and offer up a share in the $$ this will generate for me which I want to make clear could be substancial - i will be happy to face the fourm/board on this as well as pay the originator her/his cut.

    heh… glad I hopped into this board tonight...


  • I'm using squid with a non-transparent proxy in place, in combination with vnstat, works great. Very little details are left out. I read somewhere that you can use a transparent proxy similar to a non by having a file (pac or something) with the configuration pre-defined. I think internet explorer's "automatically detect settings" looks for this file.


  • Now at $230.00


  • I've been working commercially with netflow for some years now and have implemented systems which cope with billions of flows per day, calculating hundreds of customers data plans. While those systems run from cisco routers, this can be done under FreeBSD fairly easily and can monitor all the interfaces individually and then filter/aggregate them based on subnets, IP's, ports etc. The only failing currently with flow tools is no IPv6 support. I have built netflow v9 gernerator/capture software (which supports IPv6) in php to gain an understanding of it. Based on all of this knowledge, I am sure I can contribute something to the pfSense project, if so desired.


  • I've been running PRTG on a windows XP box and have had poor results with softflowd V9 data. PRTG displays a small percentage of the actual data used.
    I have switched back to the PFsense package Pfflowd. Witch seems to send data V9 that PRTG can interpret correctly. But PRTG has issues with the flow time outs when using Pfflowd.
    I also have an issue with filtering local traffic where a PC running pf sense has 2 lans. One being a static route too another network with it's own WAN gateway.
    If a package that exported flow data could be configured too only export flows between X and Y interfaces leaving Z interface out of the picture I would love to try it!

    It seems that PRTG can not filter an interface with Letters in it's name. Mine for example is (EM1) and (EM2) and (BGE0).


  • If you have a Linux or Unix system available you could run flow-tools (home page http://code.google.com/p/flow-tools) to analyse your flow records. There is a variety of filter and report options. There is a reasonable writeup in Network Flow Analysis by Michael W Lucas, ISBN 978-1-59327-203-6

    I run pfflowd on pfSense, direct the flow records to a collector program (flow-capture) on a Linux system and use flow-report and flow-nfilter on the Linux system to generate reports.


  • I would kick in $20.


  • Ill pitch in another $20.00


  • Any updates on this matter? I find it hard to believe something as simple as bandwidth usage statistics per IP would be so hard to develop..

    I'll trow in another 10$


  • that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that


  • @xbipin:

    that makes it $280 but still no1 to do it, probably some1 should mention what their target amount is and then can try and meet that

    Yes please someone give us an idea of what it would take to get this rolling. I think a package like this should be a part of PFsense.


  • Yesterday, I had a chat with the vnStat author. I asked him if he had any plans to implement this feature directly into vnStat, here are some excerpts from our conversation:

    <vergo>that would require a complete rewrite since the linux kernel doesn't provide that information directly. I wouldn't integrate that sort of feature into vnStat
    <vergo>the thing is, the kernel provides the information about traffic per interface directly so vnStat can just query it and sleep between the queries
    <vergo>filtering traffic per ip would require inspecting every packet and that's a totally different thing
    <vergo>it might be possible to cheat a little bit and use iptables for getting the data but the end result wouldn't work in anything else than linux and even that would have some restrictions</vergo></vergo></vergo></vergo>

    I asked if he had any idea of something we are looking for already exists for FreeBSD:

    <vergo>I've had some plans for writing at least some kind of proof of concept program for doing per ip stats with a console based program but haven't so far found time to start it
    <vergo>darkstats is the closest there currently is and it isn't exactly what you are searching since it's also filtering target ips, doesn't provide simple stats from console and can't survive a restart</vergo></vergo>

    So I guess we are pretty much out of luck with this bounty, as far as my understanding goes, an entirely new package is needed to accomplish the listing of usage per IP..  :(


  • Thanks for the Info.!
    I'll update the topic subject and revise this bounty for those keeping track.

    A vnstat "like" package to monitor bandwidth usage PER LAN IP


  • I will post a further $20.00 for the development of this feature/package.


  • I was asked to elaborate on my earlier post about flow tools to get per IP usage stats.

    pfSense needs a flow collector installed. I used pfflowd. pfflowd sends flow records to a collector. I used flow-capture from the flow-tools package which I installed on a Linux system. flow-capture stores its flow records in directories, one for each day.  Mostly I'm interested in finding out who has used the most data during a day so I can take appropriate action if the monthly download quota looks like being exceeded. My ISP makes available daily usage stats and from them I can see about 9GB was downloaded on 19 Nov. So to see who was downloading and from where on the Linux system holding my flow records I can:

    [root@sme ~]# pushd /var/db/flows/2011/2011-11/2011-11-19/
    /var/db/flows/2011/2011-11/2011-11-19 ~
    [root@sme 2011-11-19]# flow-cat * | flow-report -v TYPE=ip-source/destination-address/ip-source/destination-port -v SORT=+octets | more
    #  –- ---- ---- Report Information --- --- ---

    build-version:        flow-tools 0.68

    name:                default

    type:                ip-source/destination-address/ip-source/destination-port

    options:              +header,+xheader,+totals

    ip-src-addr-type:    address

    ip-dst-addr-type:    address

    sort_field:          +octets

    fields:              +key1,+key2,+key3,+key4,+flows,+octets,+packets,+duration,+other

    records:              165068

    first-flow:          1321624808 Sat Nov 19 00:00:08 2011

    last-flow:            1321711187 Sat Nov 19 23:59:47 2011

    now:                  1322602258 Wed Nov 30 07:30:58 2011

    mode:                streaming

    compress:            off

    byte order:          little

    stream version:      3

    export version:      5

    #  ['/usr/bin/flow-rptfmt', '-f', 'ascii']
    ip-source-address ip-destination-address ip-source-port ip-destination-port flows octets    packets duration
    64.188.166.206    192.168.211.244        6881          6881                4    282428402 205832  4594000
    173.194.28.84    192.168.211.244        80            51905              2    52110568  35902  468000 
    58.174.20.228    192.168.211.244        25565          58525              2    38788562  61226  2782000
    12.129.255.100    192.168.211.244        3724          56975              2    31560842  357082  7366000
    74.125.109.182    192.168.211.244        80            52042              2    25368268  17660  376000 
    125.252.225.176  192.168.211.244        80            58396              2    25020948  17238  978000 
    117.121.249.80    192.168.211.244        80            52876              2    23684584  16464  522000 
    195.8.214.79      192.168.211.244        80            50283              2    21343766  14708  578000 
    12.120.15.208    192.168.211.244        80            52877              2    21147556  14578  520000 
    125.252.225.176  192.168.211.244        80            58423              2    18952452  13060  2258000
    125.252.225.176  192.168.211.244        80            58380              2    18219946  12566  866000 
    117.121.249.75    192.168.211.244        80            57241              2    17291682  11948  462000 
    173.194.28.106    192.168.211.244        80            51947              2    16064040  11196  392000 
    195.8.214.37      192.168.211.244        80            58489              2    15804278  10890  445000 
    117.121.249.81    192.168.211.244        80            52620              2    15645356  10894  386000 
    125.252.225.151  192.168.211.244        80            52377              2    14250122  9828    354000 
    12.120.14.206    192.168.211.244        80            52606              2    14162172  9764    422000 
    125.252.225.152  192.168.211.244        80            52431              2    13742162  9576    466000 
    125.252.225.152  192.168.211.244        80            52432              2    13539082  9430    466000 
    74.125.109.143    192.168.211.244        80            49399              2    13522672  9392    298000 
    125.252.225.151  192.168.211.244        80            52874              2    11739240  8098    526000 
    74.125.10.15      192.168.211.244        80            49470              2    11368880  7976    278000 
    125.252.225.152  192.168.211.244        80            52600              2    11335216  7818    450000 
    192.168.211.244  222.154.97.65          6881          6881                22    10993548  13430  4250000
    64.233.183.132    192.168.211.216        443            42135              2    10846542  7822    1804000
    192.168.211.244  64.188.166.206        6881          6881                4    10631398  207608  4594000
    121.223.82.76    192.168.211.244        6881          6881                2    10495864  9388    3062000
    125.252.225.151  192.168.211.244        80            52397              2    9478840  6536    360000 
    12.129.255.91    192.168.211.244        3724          57334              2    9141684  103534  2410000
    192.168.211.244  12.129.255.100        56975          3724                2    8131908  178936  7366000
    222.154.97.65    192.168.211.244        6881          6881                20    7941028  13162  4184000
    195.8.214.22      192.168.211.244        80            58467              2    6694386  4618    440000 
    173.194.28.113    192.168.211.244        80            51989              2    6375462  4456    344000 
    125.252.225.151  192.168.211.244        80            53094              2    5851168  4186    1024000

    flow-cat reads a bunch of flow files and removes headers and writes a stream of flow records to stdout.  There is a flow-nfilter program which can strip specified flow records from the stream (e.g. flows between LAN and OPT1, flows over specified time intervals). flow-report has a number of reporting and sorting options. There are more advanced reporting options discussed in the book including graphing options.

    I would like to take a look at software described in http://www.manageengine.com/products/netflow/ which seems to have much more extensive reporting capability. It is commercial software but there is a free edition which handles a limited number of interfaces.


  • If you sort this forum but Most views count this topic is at the top of the list under the sticky's .
    Any response from the pfsense team?
    Is there a chance you could work this feature into the OS. I'm sick of running 2 computers just to monitor bandwidth. I just want a list per month of all lan too wan traffic sorted by lan IP.
    If you commit I'll send the coin asap!


  • im ready to pledge some more coins if some1 seriously is willing to complete this and im sure others will add more of the coins once we know what it would take to complete it


  • As I understand it, said package would monitor total bandwidth by IP across multiple NICs, so if I have multiple internal subnets routing through pfsense, I can monitor how much IPs on the multiple LANs are transferring in and out across my WAN link(s).  If this understanding is accurate, count me in for $50 if said package would also support IPv6/pfsense 2.1.  I need IPv6 accounting as well.


  • Well It's been just over 1 year and over 7000 views on this topic . And not one hint of anything from the pf team.


  • To achieve this I use pfSense as a netflow collector (using softflowd) exporting the data to nfsen (running on another machine). This gives me full analyzing capabilities using a web GUI. This gives you all the capabilities that is asked for here in a free open source way.

    But the goal is perhaps to use pfSense only.


  • @marsboer:

    But the goal is perhaps to use pfSense only.

    Exactly! Why do we need two power sucking devices for this 1 job


  • I'd like to contribute $50 towards this project.
    This would be very useful to me.


  • @Alan87i:

    @marsboer:

    But the goal is perhaps to use pfSense only.

    Exactly! Why do we need two power sucking devices for this 1 job

    I thought the same thing, but then looked at the 2 suggestions (have to go back and see if supported in freebsd).  It would work, and has many many features (maybe overkill).
    Could we not get what we all want if these 2 were bundled up in a pfsense package with some default options setup?).
      It may not be good for the embedded buys, but lately my hardware can more than handle this.
      It could be the only way we get what we all want.

    ??


  • Does anyone know what it would cost and how much time it would take to get this sort of thing built in to pfsense or for someone to do an addon package?

    Maybe the reason there hasnt been much movement is the cost and timescales especially if it would mean having to rewrite a large part of pfsense's core?

    I'd chip in a reasonable amount if need be, but some idea of costs & timescales would be useful to begin with imo.


  • @romulous:

    I've been working commercially with netflow for some years now and have implemented systems which cope with billions of flows per day, calculating hundreds of customers data plans. While those systems run from cisco routers, this can be done under FreeBSD fairly easily and can monitor all the interfaces individually and then filter/aggregate them based on subnets, IP's, ports etc. The only failing currently with flow tools is no IPv6 support. I have built netflow v9 gernerator/capture software (which supports IPv6) in php to gain an understanding of it. Based on all of this knowledge, I am sure I can contribute something to the pfSense project, if so desired.

    I would love to load this even if it were outside of pf.  Chris B suggested nfsen  - what is yours based upon?