Pfflowd non correctly counts



  • has lead test 3 collectors
    1. pfflowd
    2. ng_netflow
    3. ndsad
    as a result it turns out that figure which has counted pfflowd differs in smaller I shall shift on 802 mb was considered so
    pfflowd
    /usr/local/sbin/pfflowd  -n 127.0.0.1:9800 -S any -v 5
    pass any in keep-state
    pass any out keep-state

    ndsad (based on pcap)
    force fxp0
    ignore all
    ip 127.0.0.1
    port 9802

    ng_netflow
    ngctl -f- <<-SEQ
                  mkpeer fxp0: tee lower left
                  connect fxp0: fxp0:lower upper right
                  mkpeer fxp0:lower one2many left2right many0
                  connect fxp0:lower.left2right fxp0:lower many1 right2left
                  name fxp0:lower.right2left o2m
                  mkpeer o2m: netflow one iface0
                  name o2m:one netflow
                  mkpeer netflow: ksocket export inet/dgram/udp
                  msg netflow:export connect inet/127.0.0.1:9801
    SEQ
    all 3 flows captured by
    flow-capture -N 0 -n 288 -p /var/run/flow0.pid -V 5 -z 9 -w /usr/local/flow_data/pf 127.0.0.1/127.0.0.1/9800
    flow-capture -N 0 -n 288 -p /var/run/flow1.pid -V 5 -z 9 -w /usr/local/flow_data/ng 127.0.0.1/127.0.0.1/9801
    flow-capture -N 0 -n 288 -p /var/run/flow2.pid -V 5 -z 9 -w /usr/local/flow_data/nd 127.0.0.1/127.0.0.1/9802
    then i run download grom some ftp server to get 13gb of data. so in a result i get inform for only one ip address
    xxx.xxx.3.192
    pfflowd 12.125mb
    ng_netflow 12.985mb
    ndsad 12.982mb

    Whether now I think has sense to use a collector which have counted not correct



  • So you believe pfflowd is not generating all the correct flow records and thus giving incomplete info?
    I have not done a direct comparison but I can test it when I get some time.  In the mean time, can you provide some more info?  What are you running the ng_netflow and ndsad on?  Always grabbing the same file from the remote location?  I gather from the commands that you're using flow-tools to capture the data, correct?  An 800mb difference is pretty substantial, it's feasable that pfflowd is missing something. What version of pfsense is it running on?



  • wget mirrors some ftp server
    3 collectors work on a same time
    yes i use flow-tools to grap info

    uname
    FreeBSD olc.local 6.1-STABLE FreeBSD 6.1-STABLE #0: Fri May 12 12:08:22 NOVST 2006    root@olc.local:/usr/obj/usr/src/sys/olc  i386

    now i have not so much time but theme is truly important
    maybe i miss some thing in a tests

    sorry for my english



  • This is likely something that should be brought up to the author of pfflowd.  He has a mailing list set up for discussion of such things.  You can check out some details at http://www.mindrot.org/pfflowd.html
    I'll be following this as well.  I also plan to replicate this test to see if I get the same results.

    nb



  • You are obviously not running a pfSense kernel.

    Why not?



  • I'm wondering if he's just using a stock freebsd install.  ng_netflow is in the base install now.  Is this a pfsense box?  How hard would it be to add ng_netflow in the base pfsense?  It would be fairly useful on the embedded platforms where there is no package system.



  • @buraglio:

    I'm wondering if he's just using a stock freebsd install.  ng_netflow is in the base install now.  Is this a pfsense box?  How hard would it be to add ng_netflow in the base pfsense?  It would be fairly useful on the embedded platforms where there is no package system.

    I would be okay with this.



  • @sullrich:

    You are obviously not running a pfSense kernel.

    Why not?

    i done that test on my own machine.
    than now i repeat it on pffsense machine

    i test it using that script

    
    # cat /root/bin/netflow.sh 
    #!/bin/sh
    
    #load some kldmodules
    kldload ng_netflow
    kldload ng_ether
    kldload ng_tee
    kldload ng_one2many
    
    #for ndsad
    kldload ng_ipfw
    
    # for ndsad copied from freebsd 6.1
    kldload ipdivert.ko
    
    #pfflowd 0.6
    /usr/local/sbin/pfflowd -n 127.0.0.1:9800 -S any -v 5
    
    #ng_netflow
    ngctl -f- <<-SEQ
                   mkpeer fxp0: tee lower left
                   connect fxp0: fxp0:lower upper right
                   mkpeer fxp0:lower one2many left2right many0
                   connect fxp0:lower.left2right fxp0:lower many1 right2left
                   name fxp0:lower.right2left o2m
                   mkpeer o2m: netflow one iface0
                   name o2m:one netflow
                   mkpeer netflow: ksocket export inet/dgram/udp
                   msg netflow:export connect inet/127.0.0.1:9801
    SEQ
    
    #rule for ndsad
    ipfw add 10 tee 21000 all from any to any
    /usr/local/sbin/ndsad -d -c /usr/local/etc/ndsad.conf
    
    #% cat ndsad.conf | egrep -v "^#" | sort
    #bsd_div_copy yes
    #bsd_div_port 21000
    #dummy all
    #force fxp0
    #hash all 32
    #heap 65536
    #ignore all
    #ip     127.0.0.1
    #log /tmp/ndsad.log
    #port 9802
    
    #collector for pfflowd
    /usr/local/bin/flow-capture -N 0 -p /var/run/flow0.pid -V 5 -w /usr/local/netflow/pfflowd 127.0.0.1/127.0.0.1/9800
    #collector for ng_netflow
    /usr/local/bin/flow-capture -N 0 -p /var/run/flow2.pid -V 5 -w /usr/local/netflow/ng_netflow 127.0.0.1/127.0.0.1/9801
    #collector for ndsad
    /usr/local/bin/flow-capture -N 0 -p /var/run/flow1.pid -V 5 -w /usr/local/netflow/ndsad 127.0.0.1/127.0.0.1/9802
    
    

    and than get that results

    
    cat ng_netflow|flow-stat -f 9 -S 2
    #  --- ---- ---- Report Information --- --- ---
    #
    # Fields:    Total
    # Symbols:   Disabled
    # Sorting:   Descending Field 2
    # Name:      Source IP
    #
    # Args:      flow-stat -f 9 -S 2 
    #
    #
    # IPaddr         flows                 octets                packets
    #
    xxx.xxx.3.192     6745                  19197196933           12824622
    192.168.1.1      6698                  547846260             13412473
    192.168.1.7      3570                  2855505               51996
    192.168.1.21     1031                  330975                4080
    192.168.1.10     4082                  314801                4147
    192.168.1.29     1176                  313618                3862
    192.168.1.20     109                   56382                 685
    192.168.1.13     167                   43499                 258
    192.168.1.15     273                   34335                 273
    192.168.1.5      271                   33538                 271
    192.168.1.23     83                    19182                 84
    192.168.1.25     10                    1841                  13
    192.168.1.24     4                     1384                  6
    192.168.1.26     5                     1145                  5
    192.168.133.200  2                     1008                  2
    192.168.1.27     3                     980                   8
    4.23.190.230     4                     304                   4
    213.239.214.170  1                     76                    1
    
     cat ndsad|flow-stat -f 9 -S 2                       
    #  --- ---- ---- Report Information --- --- ---
    #
    # Fields:    Total
    # Symbols:   Disabled
    # Sorting:   Descending Field 2
    # Name:      Source IP
    #
    # Args:      flow-stat -f 9 -S 2 
    #
    #
    # IPaddr         flows                 octets                packets
    #
    xxx.xxx.3.192     3394                  19197196933           12824622
    192.168.1.1      3573                  547849008             13412456
    192.168.1.7      1948                  2857053               52015
    192.168.1.21     1339                  330975                4080
    192.168.1.10     4077                  316256                4166
    192.168.1.29     1324                  313618                3862
    192.168.1.20     233                   58020                 706
    192.168.1.13     166                   43967                 264
    192.168.1.15     273                   34335                 273
    192.168.1.5      271                   33538                 271
    192.168.1.23     82                    19182                 84
    192.168.1.25     21                    11073                 83
    192.168.1.24     4                     1384                  6
    192.168.1.26     5                     1145                  5
    192.168.133.200  2                     1008                  2
    192.168.1.27     3                     980                   8
    4.23.190.230     4                     304                   4
    213.239.214.170  1                     76                    1
    
    cat pfflowd|flow-stat -f 9 -S 2 
    #  --- ---- ---- Report Information --- --- ---
    #
    # Fields:    Total
    # Symbols:   Disabled
    # Sorting:   Descending Field 2
    # Name:      Source IP
    #
    # Args:      flow-stat -f 9 -S 2 
    #
    #
    # IPaddr         flows                 octets                packets
    #
    xxx.xxx.3.192     3382                  19197196933           12824622
    192.168.1.1      3389                  532997172             13315253
    192.168.1.21     1319                  329805                4065
    192.168.1.10     4046                  314617                4147
    192.168.1.29     1316                  313150                3856
    192.168.1.20     211                   57786                 703
    192.168.1.13     149                   43738                 263
    192.168.1.7      272                   34576                 272
    192.168.1.15     273                   34174                 273
    192.168.1.5      272                   33616                 272
    192.168.1.23     80                    18724                 82
    192.168.1.25     17                    10766                 81
    192.168.1.24     4                     1384                  6
    192.168.1.26     5                     1145                  5
    192.168.133.200  2                     1008                  2
    192.168.1.27     3                     980                   8
    4.23.190.230     4                     304                   4
    213.239.214.170  1                     76                    1
    
    

    so as we can see
    cat ng_netflow|flow-stat|grep "Total Octets"
    Total Octets                    : 19749051766
    cat ndsad|flow-stat|grep "Total Octets"
    Total Octets                    : 19749068855
    cat pfflowd|flow-stat|grep "Total Octets"         
    Total Octets                    : 19731389954



  • Can you test this on a virgin freebsd system?  Either way this really seems to be a FreeBSD issue so we'll have to gather lots of data for a freebsd-pf message.



  • It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

    pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

    –Bill



  • @billm:

    It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

    pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

    –Bill

    rules for pf was
    pass any in keep-state
    pass any out keep-state

    is there someting to miss ?



  • @freeseacher:

    @billm:

    It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

    pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

    –Bill

    rules for pf was
    pass any in keep-state
    pass any out keep-state

    is there someting to miss ?

    Yes, my point ;)

    Not all packets in a given TCP flow will be considered "in state".  Consider out of window packets, out of sequence window packets (stuff that's been ack'd and had data past it acked, but was retransmitted all the same).  "normal" TCP communications do have packets that will get blocked.  I'm reasonably confident that those packets will not cound against the PF byte count for that flow.  The easiest way to determine that is to see if the PF byte count more closely matches that of the file(s) that were transferred.  If it's under, then there's a bug somewhere, if it's over, but over by less than the other accounting types (ng_netflow is going to get all packets regardless of whether pf blocks it) then it's not a bug per se, you just have to understand what/where you're monitoring.

    –Bill


Log in to reply