Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Pfflowd non correctly counts

    Scheduled Pinned Locked Moved pfSense Packages
    12 Posts 4 Posters 6.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      freeseacher
      last edited by

      has lead test 3 collectors
      1. pfflowd
      2. ng_netflow
      3. ndsad
      as a result it turns out that figure which has counted pfflowd differs in smaller I shall shift on 802 mb was considered so
      pfflowd
      /usr/local/sbin/pfflowd  -n 127.0.0.1:9800 -S any -v 5
      pass any in keep-state
      pass any out keep-state

      ndsad (based on pcap)
      force fxp0
      ignore all
      ip 127.0.0.1
      port 9802

      ng_netflow
      ngctl -f- <<-SEQ
                    mkpeer fxp0: tee lower left
                    connect fxp0: fxp0:lower upper right
                    mkpeer fxp0:lower one2many left2right many0
                    connect fxp0:lower.left2right fxp0:lower many1 right2left
                    name fxp0:lower.right2left o2m
                    mkpeer o2m: netflow one iface0
                    name o2m:one netflow
                    mkpeer netflow: ksocket export inet/dgram/udp
                    msg netflow:export connect inet/127.0.0.1:9801
      SEQ
      all 3 flows captured by
      flow-capture -N 0 -n 288 -p /var/run/flow0.pid -V 5 -z 9 -w /usr/local/flow_data/pf 127.0.0.1/127.0.0.1/9800
      flow-capture -N 0 -n 288 -p /var/run/flow1.pid -V 5 -z 9 -w /usr/local/flow_data/ng 127.0.0.1/127.0.0.1/9801
      flow-capture -N 0 -n 288 -p /var/run/flow2.pid -V 5 -z 9 -w /usr/local/flow_data/nd 127.0.0.1/127.0.0.1/9802
      then i run download grom some ftp server to get 13gb of data. so in a result i get inform for only one ip address
      xxx.xxx.3.192
      pfflowd 12.125mb
      ng_netflow 12.985mb
      ndsad 12.982mb

      Whether now I think has sense to use a collector which have counted not correct

      1 Reply Last reply Reply Quote 0
      • B
        buraglio
        last edited by

        So you believe pfflowd is not generating all the correct flow records and thus giving incomplete info?
        I have not done a direct comparison but I can test it when I get some time.  In the mean time, can you provide some more info?  What are you running the ng_netflow and ndsad on?  Always grabbing the same file from the remote location?  I gather from the commands that you're using flow-tools to capture the data, correct?  An 800mb difference is pretty substantial, it's feasable that pfflowd is missing something. What version of pfsense is it running on?

        https://www.forwardingplane.net/

        1 Reply Last reply Reply Quote 0
        • F
          freeseacher
          last edited by

          wget mirrors some ftp server
          3 collectors work on a same time
          yes i use flow-tools to grap info

          uname
          FreeBSD olc.local 6.1-STABLE FreeBSD 6.1-STABLE #0: Fri May 12 12:08:22 NOVST 2006    root@olc.local:/usr/obj/usr/src/sys/olc  i386

          now i have not so much time but theme is truly important
          maybe i miss some thing in a tests

          sorry for my english

          1 Reply Last reply Reply Quote 0
          • B
            buraglio
            last edited by

            This is likely something that should be brought up to the author of pfflowd.  He has a mailing list set up for discussion of such things.  You can check out some details at http://www.mindrot.org/pfflowd.html
            I'll be following this as well.  I also plan to replicate this test to see if I get the same results.

            nb

            https://www.forwardingplane.net/

            1 Reply Last reply Reply Quote 0
            • S
              sullrich
              last edited by

              You are obviously not running a pfSense kernel.

              Why not?

              1 Reply Last reply Reply Quote 0
              • B
                buraglio
                last edited by

                I'm wondering if he's just using a stock freebsd install.  ng_netflow is in the base install now.  Is this a pfsense box?  How hard would it be to add ng_netflow in the base pfsense?  It would be fairly useful on the embedded platforms where there is no package system.

                https://www.forwardingplane.net/

                1 Reply Last reply Reply Quote 0
                • S
                  sullrich
                  last edited by

                  @buraglio:

                  I'm wondering if he's just using a stock freebsd install.  ng_netflow is in the base install now.  Is this a pfsense box?  How hard would it be to add ng_netflow in the base pfsense?  It would be fairly useful on the embedded platforms where there is no package system.

                  I would be okay with this.

                  1 Reply Last reply Reply Quote 0
                  • F
                    freeseacher
                    last edited by

                    @sullrich:

                    You are obviously not running a pfSense kernel.

                    Why not?

                    i done that test on my own machine.
                    than now i repeat it on pffsense machine

                    i test it using that script

                    
                    # cat /root/bin/netflow.sh 
                    #!/bin/sh
                    
                    #load some kldmodules
                    kldload ng_netflow
                    kldload ng_ether
                    kldload ng_tee
                    kldload ng_one2many
                    
                    #for ndsad
                    kldload ng_ipfw
                    
                    # for ndsad copied from freebsd 6.1
                    kldload ipdivert.ko
                    
                    #pfflowd 0.6
                    /usr/local/sbin/pfflowd -n 127.0.0.1:9800 -S any -v 5
                    
                    #ng_netflow
                    ngctl -f- <<-SEQ
                                   mkpeer fxp0: tee lower left
                                   connect fxp0: fxp0:lower upper right
                                   mkpeer fxp0:lower one2many left2right many0
                                   connect fxp0:lower.left2right fxp0:lower many1 right2left
                                   name fxp0:lower.right2left o2m
                                   mkpeer o2m: netflow one iface0
                                   name o2m:one netflow
                                   mkpeer netflow: ksocket export inet/dgram/udp
                                   msg netflow:export connect inet/127.0.0.1:9801
                    SEQ
                    
                    #rule for ndsad
                    ipfw add 10 tee 21000 all from any to any
                    /usr/local/sbin/ndsad -d -c /usr/local/etc/ndsad.conf
                    
                    #% cat ndsad.conf | egrep -v "^#" | sort
                    #bsd_div_copy yes
                    #bsd_div_port 21000
                    #dummy all
                    #force fxp0
                    #hash all 32
                    #heap 65536
                    #ignore all
                    #ip     127.0.0.1
                    #log /tmp/ndsad.log
                    #port 9802
                    
                    #collector for pfflowd
                    /usr/local/bin/flow-capture -N 0 -p /var/run/flow0.pid -V 5 -w /usr/local/netflow/pfflowd 127.0.0.1/127.0.0.1/9800
                    #collector for ng_netflow
                    /usr/local/bin/flow-capture -N 0 -p /var/run/flow2.pid -V 5 -w /usr/local/netflow/ng_netflow 127.0.0.1/127.0.0.1/9801
                    #collector for ndsad
                    /usr/local/bin/flow-capture -N 0 -p /var/run/flow1.pid -V 5 -w /usr/local/netflow/ndsad 127.0.0.1/127.0.0.1/9802
                    
                    

                    and than get that results

                    
                    cat ng_netflow|flow-stat -f 9 -S 2
                    #  --- ---- ---- Report Information --- --- ---
                    #
                    # Fields:    Total
                    # Symbols:   Disabled
                    # Sorting:   Descending Field 2
                    # Name:      Source IP
                    #
                    # Args:      flow-stat -f 9 -S 2 
                    #
                    #
                    # IPaddr         flows                 octets                packets
                    #
                    xxx.xxx.3.192     6745                  19197196933           12824622
                    192.168.1.1      6698                  547846260             13412473
                    192.168.1.7      3570                  2855505               51996
                    192.168.1.21     1031                  330975                4080
                    192.168.1.10     4082                  314801                4147
                    192.168.1.29     1176                  313618                3862
                    192.168.1.20     109                   56382                 685
                    192.168.1.13     167                   43499                 258
                    192.168.1.15     273                   34335                 273
                    192.168.1.5      271                   33538                 271
                    192.168.1.23     83                    19182                 84
                    192.168.1.25     10                    1841                  13
                    192.168.1.24     4                     1384                  6
                    192.168.1.26     5                     1145                  5
                    192.168.133.200  2                     1008                  2
                    192.168.1.27     3                     980                   8
                    4.23.190.230     4                     304                   4
                    213.239.214.170  1                     76                    1
                    
                     cat ndsad|flow-stat -f 9 -S 2                       
                    #  --- ---- ---- Report Information --- --- ---
                    #
                    # Fields:    Total
                    # Symbols:   Disabled
                    # Sorting:   Descending Field 2
                    # Name:      Source IP
                    #
                    # Args:      flow-stat -f 9 -S 2 
                    #
                    #
                    # IPaddr         flows                 octets                packets
                    #
                    xxx.xxx.3.192     3394                  19197196933           12824622
                    192.168.1.1      3573                  547849008             13412456
                    192.168.1.7      1948                  2857053               52015
                    192.168.1.21     1339                  330975                4080
                    192.168.1.10     4077                  316256                4166
                    192.168.1.29     1324                  313618                3862
                    192.168.1.20     233                   58020                 706
                    192.168.1.13     166                   43967                 264
                    192.168.1.15     273                   34335                 273
                    192.168.1.5      271                   33538                 271
                    192.168.1.23     82                    19182                 84
                    192.168.1.25     21                    11073                 83
                    192.168.1.24     4                     1384                  6
                    192.168.1.26     5                     1145                  5
                    192.168.133.200  2                     1008                  2
                    192.168.1.27     3                     980                   8
                    4.23.190.230     4                     304                   4
                    213.239.214.170  1                     76                    1
                    
                    cat pfflowd|flow-stat -f 9 -S 2 
                    #  --- ---- ---- Report Information --- --- ---
                    #
                    # Fields:    Total
                    # Symbols:   Disabled
                    # Sorting:   Descending Field 2
                    # Name:      Source IP
                    #
                    # Args:      flow-stat -f 9 -S 2 
                    #
                    #
                    # IPaddr         flows                 octets                packets
                    #
                    xxx.xxx.3.192     3382                  19197196933           12824622
                    192.168.1.1      3389                  532997172             13315253
                    192.168.1.21     1319                  329805                4065
                    192.168.1.10     4046                  314617                4147
                    192.168.1.29     1316                  313150                3856
                    192.168.1.20     211                   57786                 703
                    192.168.1.13     149                   43738                 263
                    192.168.1.7      272                   34576                 272
                    192.168.1.15     273                   34174                 273
                    192.168.1.5      272                   33616                 272
                    192.168.1.23     80                    18724                 82
                    192.168.1.25     17                    10766                 81
                    192.168.1.24     4                     1384                  6
                    192.168.1.26     5                     1145                  5
                    192.168.133.200  2                     1008                  2
                    192.168.1.27     3                     980                   8
                    4.23.190.230     4                     304                   4
                    213.239.214.170  1                     76                    1
                    
                    

                    so as we can see
                    cat ng_netflow|flow-stat|grep "Total Octets"
                    Total Octets                    : 19749051766
                    cat ndsad|flow-stat|grep "Total Octets"
                    Total Octets                    : 19749068855
                    cat pfflowd|flow-stat|grep "Total Octets"         
                    Total Octets                    : 19731389954

                    1 Reply Last reply Reply Quote 0
                    • S
                      sullrich
                      last edited by

                      Can you test this on a virgin freebsd system?  Either way this really seems to be a FreeBSD issue so we'll have to gather lots of data for a freebsd-pf message.

                      1 Reply Last reply Reply Quote 0
                      • B
                        billm
                        last edited by

                        It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

                        pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

                        –Bill

                        pfSense core developer
                        blog - http://www.ucsecurity.com/
                        twitter - billmarquette

                        1 Reply Last reply Reply Quote 0
                        • F
                          freeseacher
                          last edited by

                          @billm:

                          It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

                          pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

                          –Bill

                          rules for pf was
                          pass any in keep-state
                          pass any out keep-state

                          is there someting to miss ?

                          1 Reply Last reply Reply Quote 0
                          • B
                            billm
                            last edited by

                            @freeseacher:

                            @billm:

                            It's likely that pfflowd is only counting stuff that matches state.  Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

                            pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages.  So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

                            –Bill

                            rules for pf was
                            pass any in keep-state
                            pass any out keep-state

                            is there someting to miss ?

                            Yes, my point ;)

                            Not all packets in a given TCP flow will be considered "in state".  Consider out of window packets, out of sequence window packets (stuff that's been ack'd and had data past it acked, but was retransmitted all the same).  "normal" TCP communications do have packets that will get blocked.  I'm reasonably confident that those packets will not cound against the PF byte count for that flow.  The easiest way to determine that is to see if the PF byte count more closely matches that of the file(s) that were transferred.  If it's under, then there's a bug somewhere, if it's over, but over by less than the other accounting types (ng_netflow is going to get all packets regardless of whether pf blocks it) then it's not a bug per se, you just have to understand what/where you're monitoring.

                            –Bill

                            pfSense core developer
                            blog - http://www.ucsecurity.com/
                            twitter - billmarquette

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.