Pfflowd non correctly counts

buraglio

So you believe pfflowd is not generating all the correct flow records and thus giving incomplete info?
I have not done a direct comparison but I can test it when I get some time. In the mean time, can you provide some more info? What are you running the ng_netflow and ndsad on? Always grabbing the same file from the remote location? I gather from the commands that you're using flow-tools to capture the data, correct? An 800mb difference is pretty substantial, it's feasable that pfflowd is missing something. What version of pfsense is it running on?

freeseacher

wget mirrors some ftp server
3 collectors work on a same time
yes i use flow-tools to grap info

uname
FreeBSD olc.local 6.1-STABLE FreeBSD 6.1-STABLE #0: Fri May 12 12:08:22 NOVST 2006 root@olc.local:/usr/obj/usr/src/sys/olc i386

now i have not so much time but theme is truly important
maybe i miss some thing in a tests

sorry for my english

buraglio

This is likely something that should be brought up to the author of pfflowd. He has a mailing list set up for discussion of such things. You can check out some details at http://www.mindrot.org/pfflowd.html
I'll be following this as well. I also plan to replicate this test to see if I get the same results.

nb

sullrich

You are obviously not running a pfSense kernel.

Why not?

buraglio

I'm wondering if he's just using a stock freebsd install. ng_netflow is in the base install now. Is this a pfsense box? How hard would it be to add ng_netflow in the base pfsense? It would be fairly useful on the embedded platforms where there is no package system.

sullrich

@buraglio:

I'm wondering if he's just using a stock freebsd install. ng_netflow is in the base install now. Is this a pfsense box? How hard would it be to add ng_netflow in the base pfsense? It would be fairly useful on the embedded platforms where there is no package system.

I would be okay with this.

freeseacher

@sullrich:

You are obviously not running a pfSense kernel.

Why not?

i done that test on my own machine.
than now i repeat it on pffsense machine

i test it using that script


# cat /root/bin/netflow.sh 
#!/bin/sh

#load some kldmodules
kldload ng_netflow
kldload ng_ether
kldload ng_tee
kldload ng_one2many

#for ndsad
kldload ng_ipfw

# for ndsad copied from freebsd 6.1
kldload ipdivert.ko

#pfflowd 0.6
/usr/local/sbin/pfflowd -n 127.0.0.1:9800 -S any -v 5

#ng_netflow
ngctl -f- <<-SEQ
               mkpeer fxp0: tee lower left
               connect fxp0: fxp0:lower upper right
               mkpeer fxp0:lower one2many left2right many0
               connect fxp0:lower.left2right fxp0:lower many1 right2left
               name fxp0:lower.right2left o2m
               mkpeer o2m: netflow one iface0
               name o2m:one netflow
               mkpeer netflow: ksocket export inet/dgram/udp
               msg netflow:export connect inet/127.0.0.1:9801
SEQ

#rule for ndsad
ipfw add 10 tee 21000 all from any to any
/usr/local/sbin/ndsad -d -c /usr/local/etc/ndsad.conf

#% cat ndsad.conf | egrep -v "^#" | sort
#bsd_div_copy yes
#bsd_div_port 21000
#dummy all
#force fxp0
#hash all 32
#heap 65536
#ignore all
#ip     127.0.0.1
#log /tmp/ndsad.log
#port 9802

#collector for pfflowd
/usr/local/bin/flow-capture -N 0 -p /var/run/flow0.pid -V 5 -w /usr/local/netflow/pfflowd 127.0.0.1/127.0.0.1/9800
#collector for ng_netflow
/usr/local/bin/flow-capture -N 0 -p /var/run/flow2.pid -V 5 -w /usr/local/netflow/ng_netflow 127.0.0.1/127.0.0.1/9801
#collector for ndsad
/usr/local/bin/flow-capture -N 0 -p /var/run/flow1.pid -V 5 -w /usr/local/netflow/ndsad 127.0.0.1/127.0.0.1/9802

and than get that results


cat ng_netflow|flow-stat -f 9 -S 2
#  --- ---- ---- Report Information --- --- ---
#
# Fields:    Total
# Symbols:   Disabled
# Sorting:   Descending Field 2
# Name:      Source IP
#
# Args:      flow-stat -f 9 -S 2 
#
#
# IPaddr         flows                 octets                packets
#
xxx.xxx.3.192     6745                  19197196933           12824622
192.168.1.1      6698                  547846260             13412473
192.168.1.7      3570                  2855505               51996
192.168.1.21     1031                  330975                4080
192.168.1.10     4082                  314801                4147
192.168.1.29     1176                  313618                3862
192.168.1.20     109                   56382                 685
192.168.1.13     167                   43499                 258
192.168.1.15     273                   34335                 273
192.168.1.5      271                   33538                 271
192.168.1.23     83                    19182                 84
192.168.1.25     10                    1841                  13
192.168.1.24     4                     1384                  6
192.168.1.26     5                     1145                  5
192.168.133.200  2                     1008                  2
192.168.1.27     3                     980                   8
4.23.190.230     4                     304                   4
213.239.214.170  1                     76                    1

 cat ndsad|flow-stat -f 9 -S 2                       
#  --- ---- ---- Report Information --- --- ---
#
# Fields:    Total
# Symbols:   Disabled
# Sorting:   Descending Field 2
# Name:      Source IP
#
# Args:      flow-stat -f 9 -S 2 
#
#
# IPaddr         flows                 octets                packets
#
xxx.xxx.3.192     3394                  19197196933           12824622
192.168.1.1      3573                  547849008             13412456
192.168.1.7      1948                  2857053               52015
192.168.1.21     1339                  330975                4080
192.168.1.10     4077                  316256                4166
192.168.1.29     1324                  313618                3862
192.168.1.20     233                   58020                 706
192.168.1.13     166                   43967                 264
192.168.1.15     273                   34335                 273
192.168.1.5      271                   33538                 271
192.168.1.23     82                    19182                 84
192.168.1.25     21                    11073                 83
192.168.1.24     4                     1384                  6
192.168.1.26     5                     1145                  5
192.168.133.200  2                     1008                  2
192.168.1.27     3                     980                   8
4.23.190.230     4                     304                   4
213.239.214.170  1                     76                    1

cat pfflowd|flow-stat -f 9 -S 2 
#  --- ---- ---- Report Information --- --- ---
#
# Fields:    Total
# Symbols:   Disabled
# Sorting:   Descending Field 2
# Name:      Source IP
#
# Args:      flow-stat -f 9 -S 2 
#
#
# IPaddr         flows                 octets                packets
#
xxx.xxx.3.192     3382                  19197196933           12824622
192.168.1.1      3389                  532997172             13315253
192.168.1.21     1319                  329805                4065
192.168.1.10     4046                  314617                4147
192.168.1.29     1316                  313150                3856
192.168.1.20     211                   57786                 703
192.168.1.13     149                   43738                 263
192.168.1.7      272                   34576                 272
192.168.1.15     273                   34174                 273
192.168.1.5      272                   33616                 272
192.168.1.23     80                    18724                 82
192.168.1.25     17                    10766                 81
192.168.1.24     4                     1384                  6
192.168.1.26     5                     1145                  5
192.168.133.200  2                     1008                  2
192.168.1.27     3                     980                   8
4.23.190.230     4                     304                   4
213.239.214.170  1                     76                    1

sullrich

Can you test this on a virgin freebsd system? Either way this really seems to be a FreeBSD issue so we'll have to gather lots of data for a freebsd-pf message.

billm

It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

–Bill

freeseacher

@billm:

It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

–Bill

rules for pf was
pass any in keep-state
pass any out keep-state

is there someting to miss ?

billm

@freeseacher:

@billm:

It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.

pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.

–Bill

rules for pf was
pass any in keep-state
pass any out keep-state

is there someting to miss ?

Yes, my point ;)

Not all packets in a given TCP flow will be considered "in state". Consider out of window packets, out of sequence window packets (stuff that's been ack'd and had data past it acked, but was retransmitted all the same). "normal" TCP communications do have packets that will get blocked. I'm reasonably confident that those packets will not cound against the PF byte count for that flow. The easiest way to determine that is to see if the PF byte count more closely matches that of the file(s) that were transferred. If it's under, then there's a bug somewhere, if it's over, but over by less than the other accounting types (ng_netflow is going to get all packets regardless of whether pf blocks it) then it's not a bug per se, you just have to understand what/where you're monitoring.

–Bill