Pfflowd non correctly counts
-
has lead test 3 collectors
1. pfflowd
2. ng_netflow
3. ndsad
as a result it turns out that figure which has counted pfflowd differs in smaller I shall shift on 802 mb was considered so
pfflowd
/usr/local/sbin/pfflowd -n 127.0.0.1:9800 -S any -v 5
pass any in keep-state
pass any out keep-statendsad (based on pcap)
force fxp0
ignore all
ip 127.0.0.1
port 9802ng_netflow
ngctl -f- <<-SEQ
mkpeer fxp0: tee lower left
connect fxp0: fxp0:lower upper right
mkpeer fxp0:lower one2many left2right many0
connect fxp0:lower.left2right fxp0:lower many1 right2left
name fxp0:lower.right2left o2m
mkpeer o2m: netflow one iface0
name o2m:one netflow
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/127.0.0.1:9801
SEQ
all 3 flows captured by
flow-capture -N 0 -n 288 -p /var/run/flow0.pid -V 5 -z 9 -w /usr/local/flow_data/pf 127.0.0.1/127.0.0.1/9800
flow-capture -N 0 -n 288 -p /var/run/flow1.pid -V 5 -z 9 -w /usr/local/flow_data/ng 127.0.0.1/127.0.0.1/9801
flow-capture -N 0 -n 288 -p /var/run/flow2.pid -V 5 -z 9 -w /usr/local/flow_data/nd 127.0.0.1/127.0.0.1/9802
then i run download grom some ftp server to get 13gb of data. so in a result i get inform for only one ip address
xxx.xxx.3.192
pfflowd 12.125mb
ng_netflow 12.985mb
ndsad 12.982mbWhether now I think has sense to use a collector which have counted not correct
-
So you believe pfflowd is not generating all the correct flow records and thus giving incomplete info?
I have not done a direct comparison but I can test it when I get some time. In the mean time, can you provide some more info? What are you running the ng_netflow and ndsad on? Always grabbing the same file from the remote location? I gather from the commands that you're using flow-tools to capture the data, correct? An 800mb difference is pretty substantial, it's feasable that pfflowd is missing something. What version of pfsense is it running on? -
wget mirrors some ftp server
3 collectors work on a same time
yes i use flow-tools to grap infouname
FreeBSD olc.local 6.1-STABLE FreeBSD 6.1-STABLE #0: Fri May 12 12:08:22 NOVST 2006 root@olc.local:/usr/obj/usr/src/sys/olc i386now i have not so much time but theme is truly important
maybe i miss some thing in a testssorry for my english
-
This is likely something that should be brought up to the author of pfflowd. He has a mailing list set up for discussion of such things. You can check out some details at http://www.mindrot.org/pfflowd.html
I'll be following this as well. I also plan to replicate this test to see if I get the same results.nb
-
You are obviously not running a pfSense kernel.
Why not?
-
I'm wondering if he's just using a stock freebsd install. ng_netflow is in the base install now. Is this a pfsense box? How hard would it be to add ng_netflow in the base pfsense? It would be fairly useful on the embedded platforms where there is no package system.
-
I'm wondering if he's just using a stock freebsd install. ng_netflow is in the base install now. Is this a pfsense box? How hard would it be to add ng_netflow in the base pfsense? It would be fairly useful on the embedded platforms where there is no package system.
I would be okay with this.
-
You are obviously not running a pfSense kernel.
Why not?
i done that test on my own machine.
than now i repeat it on pffsense machinei test it using that script
# cat /root/bin/netflow.sh #!/bin/sh #load some kldmodules kldload ng_netflow kldload ng_ether kldload ng_tee kldload ng_one2many #for ndsad kldload ng_ipfw # for ndsad copied from freebsd 6.1 kldload ipdivert.ko #pfflowd 0.6 /usr/local/sbin/pfflowd -n 127.0.0.1:9800 -S any -v 5 #ng_netflow ngctl -f- <<-SEQ mkpeer fxp0: tee lower left connect fxp0: fxp0:lower upper right mkpeer fxp0:lower one2many left2right many0 connect fxp0:lower.left2right fxp0:lower many1 right2left name fxp0:lower.right2left o2m mkpeer o2m: netflow one iface0 name o2m:one netflow mkpeer netflow: ksocket export inet/dgram/udp msg netflow:export connect inet/127.0.0.1:9801 SEQ #rule for ndsad ipfw add 10 tee 21000 all from any to any /usr/local/sbin/ndsad -d -c /usr/local/etc/ndsad.conf #% cat ndsad.conf | egrep -v "^#" | sort #bsd_div_copy yes #bsd_div_port 21000 #dummy all #force fxp0 #hash all 32 #heap 65536 #ignore all #ip 127.0.0.1 #log /tmp/ndsad.log #port 9802 #collector for pfflowd /usr/local/bin/flow-capture -N 0 -p /var/run/flow0.pid -V 5 -w /usr/local/netflow/pfflowd 127.0.0.1/127.0.0.1/9800 #collector for ng_netflow /usr/local/bin/flow-capture -N 0 -p /var/run/flow2.pid -V 5 -w /usr/local/netflow/ng_netflow 127.0.0.1/127.0.0.1/9801 #collector for ndsad /usr/local/bin/flow-capture -N 0 -p /var/run/flow1.pid -V 5 -w /usr/local/netflow/ndsad 127.0.0.1/127.0.0.1/9802and than get that results
cat ng_netflow|flow-stat -f 9 -S 2 # --- ---- ---- Report Information --- --- --- # # Fields: Total # Symbols: Disabled # Sorting: Descending Field 2 # Name: Source IP # # Args: flow-stat -f 9 -S 2 # # # IPaddr flows octets packets # xxx.xxx.3.192 6745 19197196933 12824622 192.168.1.1 6698 547846260 13412473 192.168.1.7 3570 2855505 51996 192.168.1.21 1031 330975 4080 192.168.1.10 4082 314801 4147 192.168.1.29 1176 313618 3862 192.168.1.20 109 56382 685 192.168.1.13 167 43499 258 192.168.1.15 273 34335 273 192.168.1.5 271 33538 271 192.168.1.23 83 19182 84 192.168.1.25 10 1841 13 192.168.1.24 4 1384 6 192.168.1.26 5 1145 5 192.168.133.200 2 1008 2 192.168.1.27 3 980 8 4.23.190.230 4 304 4 213.239.214.170 1 76 1 cat ndsad|flow-stat -f 9 -S 2 # --- ---- ---- Report Information --- --- --- # # Fields: Total # Symbols: Disabled # Sorting: Descending Field 2 # Name: Source IP # # Args: flow-stat -f 9 -S 2 # # # IPaddr flows octets packets # xxx.xxx.3.192 3394 19197196933 12824622 192.168.1.1 3573 547849008 13412456 192.168.1.7 1948 2857053 52015 192.168.1.21 1339 330975 4080 192.168.1.10 4077 316256 4166 192.168.1.29 1324 313618 3862 192.168.1.20 233 58020 706 192.168.1.13 166 43967 264 192.168.1.15 273 34335 273 192.168.1.5 271 33538 271 192.168.1.23 82 19182 84 192.168.1.25 21 11073 83 192.168.1.24 4 1384 6 192.168.1.26 5 1145 5 192.168.133.200 2 1008 2 192.168.1.27 3 980 8 4.23.190.230 4 304 4 213.239.214.170 1 76 1 cat pfflowd|flow-stat -f 9 -S 2 # --- ---- ---- Report Information --- --- --- # # Fields: Total # Symbols: Disabled # Sorting: Descending Field 2 # Name: Source IP # # Args: flow-stat -f 9 -S 2 # # # IPaddr flows octets packets # xxx.xxx.3.192 3382 19197196933 12824622 192.168.1.1 3389 532997172 13315253 192.168.1.21 1319 329805 4065 192.168.1.10 4046 314617 4147 192.168.1.29 1316 313150 3856 192.168.1.20 211 57786 703 192.168.1.13 149 43738 263 192.168.1.7 272 34576 272 192.168.1.15 273 34174 273 192.168.1.5 272 33616 272 192.168.1.23 80 18724 82 192.168.1.25 17 10766 81 192.168.1.24 4 1384 6 192.168.1.26 5 1145 5 192.168.133.200 2 1008 2 192.168.1.27 3 980 8 4.23.190.230 4 304 4 213.239.214.170 1 76 1so as we can see
cat ng_netflow|flow-stat|grep "Total Octets"
Total Octets : 19749051766
cat ndsad|flow-stat|grep "Total Octets"
Total Octets : 19749068855
cat pfflowd|flow-stat|grep "Total Octets"
Total Octets : 19731389954 -
Can you test this on a virgin freebsd system? Either way this really seems to be a FreeBSD issue so we'll have to gather lots of data for a freebsd-pf message.
-
It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.
pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.
–Bill
-
It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.
pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.
–Bill
rules for pf was
pass any in keep-state
pass any out keep-stateis there someting to miss ?
-
It's likely that pfflowd is only counting stuff that matches state. Retransmits that got dropped for whatever reason likely don't add to the flow numbers it retains.
pfflowd gets it's data from pfsync - I don't believe it maintains a table of inflight flows, I'm pretty sure it gets it's data from the state teardown messages. So, the data comes directly from the PF state entry which means only data that pf forwarded itself.
–Bill
rules for pf was
pass any in keep-state
pass any out keep-stateis there someting to miss ?
Yes, my point ;)
Not all packets in a given TCP flow will be considered "in state". Consider out of window packets, out of sequence window packets (stuff that's been ack'd and had data past it acked, but was retransmitted all the same). "normal" TCP communications do have packets that will get blocked. I'm reasonably confident that those packets will not cound against the PF byte count for that flow. The easiest way to determine that is to see if the PF byte count more closely matches that of the file(s) that were transferred. If it's under, then there's a bug somewhere, if it's over, but over by less than the other accounting types (ng_netflow is going to get all packets regardless of whether pf blocks it) then it's not a bug per se, you just have to understand what/where you're monitoring.
–Bill