[solved] Strange RRD graphs

Desintegr · Oct 22, 2015, 3:26 PM

Hi.

I use pfSense at work.
I have recently added a VLAN for phones and I see a very strange RRD graph for this VLAN :
Traffic : http://i.imgur.com/FU1TerC.png
Packets : http://i.imgur.com/bUxcxuX.png

I also monitor the pfSense with Zabbix.
The pfsense is connected with a 1 Gbps Ethernet link to a switch and I don't see this traffic on the pfSense port :
http://i.imgur.com/hTYOOgI.png
I don't see the traffic on the switch port :
http://i.imgur.com/hWBsdJr.png

I noticed last night a strange 8 Gbps inbound traffic on this VLAN :
http://i.imgur.com/QbQxgLM.png

I activated logs for blocked traffic : nothing abnormal.
I tried to capture traffic with tcpdump : nothing abnormal.
What can be this traffic ? Could be generated by pfSense ?

The interface counters are normal and grows normally :

In/out packets 	51771976/66960906 (56.84 GB/84.03 GB)
In/out packets (pass) 	51771976/66960906 (56.84 GB/84.03 GB)
In/out packets (block) 	152324/4 (28.12 MB/312 bytes)

pfInfo shows very few Packets/Bytes for this interface :

igb0_vlan13
Cleared:     Tue Oct 20 13:00:04 2015
References:  [ States:  117                Rules: 17                 ]
In4/Pass:    [ Packets: 51785238           Bytes: 61028770913        ]
In4/Block:   [ Packets: 149183             Bytes: 29050336           ]
Out4/Pass:   [ Packets: 66978106           Bytes: 90231300364        ]
Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
In6/Block:   [ Packets: 3183               Bytes: 443213             ]
Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
Out6/Block:  [ Packets: 4                  Bytes: 312                ]

RRD graphs for other interfaces/VLANs are normal.

Thank you very much for any help.
Note : I use pfSense 2.1.5, I will try to upgrade soon.

Desintegr · Oct 22, 2015, 3:11 PM

I think the traffic is OK but the RRD graphs are wrong.

Values obtainted from pfctl for this interface :

# polling packets for interface opt9 igb0_vlan13

/sbin/pfctl -vvsI -i igb0_vlan13 | awk '\
? /In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
? /In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
? END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'

51796980:66993333:149183:0:0:0:3218:4                                 
51796985:66993341:149183:0:0:0:3218:4
51797007:66993370:149183:0:0:0:3218:4
51797441:66993922:149183:0:0:0:3218:4
51797478:66993971:149183:0:0:0:3218:4

They seem to be OK :

Values obtained from rrdtool dump :

# rrdtool dump opt9-packets.rrd

 <row><v>7.1533705977e+07</v><v> 7.1519250572e+07 </v><v> 6.7509194026e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582785385e+07 </v><v>7.1582788264e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
 <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row>

There is something wrong !
Where does these value come from ?

Values do not match :

/var/db/rrd(137): /sbin/pfctl -vvsI -i igb0_vlan13
igb0_vlan13
        Cleared:     Tue Oct 20 13:00:04 2015
        References:  [ States:  131                Rules: 17                 ]
        In4/Pass:    [ Packets: 51807584           Bytes: 61032523456        ]
        In4/Block:   [ Packets: 149183             Bytes: 29050336           ]
        Out4/Pass:   [ Packets: 67007080           Bytes: 90233134001        ]
        Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
        In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
        In6/Block:   [ Packets: 3253               Bytes: 453223             ]
        Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
        Out6/Block:  [ Packets: 4                  Bytes: 312                ]

/var/db/rrd(138): rrdtool lastupdate opt9-packets.rrd
 inpass outpass inblock outblock inpass6 outpass6 inblock6 outblock6

1445526569: 46016 965 152958 0 0 0 179 1

Desintegr · Oct 22, 2015, 3:25 PM

OK, I have found the problem.
There was many updaterrd script running.
I disabled RRD graphs, clean graphs, kill old rrd related process.
Everything is normal now.