[solved] Strange RRD graphs



  • Hi.

    I use pfSense at work.
    I have recently added a VLAN for phones and I see a very strange RRD graph for this VLAN :
    Traffic : http://i.imgur.com/FU1TerC.png
    Packets : http://i.imgur.com/bUxcxuX.png

    I also monitor the pfSense with Zabbix.
    The pfsense is connected with a 1 Gbps Ethernet link to a switch and I don't see this traffic on the pfSense port :
    http://i.imgur.com/hTYOOgI.png
    I don't see the traffic on the switch port :
    http://i.imgur.com/hWBsdJr.png

    I noticed last night a strange 8 Gbps inbound traffic on this VLAN :
    http://i.imgur.com/QbQxgLM.png

    I activated logs for blocked traffic : nothing abnormal.
    I tried to capture traffic with tcpdump : nothing abnormal.
    What can be this traffic ? Could be generated by pfSense ?

    The interface counters are normal and grows normally :

    In/out packets 	51771976/66960906 (56.84 GB/84.03 GB)
    In/out packets (pass) 	51771976/66960906 (56.84 GB/84.03 GB)
    In/out packets (block) 	152324/4 (28.12 MB/312 bytes)
    

    pfInfo shows very few Packets/Bytes for this interface :

    igb0_vlan13
    Cleared:     Tue Oct 20 13:00:04 2015
    References:  [ States:  117                Rules: 17                 ]
    In4/Pass:    [ Packets: 51785238           Bytes: 61028770913        ]
    In4/Block:   [ Packets: 149183             Bytes: 29050336           ]
    Out4/Pass:   [ Packets: 66978106           Bytes: 90231300364        ]
    Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
    In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
    In6/Block:   [ Packets: 3183               Bytes: 443213             ]
    Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
    Out6/Block:  [ Packets: 4                  Bytes: 312                ]
    

    RRD graphs for other interfaces/VLANs are normal.

    Thank you very much for any help.
    Note : I use pfSense 2.1.5, I will try to upgrade soon.



  • I think the traffic is OK but the RRD graphs are wrong.

    Values obtainted from pfctl for this interface :

    # polling packets for interface opt9 igb0_vlan13
    
    /sbin/pfctl -vvsI -i igb0_vlan13 | awk '\
    ? /In4\/Pass/ { b4pi = $4 };/Out4\/Pass/ { b4po = $4 };/In4\/Block/ { b4bi = $4 };/Out4\/Block/ { b4bo = $4 };\
    ? /In6\/Pass/ { b6pi = $4 };/Out6\/Pass/ { b6po = $4 };/In6\/Block/ { b6bi = $4 };/Out6\/Block/ { b6bo = $4 };\
    ? END {print b4pi ":" b4po ":" b4bi ":" b4bo ":" b6pi ":" b6po ":" b6bi ":" b6bo};'
    
    51796980:66993333:149183:0:0:0:3218:4                                 
    51796985:66993341:149183:0:0:0:3218:4
    51797007:66993370:149183:0:0:0:3218:4
    51797441:66993922:149183:0:0:0:3218:4
    51797478:66993971:149183:0:0:0:3218:4
    

    They seem to be OK :

    Values obtained from rrdtool dump :

    # rrdtool dump opt9-packets.rrd
    
     <row><v>7.1533705977e+07</v><v> 7.1519250572e+07 </v><v> 6.7509194026e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582785385e+07 </v><v>7.1582788264e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788283e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
     <row><v>7.1582788283e+07</v><v> 7.1582788267e+07 </v><v> 7.1582788317e+07 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 0.0000000000e+00 </v><v> 7.1582788267e+07 </v><v>7.1582788267e+07</v></row> 
    

    There is something wrong !
    Where does these value come from ?

    Values do not match :

    /var/db/rrd(137): /sbin/pfctl -vvsI -i igb0_vlan13
    igb0_vlan13
            Cleared:     Tue Oct 20 13:00:04 2015
            References:  [ States:  131                Rules: 17                 ]
            In4/Pass:    [ Packets: 51807584           Bytes: 61032523456        ]
            In4/Block:   [ Packets: 149183             Bytes: 29050336           ]
            Out4/Pass:   [ Packets: 67007080           Bytes: 90233134001        ]
            Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
            In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
            In6/Block:   [ Packets: 3253               Bytes: 453223             ]
            Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
            Out6/Block:  [ Packets: 4                  Bytes: 312                ]
    
    /var/db/rrd(138): rrdtool lastupdate opt9-packets.rrd
     inpass outpass inblock outblock inpass6 outpass6 inblock6 outblock6
    
    1445526569: 46016 965 152958 0 0 0 179 1
    


  • OK, I have found the problem.
    There was many updaterrd script running.
    I disabled RRD graphs, clean graphs, kill old rrd related process.
    Everything is normal now.


Log in to reply