in/out packets in Interfaces status page mismatch netstat,systat,tcpdump

DomiX · Sep 27, 2019, 1:15 PM

Hi,

I'm using a Dell R710 with 3 interfaces QLogic NetXtreme II BCM5709 (bce0:LAN, bce1:SDSL, bce2:Fiber) on 2.4.4-p3.
Everything works as expected (OpenVPN server). Statistics for bce1 in interfaces status page is not accurate. 21,7GB transfered with a 2Mb/s something is wrong, this connecion is used for backup OpenVPN but it's the default route for pfsense. Most traffic is DNS, NTP...

uptime
 3:08PM  up 1 day, 13:58, 1 user, load averages: 0.19, 0.11, 0.09

SDSL Interface (wan, bce1)

Status
    up
MAC Address
    XX
IPv4 Address
    XX
Subnet mask IPv4
    XX
Gateway IPv4
    XX
IPv6 Link Local
    XX
DNS servers
    XX
    XX
MTU
    1500
Media
    100baseTX <half-duplex>
In/out packets
    3580/109216894 (1.24 MiB/21.71 GiB)
In/out packets (pass)
    3580/109216894 (1.24 MiB/21.71 GiB)
In/out packets (block)
    181/1 (13 KiB/116 B)
In/out errors
    0/0
Collisions
    0

netstat -hbI bce1
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll
bce1   1.5K <Link#2>     XXX     3.9k     0     0       1.3M     3.8k     0       351K     0
bce1      - XXX        0     -     -          0        0     -          0     -
bce1      - XXX         3.6k     -     -       1.2M        0     -          0     -

systat -ifstat
      Interface           Traffic               Peak                Total
           bce1  in      0.000 KB/s          0.000 KB/s            1.313 MB
                 out     0.000 KB/s          0.000 KB/s          351.506 KB

netstat and systat for others interfaces shows correct transfer with in/out >4GB

With

tcpdump -i bce1 or 
tcpdump -i pflog0 -ettt on bce1

I do not see huge traffic.
I'm also monitoring this server with zabbix Agent and bce1 traffic is nearly 0

Why this difference with interfaces status page (from code pfctl -vvsI -i bce1) and others network tools for traffic usage ?

Thanks in advance for your help.

stephenw10 · Sep 29, 2019, 10:26 PM

@DomiX said in in/out packets in Interfaces status page mismatch netstat,systat,tcpdump:

100baseTX <half-duplex>

Is it actually running at 100Mb-HD?

If not it seems like the NIC/driber is not reporting correctly.

You might check the mac stats in the sysctl output to see which it agrees with.

Steve

DomiX · Sep 30, 2019, 9:12 AM

Hi,

I changed the ethernet cable with a 6a model and it's still 100TX half-duplex. I tried with Windows 10 and it's also half-duplex. It's connected to a Cisco C888EA (unfortunatly I do not have access, I opened a ticket to my provider for more information).

What do mean by checking mac stats in sysctl ?

sysctl dev.bce.1
dev.bce.1.com_no_buffers: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 5
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_XoffPauseFramesReceived: 0
dev.bce.1.stat_XonPauseFramesReceived: 0
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 0
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 124
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 8
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 61
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 8766
dev.bce.1.stat_EtherStatsPktsTx64Octets: 316
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 1276
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 97
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 2
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 59
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 7017
dev.bce.1.stat_EtherStatsPktsRx64Octets: 422
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsCollisions: 1
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 1
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_IfHCOutBroadcastPkts: 306
dev.bce.1.stat_IfHCOutMulticastPkts: 2
dev.bce.1.stat_IfHCOutUcastPkts: 8967
dev.bce.1.stat_IfHCInBroadcastPkts: 5
dev.bce.1.stat_IfHCInMulticastPkts: 0
dev.bce.1.stat_IfHCInUcastPkts: 8868
dev.bce.1.stat_IfHCOutBadOctets: 4
dev.bce.1.stat_IfHCOutOctets: 898526
dev.bce.1.stat_IfHCInBadOctets: 0
dev.bce.1.stat_IfHcInOctets: 2702586
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 0
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.mbuf_frag_count: 0
dev.bce.1.mbuf_alloc_failed_count: 0
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.%parent: pci1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0235 class=0x020000
dev.bce.1.%location: slot=0 function=1 dbsf=pci0:1:0:1
dev.bce.1.%driver: bce
dev.bce.1.%desc: QLogic NetXtreme II BCM5709 1000Base-T (C0)

DomiX · Sep 30, 2019, 1:48 PM

@stephenw10 According to my provider, the port is configured in 100M full-duplex but negotiation disabled.
I forced full-duplex on pfsense and rebooted but traffic is still high in status interfaces pages but systat and netstat are low.

MTU
    1500
Media
    100baseTX <full-duplex>
In/out packets
    574/5421513 (210 KiB/1.46 GiB)
In/out packets (pass)
    574/5421513 (210 KiB/1.46 GiB)
In/out packets (block)
    0/2 (0 B/188 B)
In/out errors
    0/0
Collisions
    0

systat -ifstat

           bce1  in      0.000 KB/s          0.000 KB/s          218.250 KB
                 out     0.000 KB/s          0.000 KB/s           54.666 KB

stephenw10 · Sep 30, 2019, 4:50 PM

Hmm, the mac stats look very low. That looks to agree with ifstat then.

You're sure it's bce1 and not bce0? Do the stats for any bce NIC look correct?

Steve

DomiX · Oct 1, 2019, 6:50 AM

@stephenw10
Yes, bce1 statistics are not correct with pfctl (interfaces status page). Others statistics for bce0 (LAN) or bce2 (WAN 200Mb/s) are correct.

pfctl -vvsI -i bce1
bce1
        Cleared:     Mon Sep 30 17:39:08 2019
        References:  83
        In4/Pass:    [ Packets: 1580               Bytes: 451707             ]
        In4/Block:   [ Packets: 331                Bytes: 23142              ]
        Out4/Pass:   [ Packets: 37376681           Bytes: 6059122661         ]
        Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
        In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
        In6/Block:   [ Packets: 0                  Bytes: 0                  ]
        Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
        Out6/Block:  [ Packets: 1                  Bytes: 116                ]

netstat -bh -I bce1
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll
bce1   1.5K <Link#2>      XXX     2.0k     0     0       492K     3.0k     0       326K     0
bce1      - XXbce1/6 XX        0     -     -          0        0     -          0     -
bce1      - XX XX         1.6k     -     -       441K        0     -          0     -

sysctl dev.bce.1
dev.bce.1.com_no_buffers: 0
dev.bce.1.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.1.stat_CatchupInMBUFDiscards: 0
dev.bce.1.stat_CatchupInFTQDiscards: 0
dev.bce.1.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInRuleCheckerP4Hit: 1
dev.bce.1.stat_IfInMBUFDiscards: 0
dev.bce.1.stat_IfInFTQDiscards: 0
dev.bce.1.stat_IfInRuleCheckerDiscards: 0
dev.bce.1.stat_IfInFramesL2FilterDiscards: 0
dev.bce.1.stat_XoffStateEntered: 0
dev.bce.1.stat_MacControlFramesReceived: 0
dev.bce.1.stat_FlowControlDone: 0
dev.bce.1.stat_OutXoffSent: 0
dev.bce.1.stat_OutXonSent: 0
dev.bce.1.stat_XoffPauseFramesReceived: 0
dev.bce.1.stat_XonPauseFramesReceived: 0
dev.bce.1.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsTx1024Octetsto1522Octets: 2
dev.bce.1.stat_EtherStatsPktsTx512Octetsto1023Octets: 23
dev.bce.1.stat_EtherStatsPktsTx256Octetsto511Octets: 5
dev.bce.1.stat_EtherStatsPktsTx128Octetsto255Octets: 1347
dev.bce.1.stat_EtherStatsPktsTx65Octetsto127Octets: 1624
dev.bce.1.stat_EtherStatsPktsTx64Octets: 43
dev.bce.1.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0
dev.bce.1.stat_EtherStatsPktsRx1024Octetsto1522Octets: 226
dev.bce.1.stat_EtherStatsPktsRx512Octetsto1023Octets: 16
dev.bce.1.stat_EtherStatsPktsRx256Octetsto511Octets: 1
dev.bce.1.stat_EtherStatsPktsRx128Octetsto255Octets: 21
dev.bce.1.stat_EtherStatsPktsRx65Octetsto127Octets: 1630
dev.bce.1.stat_EtherStatsPktsRx64Octets: 60
dev.bce.1.stat_EtherStatsOversizePkts: 0
dev.bce.1.stat_EtherStatsUndersizePkts: 0
dev.bce.1.stat_EtherStatsJabbers: 0
dev.bce.1.stat_EtherStatsFragments: 0
dev.bce.1.stat_EtherStatsCollisions: 0
dev.bce.1.stat_Dot3StatsLateCollisions: 0
dev.bce.1.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.1.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.1.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.1.stat_Dot3StatsAlignmentErrors: 0
dev.bce.1.stat_Dot3StatsFCSErrors: 0
dev.bce.1.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.1.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.1.stat_IfHCOutBroadcastPkts: 43
dev.bce.1.stat_IfHCOutMulticastPkts: 2
dev.bce.1.stat_IfHCOutUcastPkts: 2999
dev.bce.1.stat_IfHCInBroadcastPkts: 1
dev.bce.1.stat_IfHCInMulticastPkts: 0
dev.bce.1.stat_IfHCInUcastPkts: 1953
dev.bce.1.stat_IfHCOutBadOctets: 0
dev.bce.1.stat_IfHCOutOctets: 346999
dev.bce.1.stat_IfHCInBadOctets: 0
dev.bce.1.stat_IfHcInOctets: 512099
dev.bce.1.unexpected_attention_count: 0
dev.bce.1.dma_map_addr_tx_failed_count: 0
dev.bce.1.dma_map_addr_rx_failed_count: 0
dev.bce.1.mbuf_frag_count: 0
dev.bce.1.mbuf_alloc_failed_count: 0
dev.bce.1.l2fhdr_error_count: 0
dev.bce.1.%parent: pci1
dev.bce.1.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028 subdevice=0x0235 class=0x020000
dev.bce.1.%location: slot=0 function=1 dbsf=pci0:1:0:1
dev.bce.1.%driver: bce
dev.bce.1.%desc: QLogic NetXtreme II BCM5709 1000Base-T (C0)

pfctl -vvsI -i bce0
bce0
        Cleared:     Mon Sep 30 17:39:08 2019
        References:  30
        In4/Pass:    [ Packets: 11072294           Bytes: 2460981815         ]
        In4/Block:   [ Packets: 8457               Bytes: 160657             ]
        Out4/Pass:   [ Packets: 13905940           Bytes: 9512430255         ]
        Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
        In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
        In6/Block:   [ Packets: 124349             Bytes: 16424984           ]
        Out6/Pass:   [ Packets: 0                  Bytes: 0                  ]
        Out6/Block:  [ Packets: 3                  Bytes: 268                ]

 netstat -bh -I bce0
Name    Mtu Network       Address              Ipkts Ierrs Idrop     Ibytes    Opkts Oerrs     Obytes  Coll
bce0   1.5K <Link#1>      XXX     12M     0     0       2.5G      15M     0       9.1G     0
bce0      - XX%bce0/6 XXX        0     -     -          0        0     -          0     -
bce0      - XX XX             296k     -     -        18M     261k     -        15M     -

Thanks