LAN ping spikes



  • Hello Guys!

    I have one little problem.
    When I pinging LAN side of pFsese 2.2.4 from my workstation it sometimes got spikes - like this:

    
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time=173ms TTL=64
    Reply from 192.168.235.2: bytes=32 time=204ms TTL=64
    Reply from 192.168.235.2: bytes=32 time=56ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
    

    The same spikes can bee seen if i ping something in the internet(pFsense box does NAT for me).

    If I run the "netstat -w 1 -I em0" I see input errors exactly the same time the ping time rises:

              input            em0           output
       packets  errs idrops      bytes    packets  errs      bytes colls
           904     0     0    1013282        689     0     228110     0
           684     0     0     631475        629     0     283227     0
           922     0     0     958064        695     0     312959     0
          1055     0     0    1109577       1004     0     725150     0
          1411     0     0    1210390       1127     0     756909     0
          1182     0     0    1077355        903     0     620903     0
          1323     0     0    1470344        917     0     436095     0
          6667     0     0    1197142       1405     0    1368559     0
          9670   669     0    1016874       1612     0    1951349     0
          2511   274     0    1157574       2159     0    2524190     0
           959     0     0     974904        732     0     400994     0
          1322     0     0     956421       1342     0    1389144     0
         12974   710     0    1014202       1123     0    1440876     0
          2658   280     0     950083       2036     0    2474490     0
          1927     0     0    1430998       1753     0    1836173     0
          2853     0     0    1268560       1444     0    1466548     0
          5935     0     0    1516476       1557     0    1525974     0
          1734     0     0     896831        619     0     371422     0
          1371     0     0    1168824       1069     0     798293     0
    
    

    Printout of "sysctl -a | grep em.0"

    dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.4.2
    dev.em.0.%driver: em
    dev.em.0.%location: slot=0 function=0
    dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000
    dev.em.0.%parent: pci1
    dev.em.0.nvm: -1
    dev.em.0.debug: -1
    dev.em.0.fc: 3
    dev.em.0.rx_int_delay: 0
    dev.em.0.tx_int_delay: 66
    dev.em.0.rx_abs_int_delay: 66
    dev.em.0.tx_abs_int_delay: 66
    dev.em.0.itr: 488
    dev.em.0.rx_processing_limit: 100
    dev.em.0.eee_control: 1
    dev.em.0.link_irq: 758588
    dev.em.0.mbuf_alloc_fail: 0
    dev.em.0.cluster_alloc_fail: 0
    dev.em.0.dropped: 0
    dev.em.0.tx_dma_fail: 0
    dev.em.0.rx_overruns: 0
    dev.em.0.watchdog_timeouts: 0
    dev.em.0.device_control: 1477444168
    dev.em.0.rx_control: 67403778
    dev.em.0.fc_high_water: 18432
    dev.em.0.fc_low_water: 16932
    dev.em.0.queue0.txd_head: 544
    dev.em.0.queue0.txd_tail: 546
    dev.em.0.queue0.tx_irq: 64261306
    dev.em.0.queue0.no_desc_avail: 5
    dev.em.0.queue0.rxd_head: 404
    dev.em.0.queue0.rxd_tail: 402
    dev.em.0.queue0.rx_irq: 65211157
    dev.em.0.mac_stats.excess_coll: 0
    dev.em.0.mac_stats.single_coll: 0
    dev.em.0.mac_stats.multiple_coll: 0
    dev.em.0.mac_stats.late_coll: 0
    dev.em.0.mac_stats.collision_count: 0
    dev.em.0.mac_stats.symbol_errors: 0
    dev.em.0.mac_stats.sequence_errors: 0
    dev.em.0.mac_stats.defer_count: 0
    dev.em.0.mac_stats.missed_packets: 1414487
    dev.em.0.mac_stats.recv_no_buff: 0
    dev.em.0.mac_stats.recv_undersize: 0
    dev.em.0.mac_stats.recv_fragmented: 0
    dev.em.0.mac_stats.recv_oversize: 0
    dev.em.0.mac_stats.recv_jabber: 0
    dev.em.0.mac_stats.recv_errs: 0
    dev.em.0.mac_stats.crc_errs: 0
    dev.em.0.mac_stats.alignment_errs: 0
    dev.em.0.mac_stats.coll_ext_errs: 0
    dev.em.0.mac_stats.xon_recvd: 2056237
    dev.em.0.mac_stats.xon_txd: 22458
    dev.em.0.mac_stats.xoff_recvd: 7061457
    dev.em.0.mac_stats.xoff_txd: 1436945
    dev.em.0.mac_stats.total_pkts_recvd: 86865296
    dev.em.0.mac_stats.good_pkts_recvd: 77619751
    dev.em.0.mac_stats.bcast_pkts_recvd: 67927
    dev.em.0.mac_stats.mcast_pkts_recvd: 3282
    dev.em.0.mac_stats.rx_frames_64: 3139958
    dev.em.0.mac_stats.rx_frames_65_127: 47622772
    dev.em.0.mac_stats.rx_frames_128_255: 1542585
    dev.em.0.mac_stats.rx_frames_256_511: 449763
    dev.em.0.mac_stats.rx_frames_512_1023: 1423140
    dev.em.0.mac_stats.rx_frames_1024_1522: 23441533
    dev.em.0.mac_stats.good_octets_recvd: 39455455874
    dev.em.0.mac_stats.good_octets_txd: 102685879557
    dev.em.0.mac_stats.total_pkts_txd: 89834208
    dev.em.0.mac_stats.good_pkts_txd: 88374805
    dev.em.0.mac_stats.bcast_pkts_txd: 1508
    dev.em.0.mac_stats.mcast_pkts_txd: 48
    dev.em.0.mac_stats.tx_frames_64: 3296140
    dev.em.0.mac_stats.tx_frames_65_127: 14504847
    dev.em.0.mac_stats.tx_frames_128_255: 1033178
    dev.em.0.mac_stats.tx_frames_256_511: 805231
    dev.em.0.mac_stats.tx_frames_512_1023: 1253425
    dev.em.0.mac_stats.tx_frames_1024_1522: 67481984
    dev.em.0.mac_stats.tso_txd: 524651
    dev.em.0.mac_stats.tso_ctx_fail: 0
    dev.em.0.interrupts.asserts: 569449
    dev.em.0.interrupts.rx_pkt_timer: 69
    dev.em.0.interrupts.rx_abs_timer: 0
    dev.em.0.interrupts.tx_pkt_timer: 0
    dev.em.0.interrupts.tx_abs_timer: 5
    dev.em.0.interrupts.tx_queue_empty: 0
    dev.em.0.interrupts.tx_queue_min_thresh: 0
    dev.em.0.interrupts.rx_desc_min_thresh: 0
    dev.em.0.interrupts.rx_overrun: 174
    
    

    Every lag I've got "mac_stats.missed_packets" increased

    My box is configured
    em0 LAN interface with 10 VLANS on it(users consuming 5-20Mbit), 192.168.235.2 is untaged.
    em1 WAN (static IP)
    re0 WAN2 (DHCP)
    re1 not used.

    CPU Core2Quad q9650 3.0Ghz  top shows CPU loads 2-4% while running a speedtest on a 100Mbit connection.
    I've replaced the network adapter, switch and cable, but the behavior is the same, some time ago I've replaced the whole box and configured it from the scratch(not just restoring the config), but nothing changed.

    PS: I'm sorry for my newbie FreeBSD knowledge and for my terrible English.



  • A L2 cache collision in the CPU I dont think will show in the CPU utilisation, nor will any caching done by any nics with onboard processing capabilities like intel nics.

    Have you tried packet capturing using a separate bridged device between pfsense and your workstation to see whats actually going on with the packets across the network, you can get a better idea of what the packets are really doing then?

    Might also be useful. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/