LAN ping spikes

[andrew82]

Hello Guys!

I have one little problem.
When I pinging LAN side of pFsese 2.2.4 from my workstation it sometimes got spikes - like this:

Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time=173ms TTL=64
Reply from 192.168.235.2: bytes=32 time=204ms TTL=64
Reply from 192.168.235.2: bytes=32 time=56ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64
Reply from 192.168.235.2: bytes=32 time<1ms TTL=64

The same spikes can bee seen if i ping something in the internet(pFsense box does NAT for me).

If I run the "netstat -w 1 -I em0" I see input errors exactly the same time the ping time rises:

          input            em0           output
   packets  errs idrops      bytes    packets  errs      bytes colls
       904     0     0    1013282        689     0     228110     0
       684     0     0     631475        629     0     283227     0
       922     0     0     958064        695     0     312959     0
      1055     0     0    1109577       1004     0     725150     0
      1411     0     0    1210390       1127     0     756909     0
      1182     0     0    1077355        903     0     620903     0
      1323     0     0    1470344        917     0     436095     0
      6667     0     0    1197142       1405     0    1368559     0
      9670   669     0    1016874       1612     0    1951349     0
      2511   274     0    1157574       2159     0    2524190     0
       959     0     0     974904        732     0     400994     0
      1322     0     0     956421       1342     0    1389144     0
     12974   710     0    1014202       1123     0    1440876     0
      2658   280     0     950083       2036     0    2474490     0
      1927     0     0    1430998       1753     0    1836173     0
      2853     0     0    1268560       1444     0    1466548     0
      5935     0     0    1516476       1557     0    1525974     0
      1734     0     0     896831        619     0     371422     0
      1371     0     0    1168824       1069     0     798293     0

Printout of "sysctl -a | grep em.0"

dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.4.2
dev.em.0.%driver: em
dev.em.0.%location: slot=0 function=0
dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000
dev.em.0.%parent: pci1
dev.em.0.nvm: -1
dev.em.0.debug: -1
dev.em.0.fc: 3
dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66
dev.em.0.itr: 488
dev.em.0.rx_processing_limit: 100
dev.em.0.eee_control: 1
dev.em.0.link_irq: 758588
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.dropped: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.rx_overruns: 0
dev.em.0.watchdog_timeouts: 0
dev.em.0.device_control: 1477444168
dev.em.0.rx_control: 67403778
dev.em.0.fc_high_water: 18432
dev.em.0.fc_low_water: 16932
dev.em.0.queue0.txd_head: 544
dev.em.0.queue0.txd_tail: 546
dev.em.0.queue0.tx_irq: 64261306
dev.em.0.queue0.no_desc_avail: 5
dev.em.0.queue0.rxd_head: 404
dev.em.0.queue0.rxd_tail: 402
dev.em.0.queue0.rx_irq: 65211157
dev.em.0.mac_stats.excess_coll: 0
dev.em.0.mac_stats.single_coll: 0
dev.em.0.mac_stats.multiple_coll: 0
dev.em.0.mac_stats.late_coll: 0
dev.em.0.mac_stats.collision_count: 0
dev.em.0.mac_stats.symbol_errors: 0
dev.em.0.mac_stats.sequence_errors: 0
dev.em.0.mac_stats.defer_count: 0
dev.em.0.mac_stats.missed_packets: 1414487
dev.em.0.mac_stats.recv_no_buff: 0
dev.em.0.mac_stats.recv_undersize: 0
dev.em.0.mac_stats.recv_fragmented: 0
dev.em.0.mac_stats.recv_oversize: 0
dev.em.0.mac_stats.recv_jabber: 0
dev.em.0.mac_stats.recv_errs: 0
dev.em.0.mac_stats.crc_errs: 0
dev.em.0.mac_stats.alignment_errs: 0
dev.em.0.mac_stats.coll_ext_errs: 0
dev.em.0.mac_stats.xon_recvd: 2056237
dev.em.0.mac_stats.xon_txd: 22458
dev.em.0.mac_stats.xoff_recvd: 7061457
dev.em.0.mac_stats.xoff_txd: 1436945
dev.em.0.mac_stats.total_pkts_recvd: 86865296
dev.em.0.mac_stats.good_pkts_recvd: 77619751
dev.em.0.mac_stats.bcast_pkts_recvd: 67927
dev.em.0.mac_stats.mcast_pkts_recvd: 3282
dev.em.0.mac_stats.rx_frames_64: 3139958
dev.em.0.mac_stats.rx_frames_65_127: 47622772
dev.em.0.mac_stats.rx_frames_128_255: 1542585
dev.em.0.mac_stats.rx_frames_256_511: 449763
dev.em.0.mac_stats.rx_frames_512_1023: 1423140
dev.em.0.mac_stats.rx_frames_1024_1522: 23441533
dev.em.0.mac_stats.good_octets_recvd: 39455455874
dev.em.0.mac_stats.good_octets_txd: 102685879557
dev.em.0.mac_stats.total_pkts_txd: 89834208
dev.em.0.mac_stats.good_pkts_txd: 88374805
dev.em.0.mac_stats.bcast_pkts_txd: 1508
dev.em.0.mac_stats.mcast_pkts_txd: 48
dev.em.0.mac_stats.tx_frames_64: 3296140
dev.em.0.mac_stats.tx_frames_65_127: 14504847
dev.em.0.mac_stats.tx_frames_128_255: 1033178
dev.em.0.mac_stats.tx_frames_256_511: 805231
dev.em.0.mac_stats.tx_frames_512_1023: 1253425
dev.em.0.mac_stats.tx_frames_1024_1522: 67481984
dev.em.0.mac_stats.tso_txd: 524651
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.0.interrupts.asserts: 569449
dev.em.0.interrupts.rx_pkt_timer: 69
dev.em.0.interrupts.rx_abs_timer: 0
dev.em.0.interrupts.tx_pkt_timer: 0
dev.em.0.interrupts.tx_abs_timer: 5
dev.em.0.interrupts.tx_queue_empty: 0
dev.em.0.interrupts.tx_queue_min_thresh: 0
dev.em.0.interrupts.rx_desc_min_thresh: 0
dev.em.0.interrupts.rx_overrun: 174

Every lag I've got "mac_stats.missed_packets" increased

My box is configured
em0 LAN interface with 10 VLANS on it(users consuming 5-20Mbit), 192.168.235.2 is untaged.
em1 WAN (static IP)
re0 WAN2 (DHCP)
re1 not used.

CPU Core2Quad q9650 3.0Ghz  top shows CPU loads 2-4% while running a speedtest on a 100Mbit connection.
I've replaced the network adapter, switch and cable, but the behavior is the same, some time ago I've replaced the whole box and configured it from the scratch(not just restoring the config), but nothing changed.

PS: I'm sorry for my newbie FreeBSD knowledge and for my terrible English.

[firewalluser]

A L2 cache collision in the CPU I dont think will show in the CPU utilisation, nor will any caching done by any nics with onboard processing capabilities like intel nics.

Have you tried packet capturing using a separate bridged device between pfsense and your workstation to see whats actually going on with the packets across the network, you can get a better idea of what the packets are really doing then?

Might also be useful. http://blog.serverfault.com/2011/03/23/performance-tuning-intel-nics/