Suricata "fail 'head > kring->rtail && head < kring->rhead'"

[Marc05]

I see the following on the console when downloading reaches speeds like 8MB/s.

Jul 11 13:18:26 	kernel 		506.808937 [1688] nm_txsync_prologue igb0 TX0: fail 'head > kring->rtail && head < kring->rhead' h 689 c 689 t 512 rh 691 rc 691 rt 512 hc 691 ht 512
Jul 11 13:18:26 	kernel 		506.808990 [1791] netmap_ring_reinit called for igb0 TX0

After some time of this sustained bandwidth usage, I start seeing:

Jul 11 13:42:57 	kernel 		977.351964 [1791] netmap_ring_reinit called for igb0 TX1 
Jul 11 13:42:57 	kernel 		977.351996 [1816] netmap_ring_reinit total 1 errors
Jul 11 13:42:57 	kernel 		977.352020 [1820] netmap_ring_reinit igb0 TX1 reinit, cur 41 -> 40 tail 1024 -> 1024
Jul 11 13:42:57 	kernel 		977.775086 [1670] nm_txsync_prologue igb0 TX1: fail 'kring->nr_hwcur >= n || kring->rhead >= n || kring->rtail >= n || kring->nr_hwtail >= n' h 41 c 41 t 1024 rh 40 rc 40 rt 1024 hc 40 ht 1024
Jul 11 13:42:57 	kernel 		977.775175 [1791] netmap_ring_reinit called for igb0 TX1
Jul 11 13:42:57 	kernel 		977.775223 [1816] netmap_ring_reinit total 1 errors
Jul 11 13:42:57 	kernel 		977.775251 [1820] netmap_ring_reinit igb0 TX1 reinit, cur 41 -> 40 tail 1024 -> 1024
Jul 11 13:42:57 	kernel 		977.775584 [1791] netmap_ring_reinit called for igb0 TX1
Jul 11 13:42:57 	kernel 		977.775625 [1816] netmap_ring_reinit total 1 errors
Jul 11 13:42:57 	kernel 		977.775650 [1820] netmap_ring_reinit igb0 TX1 reinit, cur 41 -> 40 tail 1024 -> 1024

This is a VM on ESXi 7 and igb0 is a passthru NIC running Suricata 6.0.0_11. I have the following set on /boot/loader.conf.local:

hw.pci.honor_msi_blacklist="0"
net.isr.bindthreads="1"

I've looked around and I can't find anything solid on how to solve it. As is, I can't use Suricata due to it causing loss of connectivity. Any insight on this would be appreciated.

[bmeeks]

Over the last three months I've done some deep dives into the bowels of the netamp device driver in FreeBSD. I've learned quite a lot, but still have much more to learn before I achieve any level of actual "expertise" with netmap.

What I have learned is that netmap seems to have lots of issues, especially in FreeBSD-12 and higher where iflib came into use. In fact, right now, Suricata 6.x will completely stop passing traffic with netmap enabled after a short period of time. Tested this with 6.0.1, 6.0.2 and 6.0.3. All did the same thing. That's why we are still using Suricata 5.x in pfSense. And even 5.x is not 100% reliable with netmap.

I love the promise of the fast I/O that netmap promises, and the use of a true IPS mode on a firewall, but if it can't be 100% reliable, what good is it? At this point I'm tempted to just remove it from the package. I'm not venting directly on you, but just letting everyone know that I, too, am very frustrated with netmap and the poor track record it has in FreeBSD.

[Marc05]

Thank you for posting. I've been running into this issue for a while now - since the upgrade to FreeBSD 12 actually. All the time I've spent thinking somehow I've misconfigured something either on the hypervisor, pfSense, missing some tuning somewhere, etc. etc.. At least this tells me and future searchers that it's not necessarily them. I've switch back to Legacy Mode for now. I take it Snort would have the same issue in Inline mode?

[bmeeks]

I want to report that I am working offline with the Suricata developer team looking into this issue, and also the one affecting netmap operation in the Suricata 6.x binary.