Kernel keeps throwing messages
-
Any idea where to start on this?
Getting hundreds a day of these on the LAN interface. Sometimes we have to reboot the box to recover the LAN
Jun 10 04:46:26 kernel 786.725059 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:46:26 kernel 786.711024 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 125 c 125 t 0 rh 126 rc 126 rt 0 hc 126 ht 0
Jun 10 04:46:19 kernel 779.797797 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:46:19 kernel 779.783158 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 832 c 832 t 512 rh 833 rc 833 rt 512 hc 833 ht 512
Jun 10 04:46:19 kernel 779.631111 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:46:19 kernel 779.616479 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 668 c 668 t 512 rh 669 rc 669 rt 512 hc 669 ht 512
Jun 10 04:46:19 kernel 779.530546 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:46:19 kernel 779.515976 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 572 c 572 t 512 rh 573 rc 573 rt 512 hc 573 ht 512
Jun 10 04:46:10 kernel 770.564477 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:46:10 kernel 770.549800 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 919 c 919 t 512 rh 921 rc 921 rt 512 hc 921 ht 512
Jun 10 04:45:57 kernel 757.150478 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:57 kernel 757.135786 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 559 c 559 t 512 rh 561 rc 561 rt 512 hc 561 ht 512
Jun 10 04:45:44 kernel 744.711233 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:44 kernel 744.696512 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 723 c 723 t 512 rh 724 rc 724 rt 512 hc 724 ht 512
Jun 10 04:45:39 kernel 739.357396 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:39 kernel 739.343244 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 409 c 409 t 0 rh 410 rc 410 rt 0 hc 410 ht 0
Jun 10 04:45:30 kernel 730.233206 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:30 kernel 730.219548 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 65 c 65 t 0 rh 66 rc 66 rt 0 hc 66 ht 0
Jun 10 04:45:24 kernel 724.713457 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:24 kernel 724.699350 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 504 c 504 t 0 rh 505 rc 505 rt 0 hc 505 ht 0
Jun 10 04:45:21 kernel 721.675552 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:21 kernel 721.661889 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 57 c 57 t 0 rh 58 rc 58 rt 0 hc 58 ht 0
Jun 10 04:45:21 kernel 721.112173 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:21 kernel 721.098089 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 172 c 172 t 0 rh 173 rc 173 rt 0 hc 173 ht 0
Jun 10 04:45:14 kernel 714.539021 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:14 kernel 714.524312 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 685 c 685 t 512 rh 687 rc 687 rt 512 hc 687 ht 512
Jun 10 04:45:11 kernel 711.175422 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:11 kernel 711.160861 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 774 c 775 t 512 rh 775 rc 775 rt 512 hc 775 ht 512
Jun 10 04:45:06 kernel 706.561941 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:06 kernel 706.547308 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 527 c 527 t 512 rh 528 rc 528 rt 512 hc 528 ht 512
Jun 10 04:45:02 kernel 702.252605 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:02 kernel 702.238505 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 448 c 448 t 0 rh 449 rc 449 rt 0 hc 449 ht 0
Jun 10 04:45:01 kernel 701.502320 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:45:01 kernel 701.487686 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 929 c 929 t 512 rh 930 rc 930 rt 512 hc 930 ht 512
Jun 10 04:44:58 kernel 698.201869 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:58 kernel 698.187257 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 625 c 625 t 512 rh 626 rc 626 rt 512 hc 626 ht 512
Jun 10 04:44:55 kernel 695.265328 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:55 kernel 695.250221 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 1003 c 1003 t 512 rh 1004 rc 1004 rt 512 hc 1004 ht 512
Jun 10 04:44:54 kernel 694.437766 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:54 kernel 694.423071 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 645 c 645 t 512 rh 646 rc 646 rt 512 hc 646 ht 512
Jun 10 04:44:52 kernel 692.659374 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:52 kernel 692.645742 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 39 c 39 t 0 rh 40 rc 40 rt 0 hc 40 ht 0
Jun 10 04:44:44 kernel 684.266288 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:44 kernel 684.251618 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 869 c 869 t 512 rh 870 rc 870 rt 512 hc 870 ht 512
Jun 10 04:44:38 kernel 678.424579 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:38 kernel 678.410551 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 213 c 213 t 0 rh 215 rc 215 rt 0 hc 215 ht 0
Jun 10 04:44:35 kernel 675.084250 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:35 kernel 675.070153 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 266 c 266 t 0 rh 269 rc 269 rt 0 hc 269 ht 0
Jun 10 04:44:21 kernel 661.356924 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:21 kernel 661.342273 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 651 c 651 t 512 rh 652 rc 652 rt 512 hc 652 ht 512
Jun 10 04:44:11 kernel 651.818794 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:11 kernel 651.804162 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 871 c 871 t 512 rh 873 rc 873 rt 512 hc 873 ht 512
Jun 10 04:44:11 kernel 651.797807 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:11 kernel 651.783185 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 691 c 691 t 512 rh 692 rc 692 rt 512 hc 692 ht 512
Jun 10 04:44:11 kernel 651.217452 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:11 kernel 651.203345 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 225 c 225 t 0 rh 226 rc 226 rt 0 hc 226 ht 0
Jun 10 04:44:09 kernel 649.454480 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:09 kernel 649.439693 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 849 c 849 t 512 rh 850 rc 850 rt 512 hc 850 ht 512
Jun 10 04:44:07 kernel 647.328932 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:07 kernel 647.314923 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 498 c 498 t 0 rh 499 rc 499 rt 0 hc 499 ht 0
Jun 10 04:44:06 kernel 646.349726 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:06 kernel 646.335081 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 576 c 578 t 512 rh 577 rc 577 rt 512 hc 577 ht 512
Jun 10 04:44:05 kernel 645.588124 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:05 kernel 645.573528 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 795 c 795 t 512 rh 796 rc 796 rt 512 hc 796 ht 512
Jun 10 04:44:02 kernel 642.493248 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:02 kernel 642.478594 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 740 c 740 t 512 rh 741 rc 741 rt 512 hc 741 ht 512
Jun 10 04:44:00 kernel 640.560866 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:44:00 kernel 640.546236 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 693 c 693 t 512 rh 694 rc 694 rt 512 hc 694 ht 512
Jun 10 04:43:56 kernel 636.871686 [1787] netmap_ring_reinit called for igb5 TX0
Jun 10 04:43:56 kernel 636.857157 [1684] nm_txsync_prologue igb5 TX0: fail 'head > kring->rtail && head < kring->rhead' h 780 c 780 t 512 rh 781 rc 781 rt 512 hc 781 ht 512
Jun 10 04:43:52 kernel 632.686173 [1787] netmap_ring_reinit called for igb5 TX0 -
@remember said in Kernel keeps throwing messages:
Any idea where to start on this?
Hi,
Is there something like this on NGFW - IPS in inline mode?
what do these show?
cat /var/log/system.log | grep netmap
sysctl -a | grep netmap+++edit:
-
You appear to be using Suricata in Inline IPS Mode, as that mode uses the netmap kernel device. Suricata is not playing nice of late with the netmap device, especially in FreeBSD-12.2. In fact, version 6.0.2 of the Suricata binary will completely freeze the netmap-enabled interface after a random period of time. That's why I have not submitted an update to the 6.0.2 binary for the Suricata package.
You will need to turn off Inline IPS Mode with Suricata (or Snort, if you are using that package), and instead use the Legacy Blocking Mode.
-
Ok thx. I will enable legacy mode until a patch is released.
I have seen it lock up the interface completely as well.
Thanks for the insight !!!!
-
@remember said in Kernel keeps throwing messages:
Ok thx. I will enable legacy mode until a patch is released.
I have seen it lock up the interface completely as well.
Thanks for the insight !!!!
Yeah, there is some shared blame here between FreeBSD and its netmap code, and the Suricata binary itself. The netmap API has undergone at least three revisions over the last few years, and they are not backwards compatible. Couple that with the recent switchover to the new iflib wrapper for NIC drivers in FreeBSD-12 and you have a recipe for bugs. The current Suricata binary is using an older version of the netmap API.
-
@bmeeks said in Kernel keeps throwing messages:
update to the 6.0.2 binary for the Suricata package.
Hello Bill,
Do you have an ETA for this?
-
@daddygo said in Kernel keeps throwing messages:
@bmeeks said in Kernel keeps throwing messages:
update to the 6.0.2 binary for the Suricata package.
Hello Bill,
Do you have an ETA for this?
Not happening for now because Suricata 6.x is broken with netmap operation. See my earlier post in this thread. Until upstream addresses the netmap issue, I don't want to include the 6.x binary in the pfSense package. Here is my bug report to upstream: https://redmine.openinfosecfoundation.org/issues/4478. It was submitted on May 6th, and so far there has been no response at all. So not looking promising.
BTW, the other "*Sense" product is also still using the 5.x Suricata binary for the same reason.
-
@bmeeks said in Kernel keeps throwing messages:
Here is my bug report to upstream: https://redmine.openinfosecfoundation.org/issues/4478
from Bug #4478 "Nothing is logged to either the console nor the system log when this happens"
it sounds like a blind flight...
A long wait can be expected based on the unanswered Bug #4478, - well, based on what's been going on recently, I feel like everything's a little bit just going round and round...
Thanks for the answer, by the way I like this wording "the other "*Sense" product" :)
-
@bmeeks Any update on if/when this will be fixed so we can start using it?
-
@Remember said in Kernel keeps throwing messages:
@bmeeks Any update on if/when this will be fixed so we can start using it?
This problem was fixed back in August of 2021. If you are running the current Suricata package on pfSense you should not be seeing this errror. The current package version is 6.0.13 on RELEASE versions of pfSense CE and Plus, and version 7.0.0 on the DEVELOPMENT snapshots of pfSense CE and Plus.
