Netmap errors in console
-
Hi all,
I have finally installed Suricata and enabled inline mode. I setup the sid management as per others recommendations.
I am running PFsense version 2.4.2. Using one of the recommended NIC's, i350T2, using the igb driver.
All seems to be running well. Getting the proper Suricata alerts.Issue is I get tons of netmap.grab_packets bad pkt listings in the GUI and system log. No errors in the Suricata logs.
Are these errors to be ignored, or is there a problem with Suricata?Here is a sample. There were 250+ of these overnight.
Dec 11 07:04:53 kernel 093.421388 [1071] netmap_grab_packets bad pkt at 762 len 2333
Dec 11 07:04:53 kernel 093.400226 [1071] netmap_grab_packets bad pkt at 759 len 2333
Dec 11 06:30:16 kernel 016.480547 [1071] netmap_grab_packets bad pkt at 828 len 2313
Dec 11 06:30:16 kernel 016.480523 [1071] netmap_grab_packets bad pkt at 827 len 2313
Dec 11 06:30:16 kernel 016.435130 [1071] netmap_grab_packets bad pkt at 818 len 2313
Dec 11 06:30:16 kernel 016.434499 [1071] netmap_grab_packets bad pkt at 816 len 2313
Dec 11 06:30:16 kernel 016.413635 [1071] netmap_grab_packets bad pkt at 806 len 2313
Dec 11 06:13:06 kernel 986.166554 [1071] netmap_grab_packets bad pkt at 1392 len 2546
Dec 11 06:04:50 kernel 490.242882 [1071] netmap_grab_packets bad pkt at 637 len 2333
Dec 11 06:04:50 kernel 490.220744 [1071] netmap_grab_packets bad pkt at 634 len 2333
Dec 11 06:00:01 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Dec 11 06:00:00 check_reload_status Reloading filter
Dec 11 06:00:00 php /usr/local/www/pfblockerng/pfblockerng.php: The command '/sbin/ifconfig 'igb3' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Dec 11 06:00:00 php [pfBlockerNG] Starting cron process.
Dec 11 05:34:50 kernel 689.947109 [1071] netmap_grab_packets bad pkt at 1356 len 2333
Dec 11 04:34:50 kernel 090.163484 [1071] netmap_grab_packets bad pkt at 930 len 2333
Dec 11 04:34:50 kernel 090.140454 [1071] netmap_grab_packets bad pkt at 927 len 2333
Dec 11 04:09:12 kernel 552.264122 [1071] netmap_grab_packets bad pkt at 708 len 2546
Dec 11 04:09:12 kernel 552.201909 [1071] netmap_grab_packets bad pkt at 701 len 2546
Dec 11 04:09:12 kernel 552.170906 [1071] netmap_grab_packets bad pkt at 695 len 2534
Dec 11 04:00:46 kernel 046.108484 [1071] netmap_grab_packets bad pkt at 171 len 3104
Dec 11 03:39:14 check_reload_status Reloading filterAre there any tunables I should be using? And yes, I have followed the tuning guide already. I saw in other posts this issue mentioned, but never an explanation as to what causes this other than the NIC is not supported. Which is not my issue. Should I disable inline mode?
Thanks in advance for your support for a great product.
-
No one?
I guess the netmap errors are a mystery and Inline mode is not a useable feature. Shame.
-
When you say you followed the tuning guide, that includes turning off all the hardware checksum offloading, LRO and segmentation stuff. Have you done that? If so, you just may have one of those NICs that is partially supported. If it seems to work and is just spamming the console, maybe you live with it until Netmap is more mature. This may be something you post to the FreeBSD folks as that's where the driver will get updated.
Bill
-
Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.
I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.
But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.
I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.
Thanks
-
Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.
I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.
But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.
I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.
Thanks
I am not knowledgeable enough about the FreeBSD kernel to say with certainty, but my guess would be Netmap may be falsely reporting bad packets. It's true you will get some now and then, but I would not think you would see very many at all in a switched LAN environment.
Bill
-
I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
This can't be a coincidence.
Are there any scheduled events outside of the ones that Cron performs in PFsense?
Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.
-
I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
This can't be a coincidence.
Are there any scheduled events outside of the ones that Cron performs in PFsense?
Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.
There are some other tasks that happen on a periodic basis, but I do not know what all they are. That would be a question better answered by a member of the pfSense developer team.
Bill
-
Where do I submit a question for the devs? GitHub?
-
Where do I submit a question for the devs? GitHub?
Probably the best place is here at the Redmine bug site: https://redmine.pfsense.org/projects/pfsense (for reporting the Netmap bug along with supporting evidence/clues you've gathered).
If you just want to ask a question, that is better done in one of the forum threads such as General Questions.
Bill
-
@whizzy said in Netmap errors in console:
I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
This can't be a coincidence.Hi Whizzy -- sorry for the necro.
I'm going back through old posts to see what packet sizes people were running into with the bad pkt error.
Of those that I see you've posted, 3104 seems to be the largest packet size. dev.netmap.buf_size of 4096 in ui system tuneables will solve that issue.
But really, I'm more interested in your comment above that there are some patterns in the times. Do you have those old logs? I'm trying to track down why people are getting larger packets in the pipeline than they should be.