Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Netmap errors in console

    Scheduled Pinned Locked Moved
    IDS/IPS
    3
    10
    1.2k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W
      whizzy
      last edited by

      Hi all,
      I have finally installed Suricata and enabled inline mode. I setup the sid management as per others recommendations.
      I am running PFsense version 2.4.2. Using one of the recommended NIC's, i350T2, using the igb driver.
      All seems to be running well. Getting the proper Suricata alerts.

      Issue is I get tons of netmap.grab_packets bad pkt listings in the GUI and system log. No errors in the Suricata logs.
      Are these errors to be ignored, or is there a problem with Suricata?

      Here is a sample. There were 250+ of these overnight.
      Dec 11 07:04:53 kernel 093.421388 [1071] netmap_grab_packets bad pkt at 762 len 2333
      Dec 11 07:04:53 kernel 093.400226 [1071] netmap_grab_packets bad pkt at 759 len 2333
      Dec 11 06:30:16 kernel 016.480547 [1071] netmap_grab_packets bad pkt at 828 len 2313
      Dec 11 06:30:16 kernel 016.480523 [1071] netmap_grab_packets bad pkt at 827 len 2313
      Dec 11 06:30:16 kernel 016.435130 [1071] netmap_grab_packets bad pkt at 818 len 2313
      Dec 11 06:30:16 kernel 016.434499 [1071] netmap_grab_packets bad pkt at 816 len 2313
      Dec 11 06:30:16 kernel 016.413635 [1071] netmap_grab_packets bad pkt at 806 len 2313
      Dec 11 06:13:06 kernel 986.166554 [1071] netmap_grab_packets bad pkt at 1392 len 2546
      Dec 11 06:04:50 kernel 490.242882 [1071] netmap_grab_packets bad pkt at 637 len 2333
      Dec 11 06:04:50 kernel 490.220744 [1071] netmap_grab_packets bad pkt at 634 len 2333
      Dec 11 06:00:01 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
      Dec 11 06:00:00 check_reload_status Reloading filter
      Dec 11 06:00:00 php /usr/local/www/pfblockerng/pfblockerng.php: The command '/sbin/ifconfig 'igb3' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
      Dec 11 06:00:00 php [pfBlockerNG] Starting cron process.
      Dec 11 05:34:50 kernel 689.947109 [1071] netmap_grab_packets bad pkt at 1356 len 2333
      Dec 11 04:34:50 kernel 090.163484 [1071] netmap_grab_packets bad pkt at 930 len 2333
      Dec 11 04:34:50 kernel 090.140454 [1071] netmap_grab_packets bad pkt at 927 len 2333
      Dec 11 04:09:12 kernel 552.264122 [1071] netmap_grab_packets bad pkt at 708 len 2546
      Dec 11 04:09:12 kernel 552.201909 [1071] netmap_grab_packets bad pkt at 701 len 2546
      Dec 11 04:09:12 kernel 552.170906 [1071] netmap_grab_packets bad pkt at 695 len 2534
      Dec 11 04:00:46 kernel 046.108484 [1071] netmap_grab_packets bad pkt at 171 len 3104
      Dec 11 03:39:14 check_reload_status Reloading filter

      Are there any tunables I should be using? And yes, I have followed the tuning guide already. I saw in other posts this issue mentioned, but never an explanation as to what causes this other than the NIC is not supported. Which is not my issue. Should I disable inline mode?

      Thanks in advance for your support for a great product.

      1 Reply Last reply Reply Quote 0
      • W
        whizzy
        last edited by

        No one?

        I guess the netmap errors are a mystery and Inline mode is not a useable feature. Shame.

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          When you say you followed the tuning guide, that includes turning off all the hardware checksum offloading, LRO and segmentation stuff.  Have you done that?  If so, you just may have one of those NICs that is partially supported.  If it seems to work and is just spamming the console, maybe you live with it until Netmap is more mature.  This may be something you post to the FreeBSD folks as that's where the driver will get updated.

          Bill

          1 Reply Last reply Reply Quote 0
          • W
            whizzy
            last edited by

            Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.

            I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.

            But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.

            I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.

            Thanks

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by

              @whizzy:

              Thanks for your answer. Yes all the offloading is disabled. This is why I posted a thread for users to come forward with which NIC's have been error free with Suricata Inline. Unfortunately there have been no responses to that.

              I am using a very common Intel NIC. i350T2V2. I have also tried an Intel i219 with same results.

              But what I really wanted to know is if these netmap errors can just be ignored, and what they represent. Am I losing data? Or are these actual bad packets that have hit the interface. Does anyone know this? I am sure that actual bad packets arrive on occasion. I just want to know if it is an issue of netmap not being able to handle a good packet or if it is just reporting a legitimate bad packet. That's all I want to know. Since the error does not display the IP, I have no way to tell. And besides, wouldn't the packet just be resent since there would be no ACK? Sort of like an SA, RA, or PA protocol log entry from the firewall.

              I did put these issues in with FreeBSD and do realize that this is not a PFsense issue.

              Thanks

              I am not knowledgeable enough about the FreeBSD kernel to say with certainty, but my guess would be Netmap may be falsely reporting bad packets.  It's true you will get some now and then, but I would not think you would see very many at all in a switched LAN environment.

              Bill

              1 Reply Last reply Reply Quote 0
              • W
                whizzy
                last edited by

                I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
                Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
                This can't be a coincidence.
                Are there any scheduled events outside of the ones that Cron performs in PFsense?
                Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.

                Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.

                B 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks
                  last edited by

                  @whizzy:

                  I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
                  Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
                  This can't be a coincidence.
                  Are there any scheduled events outside of the ones that Cron performs in PFsense?
                  Maybe the ISP runs something on a scheduled basis. I am going to do a packet capture during the next period I expect to see more netmap errors.

                  Strange thing is my other PFsense box with identical hardware doesn't see these netmap errors, but it doesn't see nearly the same volume of traffic either.

                  There are some other tasks that happen on a periodic basis, but I do not know what all they are.  That would be a question better answered by a member of the pfSense developer team.

                  Bill

                  1 Reply Last reply Reply Quote 0
                  • W
                    whizzy
                    last edited by

                    Where do I submit a question for the devs? GitHub?

                    1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks
                      last edited by

                      @whizzy:

                      Where do I submit a question for the devs? GitHub?

                      Probably the best place is here at the Redmine bug site:  https://redmine.pfsense.org/projects/pfsense (for reporting the Netmap bug along with supporting evidence/clues you've gathered).

                      If you just want to ask a question, that is better done in one of the forum threads such as General Questions.

                      Bill

                      1 Reply Last reply Reply Quote 0
                      • B
                        boobletins @whizzy
                        last edited by

                        @whizzy said in Netmap errors in console:

                        I did a comparison of the hundreds of netmap errors I have seen over the last week and noticed some common occurrences.
                        Most happen at the same minute of every hour. Almost like on a schedule, but Cron does not have any events at the same time.
                        This can't be a coincidence.

                        Hi Whizzy -- sorry for the necro.

                        I'm going back through old posts to see what packet sizes people were running into with the bad pkt error.

                        Of those that I see you've posted, 3104 seems to be the largest packet size. dev.netmap.buf_size of 4096 in ui system tuneables will solve that issue.

                        But really, I'm more interested in your comment above that there are some patterns in the times. Do you have those old logs? I'm trying to track down why people are getting larger packets in the pipeline than they should be.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post