Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata InLine with igb NICs

    IDS/IPS
    6
    77
    7.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      newUser2pfSense
      last edited by

      stephenw10 and bmeeks...thank you both for responding. This has been ongoing for a while now in Inline IPS Mode. It doesn't happen in Legacy Mode. I've commented all of the lines out in the loader.conf.local file. I've noticed that the first release candidate of FreeBSD 12.0 has been released. I wonder how long it will take for the developers to fix the bug report?

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        If you really want to get pro-active you could try enabling suricata in inline mode on FreeBSD 12.
        I've never tried Suricata in FreeBSD directly. I know that the pfSense package has some patches also. Might not be that easy.
        It might not be necessary though since what you're actually testing is netmap with igb. Something else that uses netmap would probably also trigger it. Might have to make sure it does in 11.2 first.

        Steve

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by bmeeks

          Yes, as @stephenw10 suggested, you can certainly run the base Suricata package on FreeBSD 12 using Netmap. Just install the pkg from the CLI. Of course you won't have benefit of the pfSense package's GUI to configure things, but you could copy the basic suricata.yaml file from your pfSense box to the new hardware. You might have to adjust for interface names, and you would certainly have to manage the setup using the text-based tools within a CLI session. You can find the shell startup commands for the pfSense installation in this shell script: /usr/local/etc/rc.d/suricata.sh

          To use Inline IPS Mode you won't need any of the pfSense-specific Suricata patches. Those only add the custom Legacy Mode blocking functionality. Inline IPS Mode works with no patches. So just load up the package straight from FreeBSD ports or even compile it yourself using the local ports tree you can install with FreeBSD.

          The culprit in this bug is Netmap within the FreeBSD kernel and perhaps in combination with issues in specific NIC drivers was well. Unless the FreeBSD 12 source tree shows some commits related to Netmap, I would not get my hopes up, though. There seems to be not a whole lot of interest in addressing the Netmap issues. Maybe the user base is not large enough to warrant the developer time required to track down the issues.

          1 Reply Last reply Reply Quote 0
          • N
            newUser2pfSense
            last edited by

            stephenw10 and bmeeks...without an extra computer to test this on, I guess I'll have to wait until the developers take an interest in researching the issue and providing a fix. bmeeks - has anyone from pfSense/Netgate entered a bug report with FreeBSD by chance? Maybe another bug report might get the ball rolling specifically from a company instead of an individual. A company might be able to persuade the FreeBSD developers to take a more proactive approach. I don't know, just a thought.

            bmeeksB 1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks @newUser2pfSense
              last edited by

              @newuser2pfsense said in Network Card Tuning:

              stephenw10 and bmeeks...without an extra computer to test this on, I guess I'll have to wait until the developers take an interest in researching the issue and providing a fix. bmeeks - has anyone from pfSense/Netgate entered a bug report with FreeBSD by chance? Maybe another bug report might get the ball rolling specifically from a company instead of an individual. A company might be able to persuade the FreeBSD developers to take a more proactive approach. I don't know, just a thought.

              I am not affiliated with Netgate, so I wouldn't know about any possible bug report about Netmap submitted by them.

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by stephenw10

                I have not seen anything.
                Generally speaking submitting additional bug reports for which a report is already open is, at best, frowned upon! Better to add to existing bug reports. Even better to add to them with actual useful data.
                FreeBSD devs will rightly want to see any issues replicated in FreeBSD directly. And they will want to see that done in a current version. I would suggest that is 12 right now though maybe 11-stable might be acceptable.

                Demonstrating the bug exists in current FreeBSD and giving detailed steps to replicate that is the best way to attract developer attention.

                Steve

                1 Reply Last reply Reply Quote 0
                • N
                  newUser2pfSense
                  last edited by

                  stephenw10 and bmeeks...I appreciate all of your help. Unfortunately, I only have one computer like my pfSense instance, and of course it's being used for pfSense, so I wouldn't be able to replicate the issue to provide more information to the developers. I guess I had hoped that others who might see this post would have additional input to the FreeBSD bug report. Thanks.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    You can only do what you can do. There may well be others who can do more. Inline Suricata with igb is not that uncommon.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • B
                      boobletins
                      last edited by

                      I'm in a position to test this (and I've also been having issues with igb and em drivers + netmap).

                      I'm using a Jetway with I219-LM and I211-AT chips and it looks like the Supermicro is i350.

                      Would testing this with the lower level chips be useful, or do we expect that the i350s would work where the i2XX wouldn't?

                      If it would be useful, I can certainly put FreeBSD 12 and Suricata on a machine with i2XX...

                      1 Reply Last reply Reply Quote 1
                      • N
                        newUser2pfSense
                        last edited by

                        Hi boobletins...thank you for the response. That's very kind of you to offer to test this. I would only know the i350 as that's the card that I'm using. I wish I could offer more. My apologies.

                        B 1 Reply Last reply Reply Quote 0
                        • B
                          boobletins @newUser2pfSense
                          last edited by boobletins

                          @newuser2pfsense

                          Can you ssh and give me the result of ifconfig on the interface in question?

                          I just discovered after lots of annoyance that IPv6 Transmit Checksums were not disabled via the GUI when they appeared to be. Manually configuring it off solved this issue for me (at least I can now complete speedtests over the last hour or so with Suricata in IPS mode).

                          You might want to double check that

                          TXCSUM_IPV6

                          does not appear in your

                          ifconfig igb0

                          output (or the interface in question).

                          If it does, you might try:

                          ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso

                          then put Suricata back into IPS mode, restart Suricata, but don't reboot pfSense. See if your connection is stable. If you reboot pfSense, txcsum6 may reappear. I don't know where to permanently disable it.

                          bmeeksB stephenw10S 2 Replies Last reply Reply Quote 1
                          • bmeeksB
                            bmeeks @boobletins
                            last edited by

                            @boobletins said in Suricata InLine with igb NICs:

                            @newuser2pfsense

                            Can you ssh and give me the result of ifconfig on the interface in question?

                            I just discovered after lots of annoyance that IPv6 Transmit Checksums were not disabled via the GUI when they appeared to be. Manually configuring it off solved this issue for me (at least I can now complete speedtests over the last hour or so with Suricata in IPS mode).

                            You might want to double check that

                            TXCSUM_IPV6

                            does not appear in your

                            ifconfig igb0

                            output (or the interface in question).

                            If it does, you might try:

                            ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso

                            then put Suricata back into IPS mode, restart Suricata, but don't reboot pfSense. See if your connection is stable. If you reboot pfSense, txcsum6 may reappear. I don't know where to permanently disable it.

                            This issue is affecting more than just IPS mode in Suricata. It is impacting IPv6 connectivity for some folks on their WAN. I know there is a pfSense Redmine issue about it. The problem is within FreeBSD itself, I think. It's not a bug within Suricata itself. Suricata is a victim in this case. I think I've seen some posts about this in the IPv6 sub-forum here. I know I've seen a Redmine bug on it, too. I just had not connected directly to the IPS-Netmap problem. Good detective work!

                            1 Reply Last reply Reply Quote 0
                            • N
                              newUser2pfSense
                              last edited by newUser2pfSense

                              boobletins...Yes, TXCSUM_IPV6 is in the output of the ifconfig on my WAN interface; it's actually on all 4 interfaces on my SuperMicro ethernet adapter. If we knew where to disable it, I'm wondering if we could put that information in the loader.conf.local file? Then if we restart pfSense, I'm wondering if it would be disabled then? I don't know, just a thought.

                              bmeeks...I don't have IPv6 enabled anywhere on my pfSense instance, at least I don't believe.

                              B 1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator @boobletins
                                last edited by

                                @boobletins said in Suricata InLine with igb NICs:

                                Manually configuring it off solved this issue for me

                                It solved the netmap errors? Did you see any other errors that lead you to try this?

                                Are you actually using IPv6?

                                Steve

                                B 2 Replies Last reply Reply Quote 0
                                • N
                                  newUser2pfSense
                                  last edited by

                                  For me, I have IPv6 blocked on my WAN as a rule in the Firewall > Rules; it's practically at the top. However, I noticed a mix of IPv4 and IPv6 addresses in my System Logs > Firewall > Normal View tab. As well, I currently have Suricata running in Legacy Mode and have a ton of IPv6 addresses of SURICATA zero length padN option in the Alerts tab and all IPv6 addresses in the Blocks tab. Maybe I'm completely wrong but if I have IPv6 blocked on the WAN, should IPv6 addresses be showing up at all in the logs?

                                  bmeeksB 1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks @newUser2pfSense
                                    last edited by bmeeks

                                    @newuser2pfsense said in Suricata InLine with igb NICs:

                                    For me, I have IPv6 blocked on my WAN as a rule in the Firewall > Rules; it's practically at the top. However, I noticed a mix of IPv4 and IPv6 addresses in my System Logs > Firewall > Normal View tab. As well, I currently have Suricata running in Legacy Mode and have a ton of IPv6 addresses of SURICATA zero length padN option in the Alerts tab and all IPv6 addresses in the Blocks tab. Maybe I'm completely wrong but if I have IPv6 blocked on the WAN, should IPv6 addresses be showing up at all in the logs?

                                    You will likely have IPv6 Link-Local addresses created on your interfaces by default. I have them on all of my local firewall interfaces, including my WAN even though my ISP does not provide any type of IPv6 connectivity.

                                    A typical Windows domain will spew a lot of IPv6 stuff by default. In fact, IPv6 is a preferred communications route for Windows domain traffic unless it is explicitly disabled. Most of that will be via link-local addresses.

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      boobletins @stephenw10
                                      last edited by

                                      @stephenw10

                                      I suppose "solved" is a strong word. What I should have said is that before I couldn't complete a single speedtest and now I can complete an arbitrary number without netmap errors. Suricata also lasted through the night in IPS mode on my LAN interface (igb) without crashing which is extremely rare. I won't know if it's truly solved until it lasts through more like a week or month.

                                      I can reliably crash the interface by enabling TXCSUM_IPV6 and running a speedtest.

                                      I'm not a pfSense expert -- so when you ask if I'm using IPv6, all I know to say is that I have "Allow IPv6" enabled in the UI, and I see a smattering of IPv6 IPs in both Suricata Alerts and states (the majority are IPv4).

                                      Here is what I settled on for my boot.conf.local after referring these links:
                                      https://calomel.org/freebsd_network_tuning.html
                                      https://suricata.readthedocs.io/en/suricata-4.0.5/performance/packet-capture.html#rss

                                      kern.ipc.nmbclusters="1048576"
hw.pci.enable_msix=1

hw.em.msix=1
hw.em.smart_pwr_down=0
hw.em.num_queues=1   # https://suricata.readthedocs.io/en/suricata-4.0.5/performance/packet-capture.html#rss

# below this line is all from: https://calomel.org/freebsd_network_tuning.html
if_igb_load="YES"
hw.igb.enable_msix="1"
hw.igb.enable_aim="1"
hw.igb.rx_process_limit="100" #default
hw.igb.num_queues="3"  # (default 0 , queues equal the number of CPU real cores if queues available on card)
hw.igb.max_interrupt_rate="16000" #double default
coretemp_load="YES"
hw.intr_storm_threshold="9000" #default

if_em_load="YES"
hw.em.enable_msix="1"
hw.em.msix=1

autoboot_delay="-1"
net.isr.maxthreads="-1"
net.isr.bindthreads="1"  # (default 0, runs randomly on any one cpu core)

#Larger buffers and TCP Large Window Extensions
net.inet.tcp.rfc1323=1
net.inet.tcp.recvbuf_inc=65536     # (default 16384)
net.inet.tcp.sendbuf_inc=65536     # (default 8192)
net.inet.tcp.sendspace=65536       # (default 32768)
net.inet.tcp.mssdflt=1460   # Option 1 (default 536)
net.tcp.minmss=536  # (default 216)

#syn protection
net.inet.tcp.syncache.rexmtlimit=0  # (default 3)
                                      1 Reply Last reply Reply Quote 0
                                      • B
                                        boobletins @stephenw10
                                        last edited by boobletins

                                        @stephenw10 said in Suricata InLine with igb NICs:

                                        Did you see any other errors that lead you to try this?

                                        No -- I didn't see any specific IPv6 errors. I just started investigating the interface settings using information from here: https://calomel.org/freebsd_network_tuning.html and noticed that ifconfig showed TXCSUM_IPV6 enabled when I thought it was supposed to be disabled. Disabling it seems to have created a more stable interface with netmap enabled.

                                        I would receive 2 types of netmap errors previously "bad pkt" errors and "netmap_transmit" errors eg

                                        [2925] netmap_transmit           igb0 full hwcur 203 hwtail 204 qlen 1022 len 1514 m 0xfffff8000df20500
[1071] netmap_grab_packets       bad pkt at 419 len 2167

                                        I've tried using -txcsum6 the same on my WAN (em0) interface, but I still get bad packets there. I don't know if that has to do with the lack of msix support on that interface or my configuration settings. Still trying to figure that one out.

                                        Here's my dmesg output for em0 and igb0 in case that helps.

                                        em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> mem 0xdf100000-0xdf11ffff irq 16 at device 31.6 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:18:ce:19:cf
em0: netmap queues/slots: TX 1/1024, RX 1/1024
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
em0: link state changed to UP
em0: promiscuous mode enabled

                                        igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe000-0xe01f mem 0xdf000000-0xdf01ffff,0xdf020000-0xdf023fff irq 17 at device 0.0 on pci1
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 00:30:18:ce:19:d0
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb0: link state changed to UP
igb0: permanently promiscuous mode enabled
                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          boobletins @newUser2pfSense
                                          last edited by

                                          @newuser2pfsense said in Suricata InLine with igb NICs:

                                          I'm wondering if we could put that information in the loader.conf.local file?

                                          It looks like the right place to put this is described here:

                                          https://www.netgate.com/docs/pfsense/development/executing-commands-at-boot-time.html

                                          I tried to offer an example but Akismet thinks its spam. Let's see if this post will go through?

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            newUser2pfSense
                                            last edited by newUser2pfSense

                                            boobletins...I read through the link you provided but I'm not sure myself on what the syntax should be to add to the loader.conf.local file. I was looking to use the loader.conf.local file for an interim fix until the FreeBSD developers are able to solve the netmap issue(s) in a future release.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post