Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Chelsio T580-LP-CR not working in inline mode with Suricata

    Scheduled Pinned Locked Moved IDS/IPS
    10 Posts 4 Posters 1.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rogeroger
      last edited by

      I have a Chelsio T580-LP-CR which is used for the LAN side only (i.e., my WAN is on another NIC). I have the Suricata applied on the LAN interface (1 VLAN is trunked on the LAN interface, so the promiscuous mode is active). When I activate the Inline mode and reboot the pfsense box, I'm getting some weird message in the pfsense box console (e.g. generic_find_num_desc called, in tx 1024 rx 102, etc..). After looking them up on google, I found various people that were trying to use Inline mode without a NIC that support the Netmap feature. Now, I checked the T580 and cxgbe driver, and they do support the Netmap feature. Do I have to activate something for the T580 to actually use the Netmap feature?

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by bmeeks

        A quick search on Google turned up several links with Chelsio NIC tuning instructions for netmap. I would investigate down that alley so see if some custom sysctl parameters are required.

        Also make sure that you have turned off all the various offloading options for the NIC including checksums, LRO, etc.

        Netmap on FreeBSD is sadly still somewhat experimental in my view. This is especially true on the older FreeBSD versions.

        It's not 100% clear from your post, so are you saying netmap is just spamming the console with this message or is your pfSense box crashing? Or is it just that Suricata is not seeing or blocking traffic on the Inline IPS interface?

        R 1 Reply Last reply Reply Quote 1
        • R
          rogeroger
          last edited by rogeroger

          I searched again with the term you proposed, and I didn't find any instruction or settings that I need to activate for the netmap.
          I checked all "Hardware ... Offloading". Do you mean I should uncheck them?
          I'm running the latest version of pfsense.
          The pfsense box does not crash. However, the LAN interface on which Suricata is running (which also include the VLANS), can't be pinged (same for the vlans), so I can't access the web gui once I activate Suricata in inline mode on the LAN interface. So people said to try to change the snaplen, but it seems only available for the Legacy mode in pfsense. In the console of the pfsense box, I saw the following lines:

          Setting up interfaces microcode...cxl0: tso4 disables due to -txcsum.
          cxl0: tso6 disabled due to -txcsum6.
          cxl0: enable txcsum first.
          

          It almost look like I should leave the disable "Hardware checksum offloading" unchecked. And the usual lines:

          ...generic_find_num_desc called, in tx 1024 rx 1024
          
          1 Reply Last reply Reply Quote 0
          • ?
            A Former User
            last edited by

            This is a bit old but I don't think there has been any progress concerning this. Maybe in FreeBSD 12.x (pfsense 2.5)?

            https://forum.netgate.com/topic/110001/suricata-inline-and-vlans

            In short, inline mode (netmap) and vlans don't play well together.

            NollipfSenseN 1 Reply Last reply Reply Quote 0
            • NollipfSenseN
              NollipfSense @A Former User
              last edited by

              @jwj said in Chelsio T580-LP-CR not working in inline mode with Suricata:

              Maybe in FreeBSD 12.x (pfsense 2.5)?

              I believe so...

              pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
              pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

              1 Reply Last reply Reply Quote 0
              • R
                rogeroger @bmeeks
                last edited by rogeroger

                @bmeeks
                I tried the ifconfig command on the cxl0 interface and got the following:

                cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
                             options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...
                

                It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

                Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @rogeroger
                  last edited by bmeeks

                  @rogeroger said in Chelsio T580-LP-CR not working in inline mode with Suricata:

                  @bmeeks
                  I tried the ifconfig command on the cxl0 interface and got the following:

                  cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
                               options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...
                  

                  It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

                  Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

                  The netmap device does not support any type of hardware offloading for checksums and such. So all of that must be disabled or else the netmap device will drop the packets due to improper checksums.

                  If that Chelsio custom hardware has some type of offloading option that can't be disabled, then that very well might present a problem. You might want to take this issue up with the Chelsio hardware folks. I did find, in my earlier research, a link where they posted some netmap performance stats from one of their boxes. However, in that link they had utilized some sysctl tuning parameters.

                  The Suricata package on pfSense, when running with Inline IPS Mode enabled, uses the virgin Suricata binary with no customization. The binary in turn uses the kernel netmap device. The problems creep in due to either incomplete netmap support in NIC hardware drivers or due to the way netmap is unable to work properly with VLANs and/or limiters.

                  Also remember that pfSense-2.4.4-RELEASE is running on FreeBSD 11.2. So that is an older FreeBSD with an corresponding older netmap kernel device. There have been changes to the netmap kernel device in the later FreeBSD OS releases.

                  R 1 Reply Last reply Reply Quote 1
                  • R
                    rogeroger @bmeeks
                    last edited by

                    @bmeeks said in Chelsio T580-LP-CR not working in inline mode with Suricata:

                    @rogeroger said in Chelsio T580-LP-CR not working in inline mode with Suricata:

                    @bmeeks
                    I tried the ifconfig command on the cxl0 interface and got the following:

                    cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
                                 options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...
                    

                    It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

                    Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

                    The netmap device does not support any type of hardware offloading for checksums and such. So all of that must be disabled or else the netmap device will drop the packets due to improper checksums.

                    If that Chelsio custom hardware has some type of offloading option that can't be disabled, then that very well might present a problem. You might want to take this issue up with the Chelsio hardware folks. I did find, in my earlier research, a link where they posted some netmap performance stats from one of their boxes. However, in that link they had utilized some sysctl tuning parameters.

                    The Suricata package on pfSense, when running with Inline IPS Mode enabled, uses the virgin Suricata binary with no customization. The binary in turn uses the kernel netmap device. The problems creep in due to either incomplete netmap support in NIC hardware drivers or due to the way netmap is unable to work properly with VLANs and/or limiters.

                    Also remember that pfSense-2.4.4-RELEASE is running on FreeBSD 11.2. So that is an older FreeBSD with an corresponding older netmap kernel device. There have been changes to the netmap kernel device in the later FreeBSD OS releases.

                    I decided to remove Suricata from the LAN interface, and deleted the LAN interface. My vlans still have the LAN interface as parent interface, and it works even if the LAN interface is not activated in the list of interface in pfsense. I activated Suricata on one of the VLAN interface and it works perfectly fine in Inline mode (i.e., I'm not getting any error message in the pfsense console, and the VLAN interface stays up). I left Suricata in promiscuous mode even if it is only on one VLAN interface now.

                    NollipfSenseN 1 Reply Last reply Reply Quote 0
                    • NollipfSenseN
                      NollipfSense @rogeroger
                      last edited by

                      @rogeroger Interesting solution indeed...thank you for sharing!

                      pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                      pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                      R 1 Reply Last reply Reply Quote 0
                      • R
                        rogeroger @NollipfSense
                        last edited by

                        @NollipfSense
                        Oh well, got the same warnings later today. I guess I didn't correctly refresh the interface after applying the disable checksum. So my solution is not working!

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.