Chelsio T580-LP-CR not working in inline mode with Suricata

[rogeroger]

I have a Chelsio T580-LP-CR which is used for the LAN side only (i.e., my WAN is on another NIC). I have the Suricata applied on the LAN interface (1 VLAN is trunked on the LAN interface, so the promiscuous mode is active). When I activate the Inline mode and reboot the pfsense box, I'm getting some weird message in the pfsense box console (e.g. generic_find_num_desc called, in tx 1024 rx 102, etc..). After looking them up on google, I found various people that were trying to use Inline mode without a NIC that support the Netmap feature. Now, I checked the T580 and cxgbe driver, and they do support the Netmap feature. Do I have to activate something for the T580 to actually use the Netmap feature?

[bmeeks]

A quick search on Google turned up several links with Chelsio NIC tuning instructions for netmap. I would investigate down that alley so see if some custom sysctl parameters are required.

Also make sure that you have turned off all the various offloading options for the NIC including checksums, LRO, etc.

Netmap on FreeBSD is sadly still somewhat experimental in my view. This is especially true on the older FreeBSD versions.

It's not 100% clear from your post, so are you saying netmap is just spamming the console with this message or is your pfSense box crashing? Or is it just that Suricata is not seeing or blocking traffic on the Inline IPS interface?

[rogeroger]

I searched again with the term you proposed, and I didn't find any instruction or settings that I need to activate for the netmap.
I checked all "Hardware ... Offloading". Do you mean I should uncheck them?
I'm running the latest version of pfsense.
The pfsense box does not crash. However, the LAN interface on which Suricata is running (which also include the VLANS), can't be pinged (same for the vlans), so I can't access the web gui once I activate Suricata in inline mode on the LAN interface. So people said to try to change the snaplen, but it seems only available for the Legacy mode in pfsense. In the console of the pfsense box, I saw the following lines:

Setting up interfaces microcode...cxl0: tso4 disables due to -txcsum.
cxl0: tso6 disabled due to -txcsum6.
cxl0: enable txcsum first.

It almost look like I should leave the disable "Hardware checksum offloading" unchecked. And the usual lines:

...generic_find_num_desc called, in tx 1024 rx 1024

[jwj]

This is a bit old but I don't think there has been any progress concerning this. Maybe in FreeBSD 12.x (pfsense 2.5)?

https://forum.netgate.com/topic/110001/suricata-inline-and-vlans

In short, inline mode (netmap) and vlans don't play well together.

[NollipfSense]

@jwj said in Chelsio T580-LP-CR not working in inline mode with Suricata:

Maybe in FreeBSD 12.x (pfsense 2.5)?

I believe so...

[rogeroger]

@bmeeks
I tried the ifconfig command on the cxl0 interface and got the following:

cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
             options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...

It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

[bmeeks]

@rogeroger said in Chelsio T580-LP-CR not working in inline mode with Suricata:

@bmeeks
I tried the ifconfig command on the cxl0 interface and got the following:

cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
             options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...

It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

The netmap device does not support any type of hardware offloading for checksums and such. So all of that must be disabled or else the netmap device will drop the packets due to improper checksums.

If that Chelsio custom hardware has some type of offloading option that can't be disabled, then that very well might present a problem. You might want to take this issue up with the Chelsio hardware folks. I did find, in my earlier research, a link where they posted some netmap performance stats from one of their boxes. However, in that link they had utilized some sysctl tuning parameters.

The Suricata package on pfSense, when running with Inline IPS Mode enabled, uses the virgin Suricata binary with no customization. The binary in turn uses the kernel netmap device. The problems creep in due to either incomplete netmap support in NIC hardware drivers or due to the way netmap is unable to work properly with VLANs and/or limiters.

Also remember that pfSense-2.4.4-RELEASE is running on FreeBSD 11.2. So that is an older FreeBSD with an corresponding older netmap kernel device. There have been changes to the netmap kernel device in the later FreeBSD OS releases.

[rogeroger]

@bmeeks said in Chelsio T580-LP-CR not working in inline mode with Suricata:

@rogeroger said in Chelsio T580-LP-CR not working in inline mode with Suricata:

@bmeeks
I tried the ifconfig command on the cxl0 interface and got the following:

cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMIS,SIMPLEX,MULTICAST> metric 0 mtu 1500
             options=xxxxxx<VLAN_MTU,VLAN_HWTAGGING_JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO...

It seems that the interface have special VLAN hardware checksum offloading options. I don't know if these are the problem. I think disabling these in pfsense doesn't actually remove the VLAN hardware offloading options found on the interface. Now I need to find how to disable these!

Or found something about disabling the cxl0 interface and starting the vcxl0 instead. Not sure!

The netmap device does not support any type of hardware offloading for checksums and such. So all of that must be disabled or else the netmap device will drop the packets due to improper checksums.

If that Chelsio custom hardware has some type of offloading option that can't be disabled, then that very well might present a problem. You might want to take this issue up with the Chelsio hardware folks. I did find, in my earlier research, a link where they posted some netmap performance stats from one of their boxes. However, in that link they had utilized some sysctl tuning parameters.

The Suricata package on pfSense, when running with Inline IPS Mode enabled, uses the virgin Suricata binary with no customization. The binary in turn uses the kernel netmap device. The problems creep in due to either incomplete netmap support in NIC hardware drivers or due to the way netmap is unable to work properly with VLANs and/or limiters.

Also remember that pfSense-2.4.4-RELEASE is running on FreeBSD 11.2. So that is an older FreeBSD with an corresponding older netmap kernel device. There have been changes to the netmap kernel device in the later FreeBSD OS releases.

I decided to remove Suricata from the LAN interface, and deleted the LAN interface. My vlans still have the LAN interface as parent interface, and it works even if the LAN interface is not activated in the list of interface in pfsense. I activated Suricata on one of the VLAN interface and it works perfectly fine in Inline mode (i.e., I'm not getting any error message in the pfsense console, and the VLAN interface stays up). I left Suricata in promiscuous mode even if it is only on one VLAN interface now.

[NollipfSense]

@rogeroger Interesting solution indeed...thank you for sharing!

[rogeroger]

@NollipfSense
Oh well, got the same warnings later today. I guess I didn't correctly refresh the interface after applying the disable checksum. So my solution is not working!