Chelsio Advanced Configuration

[kklouzal]

I am trying to load the t4_tom module, as stated in the userguide, and properly enable TOE for this Chelsio t422-CR adapter, is this possible?

The Chelsio-UnifiedWire-FreeBSD-UserGuide.pdf has steps on how to do this however I'm receiving some errors.

Enable TCP offload on a T4 interface using ifconfig. An error usually indicates that t4_tom is not
loaded.
[root@host~]# ifconfig cxgbeX toe

If successful, TOE4 and TOE6 will be displayed on the list of the enabled options.

"ifconfig cxgbe0 toe" unfortunately results in the following error:
ifconfig: toe: Resource temporarily unavailable

"ifconfig cxgbe0" does NOT show TOE3 and TOE6 present in the listed options

Sub-Section 3.4 "Connection Offload Policies (COP)" step 1 explains how to load t4_tom:

If not completed already, load NIC and TOE drivers:
[root@host~]# kldload if_cxgbe
[root@host~]# kldload t4_tom

Which also results in the following errors:
kldload: can't load if_cxgbe: No such file or directory
kldload: can't load t4_tom: No such file or directory

How can I load t4_tom and enable some of the advanced features of my Chelsio card?
Specifically I'm trying to ensure the full TCP/UDP offload engine is enabled.

Thank you very much for your time and have a wonderful day!

[stephenw10]

cxgbe is already in kernel. t4_tom is not included, you would need to move it across from a FreeBSD source.

However TCP offload is unlikely to make much difference on a firewall which is mostly only forwarding packets. Even if you did get it working.

Steve

[kklouzal]

Thank you for the reply. Our use case is latency sensitive and any improvement, no matter how small, is time well spent. I do however understand that PfSense is mostly only forwarding packets so at the very least it would be interesting to benchmark the results after enabling these features.
These cards are quite expensive and even sold on the netgate store, that being said they are essentially being endorsed by the company as a preferred adapter. I see no reason why all the available features should not be readily accessible within PfSense by default? Is this a feature request that can be made somewhere?

[stephenw10]

Sure you can open a feature request here:
https://redmine.pfsense.org

Steve

[kklouzal]

For anyone interested, a feature request has been added to the bug tracker here: https://redmine.pfsense.org/issues/9091

More information on offloads from https://service.chelsio.com/beta/drivers/ChelsioUwire-FBSD-3.3.0.1/README.txt

" When a TOE capable interface is the outbound path to a destination to which a
new connection is being established, the connection should be offloaded
automatically. "

It would stand to reason that if your system was setup to use one or more Chelsio adapters, either exclusively or as an adapter that saw heavy traffic (WAN facing adapter), then every packet PfSense routed/forwarded over one such adapter would benefit from offloading.

[kklouzal]

Checking in 15 months later to see if any new information on this is available.

[stephenw10]

Did you ever try loading the module from FreeBSD?

Did it work? Help at all?

Steve

[kklouzal]

I'm not sure how to do that exactly. Previous attempts lead me to believe that I would need to build PfSense from source to include the module.

[stephenw10]

What I would expect is you copy the t4_tom.ko module from a suitable FreeBSD version and kldload it. That's the first thing to try here.

Steve

[laser22]

https://calomel.org/ has some great info on chelsio

[kklouzal]

@laser22 said in Chelsio Advanced Configuration:

https://calomel.org/ has some great info on chelsio

Indeed, very useful information, this website is an invaluable resource.

I would be willing to test the module in my system if anyone were able to provide it.

Hope this can be added to the baseline PfSense one day..

As far as firewall/packet forwarding is concerned I don't know if it will make much of a difference but there are plenty of packages that could take advantage of this, squid, maybe pfblocker, among others and with no change to their source required. TCP/UDP offload from the CPU directly onto the Chelsio T4/T5/T6 ASIC freeing up resources for the rest of the system to do normal packet forwarding/firewalling.

The issue on redmine had been assigned to someone at one point but it was eventually unassigned for some reason...

[stephenw10]

Try these. I can't test them against hardware but they load fine in 2.5.2:

[2.5.2-RELEASE][admin@252dev.stevew.lan]/boot/modules: ls
bwi_v3_ucode.ko if_wg.ko        linker.hints    t4_tom.ko       toecore.ko
[2.5.2-RELEASE][admin@252dev.stevew.lan]/boot/modules: kldload t4_tom.ko 
[2.5.2-RELEASE][admin@252dev.stevew.lan]/boot/modules: kldstat
Id Refs Address                Size Name
 1   15 0xffffffff80200000  3aea720 kernel
 2    1 0xffffffff83f19000     1000 cpuctl.ko
 3    1 0xffffffff83f1a000     2698 intpm.ko
 4    1 0xffffffff83f1d000      b40 smbus.ko
 5    1 0xffffffff83f1e000    344d8 if_wg.ko
 6    1 0xffffffff83f53000    137b0 t4_tom.ko
 7    1 0xffffffff83f67000      c7e toecore.ko

t4_tom.ko.txt
toecore.ko.txt

Remove the .txt extension.

Steve