Chelsio T5 vs. T6 SFP+ Adapters
-
Hi all,
I'm planning to add an PCI Express SFP+ network add-on card to my Supermicro 1U server and have started to look at Chelsio cards since they are well supported in FreeBSD/pfSense. I had just a couple questions for those of you know who more about these cards before I got out and make a purchase:
- Is there a good reason to go with a T6 card vs. a T5 card (previous generation)? I see that the T6 supports crypto offloading now as well (e.g. TLS/SSL, IPSec, etc.), but from what I can tell this is not yet supported in FreeBSD (I did see support for Linux though). Is there any other reason why it make more sense to go with a T6 based card? The price difference between the T5 and T6 isn't all that much so that is why I'm curious.
2) Is there a good reason to go with a regular card versus the memory free server offload ("SO") card instead (for example, the T520-CR vs. T520-SO-CR)? From what I can see the regular cards support more connections for TCP offloading (~32 for the SO-CR and ~32000 for the regular CR). Wouldn't this be advantageous to have for pfSense? Or are the benefits not really apparent until one regularly pushes multiple gigabits/second through their network?
Thanks in advance for your help, I really appreciate it.
-
I'm not sure in this case, but in many cases the hardware offloading does not apply well or at all to a router/firewall because it doesn't terminate the connection. For example, TSO and GSO can cause issues with packet pacing and buffer bloat. Great for throughput where you're CPU bound. I'm not sure how the TCP offloading works to know if it matters at all for a stateful firewall.
-
I'm not sure in this case, but in many cases the hardware offloading does not apply well or at all to a router/firewall because it doesn't terminate the connection. For example, TSO and GSO can cause issues with packet pacing and buffer bloat. Great for throughput where you're CPU bound. I'm not sure how the TCP offloading works to know if it matters at all for a stateful firewall.
Thanks Harvy - that is good info to know. Beyond what you said, I'm not sure that in my use case the additional benefits are worth justifying the incremental cost of the regular version of the card. Thanks again.
-
Well, I decided to go with the previous generation memory free server offload card, i.e. the Chelsio T520-SO-CR, since I found a pretty good deal on it. Will follow up here with some impressions once everything is installed and up and running.
-
@tman222 ...(3 eternities later). So, I have a Chelsio T6425 and a T520-CR and was provoked by reading this so I tested both. I'm glad I did (see the proxy comment), here's my thoughts:
- Based on deals I see, the T6 is twice the price, I got mine on Amazon for $US250. It's not twice as good. If you like shiny stuff, go for it but it's largely wasted money. I bought it becuase I can use it elsewhere if it does not spark joy in my life.
- T5 and T6 are both lovely cards for PFSense in operation (and didn't work on OPNsense)
- Both cards require significant cooling. Zip tying a 40mm fan to the NIC heatsink is a good idea.
- I'm currently running a proxy on a PFSense install with the T6 (just to get aound an annoying IPv6 issue) and it's shockingly quick. My wifi clients are seeing significant speed and speed stability increases (I can speed test and peg it at 600Mbit/s on wifi 5).
*The hardware accel on the T6 in general can be very picky about OS, params and which way the wnd is blowing. Occasionally you'll see mind blowing things like line rate transfer with no CPU usage. In practice it only does this went he stars align (chelsio adapters both ends, right kernel version, right protocol, right caching blah blah). On PFSense, I saw you had to put a few params in /boot/loader.conf to get it to enable features. This seemed in my unscientific testing to be better than the T5 but I can't put a number on it. - I tried the T6 on VPN and was seeing really low CPU usage (5-6%) for a single client OpenVPN running at 700Mbit/s-ish. I think the limiting factor was my VPN provider.
-
Mmm, interesting.
I would only expect the TCP off-loading options to do anything in something like the proxy case you mention where TCP connections are terminated on pfSense. For a standard firewall/routing scenario it does nothing as packets are simply forwarded.Steve
-
All I have are suspicions at the moment, maybe using the proxy means:
(Basing on this diagram https://images.app.goo.gl/rkh8VevthDQBQq3Y8)-
the dns request isn’t happening over wifi -> lower latency
-
the dns result might be getting cached by the proxy -> no dns lookup
-
Something like a fallback from ipv6 to ipv4, bad MTU, retry etc gets renegotiated without the client knowing
-
The connection through the proxy might mean the browsers limit of concurrent connections isn’t respected -> more simultaneous requests
-
There’s some different buffering effect where data is being more efficiently marshalled over wifi
-
The router has more horsepower than my wifi clients to do some of the functions (like dns req)
I looked at the logs just to check and it was https. I’m not caching https.
-
-
@mikefromoz - thanks for the comparison and insight, this is very helpful info! I figured I would follow up as well to share my experiences with these cards. I initially bought a Chelsio T520-SO-CR in 2018, but have upgraded since to a pair of Chelsio T540-SO-CR's. I have been using these cards for about 1 - 2 years now - one of them in my pfSense firewall, the other in a Promox hypervisor. They have been rock solid - I definitely recommend them. From time to time I see them pop up in the Chelsio store - I think it's a pretty good deal for a quad port 10Gbit card:
https://store.chelsio.com/collections/current-adapters/products/t540-lp-cr-4-port-1-10gbe-low-profile-uwire-adapters-with-pci-e-x8-gen3-32kconn-sfp-connector