x4 SFP+ Expansion on XG-1541

GTAXL

Hello, I have a Netgate XG-1541. I'm looking at a handful of quad SFP+ NICs and have a few questions. I am considering the following options,
Chelsio CH-T540-SO-CR $100-250 used, currently unavail on eBay
Supermicro AOC-STG-i4S (based on Intel XL710-BM1 Chipset) $275 used / $400ish new
FS.com XL710BM1-4SP P/N: #75602 (based on Intel XL710-BM1) $639 new
Netgate Direct 4-PORT 10 GBE SFP+ CARD (based on Intel X710BM2) $879

So my questions are,

For 10Gbps SFP+, Intel vs Chelsio?
I am aware earlier models of the XG-1541 used Chelsio cards for SFP+ expansion, but Netgate appears to have shifted towards just Intel. Why is this? Is Intel superior, or is it simply a supply chain decision? From what I've researched, there are apparent pros and cons to each. These might be myths but I'd like Netgate's feedback,
Chelsio cards run so hot, Intel cards would be cooler.
Intel cards are picky about optics, they really want genuine intel modules.
Some Chelsio cards are picky about DACs, and want genuine Chelsio dac cables.
Chelsio's driver doesn't support ALTQ, the Intel one appeared to neither but was later re-supported.
iXsystems says Chelsio is absolute best on FreeBSD, Netgate says Intel. iXsystems appears to have been working very tightly with Chelsio in tuning the drivers and making it robust.
Why is Netgate's NIC card so expensive? Is it a genuine Intel NIC or a "based" version that uses the Intel controller, ex. Supermicro and FS.com NICs. Why should I pay nearly a grand when I can get essentially the same NIC, maybe not an Intel branded one, but for a couple hundred dollars?
What is the difference between XL710-BM1 and BM2?
How picky are Intel SFP+ cards in terms of module support? Ex. Mellanox cards will accept pretty much any module you throw at them, as do most Chelsio cards. I plan on potentially using a GPON ONU or EPON ONU SFP module down the road.
What are the specific make and model of the SFP+ cards you sell? Your store just lists the controller used. Where are these sourced, do you buy them directly from Intel?
Is the XG-1541 slot full height or half height? I suppose when I get a NIC, I need to make sure the bracket is the right size. I know some cards will include both low profile and normal profile brackets.

I really am just looking for a card that's very solid on pfSense, not picky about modules and somewhat affordable. What has my confused is different people saying different things.

rcoleman-netgate

@gtaxl said in x4 SFP+ Expansion on XG-1541:

Why is Netgate's NIC card so expensive? Is it a genuine Intel NIC or a "based" version that uses the Intel controller, ex. Supermicro and FS.com NICs. Why should I pay nearly a grand when I can get essentially the same NIC, maybe not an Intel branded one, but for a couple hundred dollars?

[...]

I really am just looking for a card that's very solid on pfSense, not picky about modules and somewhat affordable. What has my confused is different people saying different things.

The card we sell, as I understand it, is supported on FreeBSD and will work. And tested.

I can't comment on the price -- I'm just a tech.

The spec difference between the BM1 and the BM2 is the support of channels per chip. I suspect a BM1 card has 4 ICs per card and the BM2 has just 2.

GTAXL

@rcoleman-netgate said in x4 SFP+ Expansion on XG-1541:

The card we sell, as I understand it, is supported on FreeBSD and will work. And tested.

Yeah I understand that, but every other NIC I listed will work on FreeBSD. My question is which one is the best. You used to use Chelsio, now it's Intel.

@rcoleman-netgate said in x4 SFP+ Expansion on XG-1541:

I can't comment on the price -- I'm just a tech.

Well do you know the specific make and model of the 4-port SFP+ card you sell? That would help me compare the price. At the price you are selling in the Netgate store, I would expect it to be a brand new Intel branded card directly from Intel.

@rcoleman-netgate said in x4 SFP+ Expansion on XG-1541:

The spec difference between the BM1 and the BM2 is the support of channels per chip. I suspect a BM1 card has 4 ICs per card and the BM2 has just 2.

Ok, so what would this mean to the enduser? The cards are virtually the same, same driver used, should be just as solid, just laid out differently?

A lot of my questions went un-answered, but as it stands I'm currently looking at the Supermicro card. I originally was gonna go Chelsio, because my Chelsio card in my TrueNAS box has been solid. But Netgate insists on Intel.

Jarhead

@gtaxl
I can tell you this, my Chelsio T-540 worked perfectly as long as you don't use Wireguard. For some reason Wireguard and Chelsio combo would cause a crash. This was on 2.6.0, hoping 2.7.0 will make a change but I'm guessing Wireguard itself will need to be changed.

stephenw10

As you said the Chelsio NICs run waaay hotter. You absolutely need additional cooling if you use one.
The Chelsio NICs have a bunch of hardware off-loading capability that isn't used by pfSense at all. However at one time they were really the only option for 10G.

You might also consider one of the older X500 based Intel NICs that use the ix driver rather than ixl if cost is the primary decision driver here.

The NIC we sell is the only one we actively test each release with.

Steve

GTAXL

@stephenw10 said in x4 SFP+ Expansion on XG-1541:

As you said the Chelsio NICs run waaay hotter. You absolutely need additional cooling if you use one.

So is the additional fan necessary when using Intel cards? I already jumped the gun and bought the fan and bracket from SuperMicro's store. I guess if it's too loud or un-necessary, I'll put a Noctua low-noise adapter on it.

@stephenw10 said in x4 SFP+ Expansion on XG-1541:

You might also consider one of the older X500 based Intel NICs that use the ix driver rather than ixl if cost is the primary decision driver here.

I bought a used Supermicro AOC-STG-i4S because it seemed the cheapest and uses the same chipset as the NIC you sell. I'm hoping it'll work out and not be picky about modules.

@stephenw10 said in x4 SFP+ Expansion on XG-1541:

The NIC we sell is the only one we actively test each release with.

So you prefer Intel for 10Gbps SFP+ over Chelsio, why is this? Superior driver? Or just the fact they run cooler?

stephenw10

I can't speak for Netgate directly on this, I wasn't involved in that decision.

Personally I prefer Intel because you're not paying for features that pfSense can't use. Both in money and power consumption/heat. In FreeNAS that's different. I imagine they might well use the TCP offloading the Chelsio NICs offer. Though I've never looked into it.

More cooling never hurts!

Steve