Load updated Intel IX module to get 10Gbps

[ogghi]

@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess...

[stephenw10]

The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.

[ogghi]

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:
login-to-view

[stephenw10]

That is a total throughput value though. You won't see that in an iperf3 test across an single link.

Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.

[ogghi]

@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least?

[stephenw10]

For a single TCP stream like that I would look for fewer cores with a higher frequency. And something newer is obviously more efficient so some relatively recent i3 or i5 for example.

There a few threads discussing specific options.

[ogghi]

@stephenw10 So you are saying that basically I could take an old desktop I have laying around and add some SFP+ card and that's it?
Just because of the higher CPU clock speed it would be faster?

Netgate 8200 for example has only 2.4GHz compared to our current Intel Xeon CPU D-1518 with 2.20GHz.

There won't be much of a difference then?

Also, can we be sure this is actually the limiting factor?

Eg. if I do an Iperf3 with 8 streams, this is not getting faster either? I am missing something conceptional here?

[stephenw10]

iperf itself is deliberately single threaded so you may be limited there. You might need to run multiple iperf processes.

No there's not much difference between the D1518 and C3758:
https://www.cpubenchmark.net/compare/2799vs3696vs4746vs3280/Intel-Xeon-D-1518-vs-Intel-Atom-C3758-vs-Intel-i3-12300-vs-Intel-i3-8300

[ogghi]

@stephenw10
So in theory I could use an old desktop PC with a CPU with 4+GHz and 16GB RAM, add a Intel X520-DA2 card and should be getting better bandwidth routed?

thanks :)

[stephenw10]

If you have one I would certainly try it. Be aware some of those old systems can be very power hungry though.

[ogghi]

@stephenw10
in the office it does not matter to have even 100W more running all time here, then I would first tell people to not leave their PCs running
I'll buy a card, install PFsense on a PC, backup current instance, restore on new machine, re-assign network interfaces where needed, test. Sounds like a plan?

[ogghi]

@stephenw10
Ok, seems that after all I shall not build a system myself, rather get an official appliance or something with warranty and possibly support.
What appliance would manage to give us the 10Gb here?

[Patch]

@ogghi said in Load updated Intel IX module to get 10Gbps:

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option

What traffic mix is the operational requirement? More specifically

• WAN type
• VPN connections and load
• typical number of concurrent connections, especially high bandwidth connections.

The reason I ask is I wonder how well the test results correlate with the operational performance under likely load conditions.

[ogghi]

@Patch
Hi there.

WAN Type is Init7 10/10Gbps fiber.
VPN connections (speed not relevant here) maximum of 20 ppl on OpenVPN, 3 WireGuard tunnels.

pfTop: Up State 1-100/14500
Are those concurrent connections?

High bandwidth usually would be the backup server sending backups to S3 storage outside...otherwise not too much high bandwidth things happening.

[ogghi]

@Patch any idea?

[stephenw10]

The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.

[ogghi]

@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?

Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
?

[stephenw10]

Well in a test like that I'd expect to see 7-8Gbps through the D1541 so seeing 5Gbps with a D1518 is not wildly low.

But as I said we have seen reports of dramatically higher throughput using other NICs. I've not tested that myself to confirm though.

[ogghi]

@stephenw10
I guess I figured out what HW we have here:
Must be one of those guys, except we don't have the 4 port Ethernet card in:
https://www.newegg.com/supermicro-sys-5018d-fn8t-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/370-0003-000G9

Also now I am thinking: If the Mellanox cards might give us better throughput, I could get a MCX4121A-ACAT which as of this list: https://www.freebsd.org/releases/12.1R/hardware/#support
is supported officially by FreeBSD?
I found that card for ~230 bucks :)

You think it's worth a try?

Best regards!

[stephenw10]

@ogghi said in Load updated Intel IX module to get 10Gbps:

MCX4121A-ACAT

We have seen a number of reports of issues with that card specifically so, no, I wouldn't get that one. The user who reported getting close to 25Gbps was using a ConnectX-5 NIC.