Load updated Intel IX module to get 10Gbps

ogghi

@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess...

stephenw10

The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.

ogghi

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:

stephenw10

That is a total throughput value though. You won't see that in an iperf3 test across an single link.

Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.

ogghi

@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least?

stephenw10

For a single TCP stream like that I would look for fewer cores with a higher frequency. And something newer is obviously more efficient so some relatively recent i3 or i5 for example.

There a few threads discussing specific options.

ogghi

@stephenw10 So you are saying that basically I could take an old desktop I have laying around and add some SFP+ card and that's it?
Just because of the higher CPU clock speed it would be faster?

Netgate 8200 for example has only 2.4GHz compared to our current Intel Xeon CPU D-1518 with 2.20GHz.

There won't be much of a difference then?

Also, can we be sure this is actually the limiting factor?

Eg. if I do an Iperf3 with 8 streams, this is not getting faster either? I am missing something conceptional here?

stephenw10

iperf itself is deliberately single threaded so you may be limited there. You might need to run multiple iperf processes.

No there's not much difference between the D1518 and C3758:
https://www.cpubenchmark.net/compare/2799vs3696vs4746vs3280/Intel-Xeon-D-1518-vs-Intel-Atom-C3758-vs-Intel-i3-12300-vs-Intel-i3-8300

ogghi

@stephenw10
So in theory I could use an old desktop PC with a CPU with 4+GHz and 16GB RAM, add a Intel X520-DA2 card and should be getting better bandwidth routed?

thanks :)

stephenw10

If you have one I would certainly try it. Be aware some of those old systems can be very power hungry though.

ogghi

@stephenw10
in the office it does not matter to have even 100W more running all time here, then I would first tell people to not leave their PCs running
I'll buy a card, install PFsense on a PC, backup current instance, restore on new machine, re-assign network interfaces where needed, test. Sounds like a plan?

ogghi

@stephenw10
Ok, seems that after all I shall not build a system myself, rather get an official appliance or something with warranty and possibly support.
What appliance would manage to give us the 10Gb here?

Patch

@ogghi said in Load updated Intel IX module to get 10Gbps:

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option

What traffic mix is the operational requirement? More specifically

WAN type
VPN connections and load
typical number of concurrent connections, especially high bandwidth connections.

The reason I ask is I wonder how well the test results correlate with the operational performance under likely load conditions.

ogghi

@Patch
Hi there.

WAN Type is Init7 10/10Gbps fiber.
VPN connections (speed not relevant here) maximum of 20 ppl on OpenVPN, 3 WireGuard tunnels.

pfTop: Up State 1-100/14500
Are those concurrent connections?

High bandwidth usually would be the backup server sending backups to S3 storage outside...otherwise not too much high bandwidth things happening.

ogghi

@Patch any idea?

stephenw10

The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.

ogghi

@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?

Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
?

stephenw10

Well in a test like that I'd expect to see 7-8Gbps through the D1541 so seeing 5Gbps with a D1518 is not wildly low.

But as I said we have seen reports of dramatically higher throughput using other NICs. I've not tested that myself to confirm though.

ogghi

@stephenw10
I guess I figured out what HW we have here:
Must be one of those guys, except we don't have the 4 port Ethernet card in:
https://www.newegg.com/supermicro-sys-5018d-fn8t-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/370-0003-000G9

Also now I am thinking: If the Mellanox cards might give us better throughput, I could get a MCX4121A-ACAT which as of this list: https://www.freebsd.org/releases/12.1R/hardware/#support
is supported officially by FreeBSD?
I found that card for ~230 bucks :)

You think it's worth a try?

Best regards!

stephenw10

@ogghi said in Load updated Intel IX module to get 10Gbps:

MCX4121A-ACAT

We have seen a number of reports of issues with that card specifically so, no, I wouldn't get that one. The user who reported getting close to 25Gbps was using a ConnectX-5 NIC.