Load updated Intel IX module to get 10Gbps

ogghi

@stephenw10 So you are saying that basically I could take an old desktop I have laying around and add some SFP+ card and that's it?
Just because of the higher CPU clock speed it would be faster?

Netgate 8200 for example has only 2.4GHz compared to our current Intel Xeon CPU D-1518 with 2.20GHz.

There won't be much of a difference then?

Also, can we be sure this is actually the limiting factor?

Eg. if I do an Iperf3 with 8 streams, this is not getting faster either? I am missing something conceptional here?

stephenw10

iperf itself is deliberately single threaded so you may be limited there. You might need to run multiple iperf processes.

No there's not much difference between the D1518 and C3758:
https://www.cpubenchmark.net/compare/2799vs3696vs4746vs3280/Intel-Xeon-D-1518-vs-Intel-Atom-C3758-vs-Intel-i3-12300-vs-Intel-i3-8300

ogghi

@stephenw10
So in theory I could use an old desktop PC with a CPU with 4+GHz and 16GB RAM, add a Intel X520-DA2 card and should be getting better bandwidth routed?

thanks :)

stephenw10

If you have one I would certainly try it. Be aware some of those old systems can be very power hungry though.

ogghi

@stephenw10
in the office it does not matter to have even 100W more running all time here, then I would first tell people to not leave their PCs running
I'll buy a card, install PFsense on a PC, backup current instance, restore on new machine, re-assign network interfaces where needed, test. Sounds like a plan?

ogghi

@stephenw10
Ok, seems that after all I shall not build a system myself, rather get an official appliance or something with warranty and possibly support.
What appliance would manage to give us the 10Gb here?

Patch

@ogghi said in Load updated Intel IX module to get 10Gbps:

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option

What traffic mix is the operational requirement? More specifically

WAN type
VPN connections and load
typical number of concurrent connections, especially high bandwidth connections.

The reason I ask is I wonder how well the test results correlate with the operational performance under likely load conditions.

ogghi

@Patch
Hi there.

WAN Type is Init7 10/10Gbps fiber.
VPN connections (speed not relevant here) maximum of 20 ppl on OpenVPN, 3 WireGuard tunnels.

pfTop: Up State 1-100/14500
Are those concurrent connections?

High bandwidth usually would be the backup server sending backups to S3 storage outside...otherwise not too much high bandwidth things happening.

ogghi

@Patch any idea?

stephenw10

The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.

ogghi

@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?

Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
?

stephenw10

Well in a test like that I'd expect to see 7-8Gbps through the D1541 so seeing 5Gbps with a D1518 is not wildly low.

But as I said we have seen reports of dramatically higher throughput using other NICs. I've not tested that myself to confirm though.

ogghi

@stephenw10
I guess I figured out what HW we have here:
Must be one of those guys, except we don't have the 4 port Ethernet card in:
https://www.newegg.com/supermicro-sys-5018d-fn8t-intel-xeon-processor-d-1518-2-2-ghz-cpu-tdp-support-35w-fcbga-1667/p/370-0003-000G9

Also now I am thinking: If the Mellanox cards might give us better throughput, I could get a MCX4121A-ACAT which as of this list: https://www.freebsd.org/releases/12.1R/hardware/#support
is supported officially by FreeBSD?
I found that card for ~230 bucks :)

You think it's worth a try?

Best regards!

stephenw10

@ogghi said in Load updated Intel IX module to get 10Gbps:

MCX4121A-ACAT

We have seen a number of reports of issues with that card specifically so, no, I wouldn't get that one. The user who reported getting close to 25Gbps was using a ConnectX-5 NIC.

ogghi

@stephenw10
All right, I'll try and get a Mellanox MCX512A-ACAT aka ConnectX-5 EN and report back :)

Thanks!

stephenw10

I will say that I've never tested that myself. It would be good to get a second test with it though as that first result was very surprising.

tman222

I ran a Xeon D-1518 based pfSense system up until recently and the best performance I saw, if I recall correctly, was around ~6-7Gbit/s routing traffic between two different internal network segments (no NAT, no IDS/IPS) via an iperf3 test (single stream). I imagine with NAT in the picture, performance through WAN would have been a bit lower than that.

stephenw10

With the included ix NICs I assume? That's about what I'd expect. Which is why the reports of 25Gbps with Mellanox NICs are so surprising.

ogghi

@stephenw10
hi there, I got the card.
How would I go to install it actually?
Install the card, change LAN setting to one of the new ports, apply, switch over cables?
Then do the same for WAN?

stephenw10

Yes, pretty much exactly that. Just switch the interface assignments to the new NICs.