Load updated Intel IX module to get 10Gbps

[stephenw10]

It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.

However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.

[ogghi]

@stephenw10
That's a tough one then?
How could I figure out what is limiting here?

[stephenw10]

Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611

I still find that hard to believe though.

[ogghi]

@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess...

[stephenw10]

The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.

[ogghi]

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:
login-to-view

[stephenw10]

That is a total throughput value though. You won't see that in an iperf3 test across an single link.

Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.

[ogghi]

@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least?

[stephenw10]

For a single TCP stream like that I would look for fewer cores with a higher frequency. And something newer is obviously more efficient so some relatively recent i3 or i5 for example.

There a few threads discussing specific options.

[ogghi]

@stephenw10 So you are saying that basically I could take an old desktop I have laying around and add some SFP+ card and that's it?
Just because of the higher CPU clock speed it would be faster?

Netgate 8200 for example has only 2.4GHz compared to our current Intel Xeon CPU D-1518 with 2.20GHz.

There won't be much of a difference then?

Also, can we be sure this is actually the limiting factor?

Eg. if I do an Iperf3 with 8 streams, this is not getting faster either? I am missing something conceptional here?

[stephenw10]

iperf itself is deliberately single threaded so you may be limited there. You might need to run multiple iperf processes.

No there's not much difference between the D1518 and C3758:
https://www.cpubenchmark.net/compare/2799vs3696vs4746vs3280/Intel-Xeon-D-1518-vs-Intel-Atom-C3758-vs-Intel-i3-12300-vs-Intel-i3-8300

[ogghi]

@stephenw10
So in theory I could use an old desktop PC with a CPU with 4+GHz and 16GB RAM, add a Intel X520-DA2 card and should be getting better bandwidth routed?

thanks :)

[stephenw10]

If you have one I would certainly try it. Be aware some of those old systems can be very power hungry though.

[ogghi]

@stephenw10
in the office it does not matter to have even 100W more running all time here, then I would first tell people to not leave their PCs running
I'll buy a card, install PFsense on a PC, backup current instance, restore on new machine, re-assign network interfaces where needed, test. Sounds like a plan?

[ogghi]

@stephenw10
Ok, seems that after all I shall not build a system myself, rather get an official appliance or something with warranty and possibly support.
What appliance would manage to give us the 10Gb here?

[Patch]

@ogghi said in Load updated Intel IX module to get 10Gbps:

@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option

What traffic mix is the operational requirement? More specifically

• WAN type
• VPN connections and load
• typical number of concurrent connections, especially high bandwidth connections.

The reason I ask is I wonder how well the test results correlate with the operational performance under likely load conditions.

[ogghi]

@Patch
Hi there.

WAN Type is Init7 10/10Gbps fiber.
VPN connections (speed not relevant here) maximum of 20 ppl on OpenVPN, 3 WireGuard tunnels.

pfTop: Up State 1-100/14500
Are those concurrent connections?

High bandwidth usually would be the backup server sending backups to S3 storage outside...otherwise not too much high bandwidth things happening.

[ogghi]

@Patch any idea?

[stephenw10]

The fastest one you can get! If you need to pass a single stream TCP connection through it at close to 10Gbps at least. Like sending backups to S3.

[ogghi]

@stephenw10 What do you mean with the fastest?
One with the fastest possible CPU?

Also I am wondering if the problem is really due to CPU clock / single stream? If I run multiple tests / transfers at the same time to different hosts, those tests will share those ~5Gbit...
?