Load updated Intel IX module to get 10Gbps
-
@stephenw10
LOL, easy, but sometimes still hidden from men's eyes...Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz 8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (active) QAT Crypto: NoWhen our provider once ran a check (some raw ping flood I guess?) they filled up the 10Gbps.
Also I tried some other iperf3 servers, when they don't report busy, always the same limitation in speed.
Also this is something visible when downloading / uploading from/to a fast server in the net. -
Hmm if you're able to test between two local machines with iperf I would try that first. And if you can confirm they will reach 10Gbps when on the same subnet that eliminates any doubt.
-
@stephenw10 Thanks for getting back to me that quickly!
From a VM on an ESX host to a VM on another ESX host I get ~9Gbps, so looks fine.From or to one of those VMs to the pFsense (I installed iperf3 to test) I get even less, around 2,7Gbps.
This seems odd? To exclude the network behind the pFsense I also once tried to have a PC with a 10Gb fiber card directly attached to the switch that comes right after the pFsense.
Same results though. -
Running iperf to or from pfSense dircetly will always give a lower result. Both because iperf itself uses significant CPU cycles and because pfSense is optimised as a router and not a server.
Test through pfSense between different interfaces if you can.
-
@stephenw10 our setup here is more simple than others perhaps.
Just ix0 for WAN, ix1 for LAN.
Some VLAN interfaces on LAN (ix1) and a slower backup link on igb5 -
Well if you're able to test against a local server on the WAN side that would be a good test.
-
@stephenw10 I should probably enable post notifications...
The only way here would be for me to disconnect the WAN interface from the modem and attach a server to it?Only other way could be if the provider modem has another 10Gb port and I connect a server there, then give it another public IP we still have. But then traffic would already passing through the modem (which should certainly support 10Gb as the provider uses it for lots of customers)
-
Hmm, you could use a switch on the WAN side but it would obviously have to be 10G capable.
That CPU is probably somewhere between the 1537 and 1541 for routing that traffic. I'd expect to see somewhere around 5-6Gbps with pf and NAT enabled.
-
@stephenw10
What do you mean with between 1537 and 1541?
The Intel(R) Xeon(R) CPU D-1518 ?So you think the limiting factor here could be the CPU?
Would we need to get another Netgate certified appliance here to get the full 10Gb?
-
It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.
However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.
-
@stephenw10
That's a tough one then?
How could I figure out what is limiting here? -
Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611
I still find that hard to believe though.
-
@stephenw10
Interesting!
I am wondering, should I try getting such a Mellanox card?
Or should we be fine achieving 10Gbps with a 6100 for example?
That would be fine to buy I guess... -
The C3558 in the 6100 will not pass 10Gbps either. Again there are a lot of variables but I'd expect to see something in the 3-4Gbps range for an equivalent config.
-
@stephenw10
So to get 10Gbps we would need much bigger hardware then?
I thought the 6100 could be an option, looking at this:
-
That is a total throughput value though. You won't see that in an iperf3 test across an single link.
Yes you need significant CPU power if you want to pass 10Gbps using a single TCP stream like downloading a file.
-
@stephenw10 what appliance would you suggest here?
Not sure if budget allows me to get another PFsense, but I could ask
Otherwise, building something could be an option?
What specs would you recommend to have at least? -
For a single TCP stream like that I would look for fewer cores with a higher frequency. And something newer is obviously more efficient so some relatively recent i3 or i5 for example.
There a few threads discussing specific options.
-
@stephenw10 So you are saying that basically I could take an old desktop I have laying around and add some SFP+ card and that's it?
Just because of the higher CPU clock speed it would be faster?Netgate 8200 for example has only 2.4GHz compared to our current Intel Xeon CPU D-1518 with 2.20GHz.
There won't be much of a difference then?
Also, can we be sure this is actually the limiting factor?
Eg. if I do an Iperf3 with 8 streams, this is not getting faster either? I am missing something conceptional here?
-
iperf itself is deliberately single threaded so you may be limited there. You might need to run multiple iperf processes.
No there's not much difference between the D1518 and C3758:
https://www.cpubenchmark.net/compare/2799vs3696vs4746vs3280/Intel-Xeon-D-1518-vs-Intel-Atom-C3758-vs-Intel-i3-12300-vs-Intel-i3-8300
