10 GBit questions

rvdbijl

To wrap up this post -- I have my R210-II in with the E3-1275v2 CPU and 16GB RAM. I loaded pfSense, restored my backup and replaced the old i7 box with this one. I haven't tested with a 10G to 10G connection yet, but the 10G to 2.5G connection on two of my PC's seems to be able to push 2Gbit up and down to my ISP with no issues. I'll do some more benchmarking and post the results in the next few days.

Very happy that this box also seems to use ~50W while running, and is quiet as a mouse (once the fans have done their test when the system boots).

rvdbijl

And here are some results --

on a 2.5Gbit NIC and on a 10GBit NIC the performance is identical. I have no (easy) way to get more bandwidth on the WAN side of my connection, so I can't test beyond that. What I can see is how busy the box is while doing this Speedtest:

Not too shabby .. Looks like there is some more headroom.

Doing iperf3 testing is a bit more .. interesting. On a 10Gbit NIC I see this with pfSense as the client and my 10Gbit NIC PC as the server (the server is a Core i7-10700T running Win10):

And the process running iperf3 is using ~18% CPU:

The reverse path is worse (pfSense as server, 10Gbit NIC as client):

With the utilization here:

That is weird -- why does this path suck up 42-43% of CPU?

Multiple parallel threads don't seem to help here either. I'm guessing that this is some strange artifact of iperf3 running on the pfSense box?

In any case, I don't see this while running a speedtest in either direction (up or down) to my ISP. Solid 2Gbit as they promised. We'll see how this box does if/when my ISP raises speeds again. ;) I may try a VLAN-VLAN routing through this box and see how much data I can push, but that'll require another 10 Gbit NIC which I don't have .. yet .. ;)

Hope this benchmark data is at least helpful to some folks.

stephenw10

iperf is deliberately single threaded no matter how many parallel streams you set. To use multiple cores you need to run multiple instances of iperf.
iperf ends traffic from the client to the server by default so you're seeing worse performance when pfSense is receiving. Some of the hardware off-loading options may improve that.

johnpoz

@stephenw10 said in 10 GBit questions:

need to run multiple instances of iperf.

Or you could use the new beta that is out..

https://github.com/esnet/iperf/releases/tag/3.13-mt-beta2

iperf3 was originally designed as a single-threaded
program. Unfortunately, as network speeds increased faster than CPU
clock rates, this design choice meant that iperf3 became incapable of
using the bandwidth of the links in its intended operating environment
(high-performance R&E networks with Nx10Gbps or Nx100Gbps network
links and paths).

We have created a variant of iperf3 that uses a separate thread
(pthread) for each test stream. As the streams run more-or-less
independently, this should remove some of the performance bottlenecks,
and allow iperf3 to perform higher-speed tests, particularly on
100+Gbps paths. This version has recorded transfers as high as 148Gbps
in internal testing at ESnet.

stephenw10

Ooo, that's fun.