Why am I only getting 1 gigabit speed instead of 10 gigabit on my new Netgate XG-7100?

stephenw10

So you are connected directly with both ends using VLAN 11 tagged traffic? But you would otherwise usually be using a 10G switch?

You are actually seeing more that 1G so it is linked at 10G.

How are you running the test? More than 1 parallel stream?

Testing to/from pfSense directly will always give a worse result than expected. pfSense is optimised as a router and not as a server, it performs badly as a TCP endpoint. The iperf3 process itself uses significant CPU cycles that would otherwise be forwarding traffic.
You should test between two clients through pfSense if you can.

Steve

mer

Running iperf server and client on the same host, exactly what does that do? I can't believe that it actually pushes traffic out of the physical interface; on a Linux/*nix system I would expect the data to never leave the box, probably on the stack with zero copy stuff going on. Basically I'm not sure what that test would exactly "prove".

netblues

@mer
Same goes for bsd.
This is on a kvm virtualized pf, on i5. All physical interfaces are at 1Gbit.

Connecting to host 192.168.31.1, port 5201
[ 5] local 192.168.31.1 port 33173 connected to 192.168.31.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.54 GBytes 39.0 Gbits/sec 0 864 KBytes
[ 5] 1.00-2.00 sec 4.63 GBytes 39.8 Gbits/sec 0 1.62 MBytes
[ 5] 2.00-3.00 sec 4.45 GBytes 38.3 Gbits/sec 0 2.01 MBytes
[ 5] 3.00-4.00 sec 4.35 GBytes 37.4 Gbits/sec 0 2.01 MBytes
[ 5] 4.00-5.00 sec 4.36 GBytes 37.4 Gbits/sec 0 2.01 MBytes
[ 5] 5.00-6.00 sec 4.43 GBytes 38.1 Gbits/sec 0 2.01 MBytes
[ 5] 6.00-7.00 sec 3.13 GBytes 26.9 Gbits/sec 0 2.01 MBytes
[ 5] 7.00-8.00 sec 4.35 GBytes 37.4 Gbits/sec 0 2.01 MBytes
[ 5] 8.00-9.00 sec 4.46 GBytes 38.3 Gbits/sec 0 2.01 MBytes
[ 5] 9.00-10.00 sec 4.40 GBytes 37.8 Gbits/sec 0 2.01 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 43.1 GBytes 37.0 Gbits/sec 0 sender
[ 5] 0.00-10.05 sec 43.1 GBytes 36.8 Gbits/sec receiver

iperf Done.

stephenw10

Indeed, running client and server on the same box is only testing it's ability to run iperf3.

mer

@stephenw10, @netblues Thanks for the confirmation. I read the OP a few times earlier today and having no input for the stated problem, the last test just kept bothering me.

Fallon

@stephenw10 said in Why am I only getting 1 gigabit speed instead of 10 gigabit on my new Netgate XG-7100?:

So you are connected directly with both ends using VLAN 11 tagged traffic? But you would otherwise usually be using a 10G switch?

Correct. I was connected via my 10gb switches & VLAN 11 or 12 when I went to validate performance & found 1G rather than the expected 10G, I disassembled things down to the minimum possible config to eliminate the switches as the potential bottleneck. I left things with VLAN 11to avoid reconfiguring the interfaces any further & just tagged VLAN 11 on the workstation itself.

You are actually seeing more that 1G so it is linked at 10G.

How are you running the test? More than 1 parallel stream?

Just 1 stream. iperf3 as a server on the pfSense box & client on the workstation. I also ran client AND server on the workstation as a test & got expected 10G speeds.

Testing to/from pfSense directly will always give a worse result than expected. pfSense is optimised as a router and not as a server, it performs badly as a TCP endpoint. The iperf3 process itself uses significant CPU cycles that would otherwise be forwarding traffic.
You should test between two clients through pfSense if you can.

Seems a little odd to me. https://github.com/mendel5/iperf3-results does indicate you shouldn't be seeing above 940 MBit/s on a gigabit connection & I'm a hair above that. I was assuming it was rounding or buffering inconsistencies in the output or something.

only 1 10G endpoint at the moment. Unless I pull the currently unused 2 port NIC out of my XG-7100 & put it in the wife's PC for testing. That's living dangerously though so I'll probably have to sort out another endpoint or a 10G thunderbolt adapter for my work laptop, if that will play nicely. Or find a spare PC T 10G NIC.

stephenw10

Yeah you are seeing more than the 941Mbps maximum Gigabit will give you which indicates it _is_linked at 10G.

With 1 iperf process and 1 stream you are using using 1 NIC queue and potentially 1CPU core.
Try running it with 4 or 8 streams. You should at least see some numbers that are significantly above 1G. But it will still be quite a lot lower than testing through the firewall.

Steve

Fallon

@stephenw10 said in Why am I only getting 1 gigabit speed instead of 10 gigabit on my new Netgate XG-7100?:

Yeah you are seeing more than the 941Mbps maximum Gigabit will give you which indicates it _is_linked at 10G.

With 1 iperf process and 1 stream you are using using 1 NIC queue and potentially 1CPU core.
Try running it with 4 or 8 streams. You should at least see some numbers that are significantly above 1G. But it will still be quite a lot lower than testing through the firewall.

Steve

Firing up 5 instances of iperf on the pfsense box

iperf3 -s -p 5101&; iperf3 -s -p 5102&; iperf3 -s -p 5103&; iperf3 -s -p 5104&; iperf3 -s -p 5105 &

Then launching 5 concurrent clients on my windows box

start iperf3.exe -c 10.10.11.1 -t 100 -p 5101
start iperf3.exe -c 10.10.11.1 -t 100 -p 5102
start iperf3.exe -c 10.10.11.1 -t 100 -p 5103
start iperf3.exe -c 10.10.11.1 -t 100 -p 5104
start iperf3.exe -c 10.10.11.1 -t 100 -p 5105

Gets me 5 concurrent stream that each wrap up in the 500Mbits-1.1Gbit/sec range, so a couple gigabit. Which i guess answers by fundamental fundamental question of "Did I get this thing hooked up through several hops at 10 gigabit?", enough to stop messing with it for now. Well, at least when I unwind my changes & put it back in the rack.

Fallon

Oh, looks like you can do parallel threads natively too.

8 threads

iperf3.exe -c 10.10.11.1 -t 10 -P 8

gets just over 4.5 gigabit

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   548 MBytes   460 Mbits/sec                  sender
[  4]   0.00-10.00  sec   548 MBytes   460 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   763 MBytes   640 Mbits/sec                  sender
[  6]   0.00-10.00  sec   763 MBytes   640 Mbits/sec                  receiver
[  8]   0.00-10.00  sec   783 MBytes   657 Mbits/sec                  sender
[  8]   0.00-10.00  sec   783 MBytes   657 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec   769 MBytes   645 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   769 MBytes   645 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec   538 MBytes   451 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   537 MBytes   451 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec   765 MBytes   642 Mbits/sec                  sender
[ 14]   0.00-10.00  sec   765 MBytes   642 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec   762 MBytes   639 Mbits/sec                  sender
[ 16]   0.00-10.00  sec   762 MBytes   639 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec   540 MBytes   453 Mbits/sec                  sender
[ 18]   0.00-10.00  sec   540 MBytes   453 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  5.34 GBytes  4.59 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  5.34 GBytes  4.58 Gbits/sec                  receiver

iperf Done.

stephenw10

Ah, there we go that's about what I'd expect to see.

One of the interesting things about iperf is that it's deliberately designed to be single threaded. Running multiple parallel streams using the '-P' switch does not change that, you are still running one iperf process. But, as you already tried, that means you can run it multiple times to test combinations of CPU cores and streams. You still see a better result using -P because the firewall and the NICs can use multiple queues and therefore CPU cores to move that traffic.

Steve