Slow upload speeds on HP Z2 G9 PFSense Box

[Bear]

@stephenw10

Now, if I do the iperf3 from my Mac Studio with 10GBE going /through/ the 6100 Max...

Connecting to host speedtest.dal13.us.leaseweb.net, port 5201
[  7] local 75.31.24.8 port 61252 connected to 209.58.159.68 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.01   sec   186 MBytes  1.55 Gbits/sec                  
[  7]   1.01-2.00   sec   247 MBytes  2.08 Gbits/sec                  
[  7]   2.00-3.01   sec   249 MBytes  2.08 Gbits/sec                  
[  7]   3.01-4.01   sec   247 MBytes  2.07 Gbits/sec                  
[  7]   4.01-5.01   sec   248 MBytes  2.08 Gbits/sec                  
[  7]   5.01-6.01   sec   247 MBytes  2.07 Gbits/sec                  
[  7]   6.01-7.01   sec   248 MBytes  2.08 Gbits/sec                  
[  7]   7.01-8.01   sec   220 MBytes  1.84 Gbits/sec                  
[  7]   8.01-9.00   sec   207 MBytes  1.74 Gbits/sec                  
[  7]   9.00-10.01  sec   216 MBytes  1.81 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.01  sec  2.26 GBytes  1.94 Gbits/sec    sender
[  7]   0.00-10.02  sec  2.26 GBytes  1.94 Gbits/sec    receiver

And do the same iperf3 test from the 6100 Max itself, just a moment later to the same iperf3 server...

Connecting to host speedtest.dal13.us.leaseweb.net, port 5201
[  5] local 75.31.24.1 port 8528 connected to 209.58.159.68 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.4 MBytes  95.3 Mbits/sec   60    145 KBytes
[  5]   1.00-2.00   sec  6.88 MBytes  57.7 Mbits/sec   37   79.5 KBytes
[  5]   2.00-3.00   sec  11.9 MBytes  99.6 Mbits/sec    0    156 KBytes
[  5]   3.00-4.01   sec  10.5 MBytes  87.6 Mbits/sec   68   53.7 KBytes
[  5]   4.01-5.02   sec  7.50 MBytes  62.3 Mbits/sec   19    108 KBytes
[  5]   5.02-6.02   sec  10.6 MBytes  88.9 Mbits/sec   19   42.7 KBytes
[  5]   6.02-7.00   sec  7.88 MBytes  67.2 Mbits/sec    0    118 KBytes
[  5]   7.00-8.00   sec  15.6 MBytes   131 Mbits/sec    0    194 KBytes
[  5]   8.00-9.01   sec  23.8 MBytes   197 Mbits/sec    0    271 KBytes
[  5]   9.01-10.02  sec  31.8 MBytes   263 Mbits/sec    0    348 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   138 MBytes   115 Mbits/sec  203    sender
[  5]   0.00-10.03  sec   137 MBytes   115 Mbits/sec         receiver

[stephenw10]

Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.

Can you test using the i5?

Was that with multiple parallel streams?

The default direction for iperf3 is from the client to the server, so upload. Try using -R to get a download result.

[Bear]

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.

Can you test using the i5?

Was that with multiple parallel streams?

The default direction for iperf3 is from the client to the server, so upload. Try using -R to get a download result.

6100 Max With -R and -P 120

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R

[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.28  sec  3.21 GBytes  2.68 Gbits/sec  9325    sender
[SUM]   0.00-10.00  sec  2.88 GBytes  2.47 Gbits/sec          receiver

Mac Studio behind 6100 Max with -R and -P 120

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R
[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.01  sec  5.26 GBytes  4.52 Gbits/sec  11595     sender
[SUM]   0.00-10.00  sec  5.12 GBytes  4.40 Gbits/sec            receiver

6100 Max with just -P 120

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120

[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.07  sec  3.10 GBytes  2.65 Gbits/sec  12761    sender
[SUM]   0.00-10.09  sec  3.04 GBytes  2.59 Gbits/sec           receiver

Mac Studio behind 6100 Max with just -P 120

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120
[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.01  sec  2.06 GBytes  1.77 Gbits/sec           sender
[SUM]   0.00-10.02  sec  2.05 GBytes  1.76 Gbits/sec           receiver

Not quite my SpeedTest Results, but enough that I can try the same on the i5 later tonight and append this thread with some of those numbers.

[stephenw10]

Probably hitting the CPU core limit on the 6100 there. But it's at least the same both ways and more than the 400Mps limit you hit previously. So try the i5 there and see if it still hits it.

[Bear]

@stephenw10

I'll run the same tests tonight - I need to take my network offline to jump to the i5-based system, so it'll need to be after hours this evening, but I'll get some data here soon. Thanks for the help. :)

-Bear

[Bear]

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Probably hitting the CPU core limit on the 6100 there. But it's at least the same both ways and more than the 400Mps limit you hit previously. So try the i5 there and see if it still hits it.

Getting a No Active Repos when I try to install iperf3 on the i5.

I'm going to try a clean install (And not use ZFS as the fs this time), restore my old config and try again.

Will likely be a tomorrow thing, but I'll update as soon as I have more data.

[stephenw10]

If you just tried to install it at the CLI without visiting the webgui first you probably need to run: pfSense-repoc -N first. Then retry.

[Bear]

@stephenw10 Looks like when I swapped NICs, I lost my registration for some reason.

What do I need to do in order to get a new token? It's not on my PDF. Order SO25-338254 - Sent you my NDI via Chat.

[Bear]

@stephenw10

I've got the numbers from the i5-14500 /w the X520, flow control off.

With - P -R

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R
[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.01  sec  5.20 GBytes  4.46 Gbits/sec  10655       sender
[SUM]   0.00-10.00  sec  5.07 GBytes  4.35 Gbits/sec              receiver

With -P

# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120
[ ID]   Interval         Transfer     Bitrate         Retr
[SUM]   0.00-10.01  sec  1.61 GBytes  1.38 Gbits/sec  27815      sender
[SUM]   0.00-10.02  sec  1.60 GBytes  1.37 Gbits/sec             receiver

Note that with Speed Test, I initially see speeds of 1.6-1.7Gbit that quickly slow down to 400-650Mbits. This does not happen with the 6100.

[stephenw10]

Hmm, so that looks like the same result as a client behind the 6100.

Does a client behind the i5 still see a much lower limit on iperf? Like ~400Mbps?

If so that implies a LAN side issue with traffic going into pfSense.

[Bear]

@stephenw10 Yes, a client behind the i5 sees a much lower limit. Though I'm not sure where to go with this from here. Same cables, same switch between both units. Same config even, since it was a restore.

[stephenw10]

Hmm, well that seems to confirm some issue on the LAN side then.

Still no input errors on the LAN interface?

Can you try connecting a client to the LAN directly to rule out some low level issue between the LAN and switch?

[Bear]

@stephenw10 I've tried that. Still getting the same speeds.

[stephenw10]

So to be clear the client behind the i5 sees ~400Mbps both for iperf and speedtest.net? Whether or not via the switch. And no errors are shown on the NIC.

Hmm. What about an iperf test between the client and pfSense directly? Does that also get throttled for traffic going into the LAN NIC?

[Bear]

@Bear To put a finer point on it, there is no "real" LAN side besides physical, since I'm running a filtered bridge. The LAN side is a filtered port on the same NIC that connects to a Netgear L3 multigig switch.

[stephenw10]

Ah, yes I forgot about that. From a packets-in packets-out perspective it makes no difference. It still feels like the internal NIC causing the issue.

However I think it would be worth testing a more basic routing setup if you can just to be sure it still shows the same issue. Hard to see why the 6100 would behave any differently but the fact you're running a bridged setup is at least unusual.

[Bear]

@stephenw10

I’ve already tried two completely different Intel NICs. The i7-14500 has more than enough power to run in filtering bridge mode since the 6100 is doing it as well using the same config.

I’ve gathered a bunch of data already - I can’t reconfigure the network as I’m running a /25 of public IPs with lots of firewall rules.

Is there a potential issue with the software and more recent Intel hardware here? Again, I’m running as plain and supported a piece of hardware as I can, save for it being a much later model Intel system.

[stephenw10]

You replaced the i5 with an i7? Or is that the client you're testing from? Not that it should matter.

Bridging has always been somewhat fragile in pfSense/FreeBSD and can create some unexpected traffic scenarios. It would be good to rule that out entirely if you can. But, yes, I agree that it seems unlikely here since the 6100 passes it OK.

[Bear]

@stephenw10 Sorry, it’s an i5-14500.

[Bear]

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

You replaced the i5 with an i7? Or is that the client you're testing from? Not that it should matter.

Bridging has always been somewhat fragile in pfSense/FreeBSD and can create some unexpected traffic scenarios. It would be good to rule that out entirely if you can. But, yes, I agree that it seems unlikely here since the 6100 passes it OK.

Any other suggestions for diagnostics here? I'm just about at my wit's end - This is a relatively high end workstation with ECC RAM, and otherwise all standardized components.