Slow upload speeds on HP Z2 G9 PFSense Box
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
Yes there is a CLI client. In fact there are two:
[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest py311-speedtest-cli-2.1.3 Command line interface for testing internet bandwidth speedtest-go-1.7.9 CLI and Go API to test internet speed using speedtest.netYou can install and run those directly:
[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: py311-speedtest-cli: 2.1.3 [pfSense] Number of packages to be installed: 1 52 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100% 52 KiB 53.1kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing py311-speedtest-cli-2.1.3... [1/1] Extracting py311-speedtest-cli-2.1.3: 100% [25.03-RC][admin@5100.stevew.lan]/root: rehashAs I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.
Sadly, it's completely unreliable running on the FW. Getting 450mbit up and 45 down on the 6100, since it's picking a completely different server than even my other FreeBSD box.
-
Tried one more test - Put a QNAP Multigig switch in between the PFSense box and the RG.
The 6100 maintained its speeds.
The HP Box also maintained its lower upload speeds.
Went into the Z2G9's BIOS (it's running the latest) and disabled PCI Express Power Management.
Upload speeds remained at 450-700Mbits, their typical range. Going back to the 6100 yielded the usual 3800-4500 up and down.
Almost at my wit's end. This is a very standard build using HP workstation components and Intel NICs. No idea why I'm seeing this.
Any ideas?
-Bear
-
You can specify which server it uses:
--list Display a list of speedtest.net servers sorted by distance --server SERVER Specify a server ID to test against. Can be supplied multiple times -
✓ Test Server: [68863] 0.00km San Antonio, TX by AT&T ✓ Latency: 9.472954ms Jitter: 1.390244ms Min: 8.694556ms Max: 13.568589ms ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 1350.20 Mbps (Used: 1708.20MB) (Latency: 15ms Jitter: 10ms Min: 8ms Max: 44ms) ✓ Upload: 763.02 Mbps (Used: 930.43MB) (Latency: 11ms Jitter: 2ms Min: 8ms Max: 13ms) ✓ Packet Loss: N/AI'm getting really slow speeds from the 6100 vs my Mac or Windows system with 10GBE to the point where this up speed is aligned with the ~750mbit I get on those systems with the HP/X520.
In other words, the slow performance test locally on the 6100 would mask the issue as I get 3500+mbits up when running speedtest from a local system vs 760 from the 6100 itself.
-
Hmm, OK. Probably not useful data there then.
Do you see the same in an iperf test?
-
@stephenw10 Haven't run iPerf yet. I'll try that later, but I'm not sure what data we're after here; the performance is identical for low upload speeds using 2 different Intel NICs in the same x16 slot @ x8, and both WAN and LAN are on the same NIC.
Flow control is off, PCI power management is off, and with an intermediary multigig switch, results remain identical.
-
If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.
Also you can run iperf directly from pfSense with more useful results.
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.
Also you can run iperf directly from pfSense with more useful results.
IPerf isn't jibing with speedtest. Here's my recent speedtest result: https://www.speedtest.net/my-result/d/739237147
Here's a few iPERF 3 runs to supposedly local addresses:
Connecting to host 37.19.206.20, port 5201 [ 5] local 75.31.24.1 port 38531 connected to 37.19.206.20 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.2 MBytes 237 Mbits/sec 0 6.41 MBytes [ 5] 1.00-2.00 sec 46.9 MBytes 393 Mbits/sec 0 6.41 MBytes [ 5] 2.00-3.01 sec 47.6 MBytes 397 Mbits/sec 0 6.41 MBytes [ 5] 3.01-4.00 sec 46.9 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 4.00-5.00 sec 47.2 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 5.00-6.00 sec 47.2 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 6.00-7.03 sec 48.2 MBytes 395 Mbits/sec 0 6.41 MBytes [ 5] 7.03-8.00 sec 46.0 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 8.00-9.00 sec 47.5 MBytes 398 Mbits/sec 0 6.41 MBytes [ 5] 9.00-10.00 sec 47.4 MBytes 397 Mbits/sec 0 6.41 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 453 MBytes 380 Mbits/sec 0 sender [ 5] 0.00-10.05 sec 453 MBytes 379 Mbits/sec receiverConnecting to host dal.speedtest.clouvider.net, port 5200 [ 5] local 75.31.24.1 port 48312 connected to 2.56.188.136 port 5200 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.01 sec 5.62 MBytes 46.9 Mbits/sec 8 92.8 KBytes [ 5] 1.01-2.00 sec 11.2 MBytes 94.8 Mbits/sec 0 162 KBytes [ 5] 2.00-3.00 sec 17.8 MBytes 149 Mbits/sec 0 231 KBytes [ 5] 3.00-4.02 sec 20.9 MBytes 172 Mbits/sec 3 220 KBytes [ 5] 4.02-5.00 sec 19.1 MBytes 163 Mbits/sec 4 179 KBytes [ 5] 5.00-6.01 sec 19.2 MBytes 160 Mbits/sec 0 248 KBytes [ 5] 6.01-7.00 sec 24.8 MBytes 210 Mbits/sec 0 316 KBytes [ 5] 7.00-8.01 sec 22.2 MBytes 186 Mbits/sec 81 114 KBytes [ 5] 8.01-9.01 sec 10.9 MBytes 91.1 Mbits/sec 46 34.7 KBytes [ 5] 9.01-10.00 sec 6.12 MBytes 51.5 Mbits/sec 0 104 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 158 MBytes 132 Mbits/sec 142 sender [ 5] 0.00-10.01 sec 157 MBytes 132 Mbits/sec receiverI can consistently get similar speeds to other providers as well. iperf3 isn't getting me close to a gigabit. Real world transfers are in the 2.5-3.5Gbit range usually.
-
Now, if I do the iperf3 from my Mac Studio with 10GBE going /through/ the 6100 Max...
Connecting to host speedtest.dal13.us.leaseweb.net, port 5201 [ 7] local 75.31.24.8 port 61252 connected to 209.58.159.68 port 5201 [ ID] Interval Transfer Bitrate [ 7] 0.00-1.01 sec 186 MBytes 1.55 Gbits/sec [ 7] 1.01-2.00 sec 247 MBytes 2.08 Gbits/sec [ 7] 2.00-3.01 sec 249 MBytes 2.08 Gbits/sec [ 7] 3.01-4.01 sec 247 MBytes 2.07 Gbits/sec [ 7] 4.01-5.01 sec 248 MBytes 2.08 Gbits/sec [ 7] 5.01-6.01 sec 247 MBytes 2.07 Gbits/sec [ 7] 6.01-7.01 sec 248 MBytes 2.08 Gbits/sec [ 7] 7.01-8.01 sec 220 MBytes 1.84 Gbits/sec [ 7] 8.01-9.00 sec 207 MBytes 1.74 Gbits/sec [ 7] 9.00-10.01 sec 216 MBytes 1.81 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 7] 0.00-10.01 sec 2.26 GBytes 1.94 Gbits/sec sender [ 7] 0.00-10.02 sec 2.26 GBytes 1.94 Gbits/sec receiverAnd do the same iperf3 test from the 6100 Max itself, just a moment later to the same iperf3 server...
Connecting to host speedtest.dal13.us.leaseweb.net, port 5201 [ 5] local 75.31.24.1 port 8528 connected to 209.58.159.68 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 11.4 MBytes 95.3 Mbits/sec 60 145 KBytes [ 5] 1.00-2.00 sec 6.88 MBytes 57.7 Mbits/sec 37 79.5 KBytes [ 5] 2.00-3.00 sec 11.9 MBytes 99.6 Mbits/sec 0 156 KBytes [ 5] 3.00-4.01 sec 10.5 MBytes 87.6 Mbits/sec 68 53.7 KBytes [ 5] 4.01-5.02 sec 7.50 MBytes 62.3 Mbits/sec 19 108 KBytes [ 5] 5.02-6.02 sec 10.6 MBytes 88.9 Mbits/sec 19 42.7 KBytes [ 5] 6.02-7.00 sec 7.88 MBytes 67.2 Mbits/sec 0 118 KBytes [ 5] 7.00-8.00 sec 15.6 MBytes 131 Mbits/sec 0 194 KBytes [ 5] 8.00-9.01 sec 23.8 MBytes 197 Mbits/sec 0 271 KBytes [ 5] 9.01-10.02 sec 31.8 MBytes 263 Mbits/sec 0 348 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.02 sec 138 MBytes 115 Mbits/sec 203 sender [ 5] 0.00-10.03 sec 137 MBytes 115 Mbits/sec receiver -
Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.
Can you test using the i5?
Was that with multiple parallel streams?
The default direction for iperf3 is from the client to the server, so upload. Try using
-Rto get a download result. -
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.
Can you test using the i5?
Was that with multiple parallel streams?
The default direction for iperf3 is from the client to the server, so upload. Try using
-Rto get a download result.6100 Max With -R and -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.28 sec 3.21 GBytes 2.68 Gbits/sec 9325 sender [SUM] 0.00-10.00 sec 2.88 GBytes 2.47 Gbits/sec receiverMac Studio behind 6100 Max with -R and -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 5.26 GBytes 4.52 Gbits/sec 11595 sender [SUM] 0.00-10.00 sec 5.12 GBytes 4.40 Gbits/sec receiver6100 Max with just -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.07 sec 3.10 GBytes 2.65 Gbits/sec 12761 sender [SUM] 0.00-10.09 sec 3.04 GBytes 2.59 Gbits/sec receiverMac Studio behind 6100 Max with just -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 2.06 GBytes 1.77 Gbits/sec sender [SUM] 0.00-10.02 sec 2.05 GBytes 1.76 Gbits/sec receiverNot quite my SpeedTest Results, but enough that I can try the same on the i5 later tonight and append this thread with some of those numbers.
-
Probably hitting the CPU core limit on the 6100 there. But it's at least the same both ways and more than the 400Mps limit you hit previously. So try the i5 there and see if it still hits it.
-
I'll run the same tests tonight - I need to take my network offline to jump to the i5-based system, so it'll need to be after hours this evening, but I'll get some data here soon. Thanks for the help. :)
-Bear
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
Probably hitting the CPU core limit on the 6100 there. But it's at least the same both ways and more than the 400Mps limit you hit previously. So try the i5 there and see if it still hits it.
Getting a No Active Repos when I try to install iperf3 on the i5.
I'm going to try a clean install (And not use ZFS as the fs this time), restore my old config and try again.
Will likely be a tomorrow thing, but I'll update as soon as I have more data.
-
If you just tried to install it at the CLI without visiting the webgui first you probably need to run:
pfSense-repoc -Nfirst. Then retry. -
@stephenw10 Looks like when I swapped NICs, I lost my registration for some reason.
What do I need to do in order to get a new token? It's not on my PDF. Order SO25-338254 - Sent you my NDI via Chat.
-
I've got the numbers from the i5-14500 /w the X520, flow control off.
With - P -R
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 5.20 GBytes 4.46 Gbits/sec 10655 sender [SUM] 0.00-10.00 sec 5.07 GBytes 4.35 Gbits/sec receiverWith -P
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 1.61 GBytes 1.38 Gbits/sec 27815 sender [SUM] 0.00-10.02 sec 1.60 GBytes 1.37 Gbits/sec receiverNote that with Speed Test, I initially see speeds of 1.6-1.7Gbit that quickly slow down to 400-650Mbits. This does not happen with the 6100.
-
Hmm, so that looks like the same result as a client behind the 6100.
Does a client behind the i5 still see a much lower limit on iperf? Like ~400Mbps?
If so that implies a LAN side issue with traffic going into pfSense.
-
@stephenw10 Yes, a client behind the i5 sees a much lower limit. Though I'm not sure where to go with this from here. Same cables, same switch between both units. Same config even, since it was a restore.
-
Hmm, well that seems to confirm some issue on the LAN side then.
Still no input errors on the LAN interface?
Can you try connecting a client to the LAN directly to rule out some low level issue between the LAN and switch?
