Slow upload speeds on HP Z2 G9 PFSense Box
-
@stephenw10 Not seeing any errors on the LAN side at all. Interrupts aren't too bad either.
I'm running a filtered bridge. Same config as on the 6100.
Getting the same results as when I used the 710 as well, completely different NIC.
-
How are you testing? Do you see any packet loss?
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
How are you testing? Do you see any packet loss?
Just using speedtest and fast.com from 10GBE Macs and Linux boxes with Intel NICs - Seeing zero packet loss and 8-10ms ping times.
-
Hmm, you might try running a test from pfSense directly. It won't be accurate but if you see the same big difference between up and downloads that at least narrows it down to the WAN side.
Can you see anything about the link state from the connected modem.
-
Both WAN and LAN are on the same X520 NIC. Is there a native speedtest on pfsense?
-
Yes there is a CLI client. In fact there are two:
[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest py311-speedtest-cli-2.1.3 Command line interface for testing internet bandwidth speedtest-go-1.7.9 CLI and Go API to test internet speed using speedtest.netYou can install and run those directly:
[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: py311-speedtest-cli: 2.1.3 [pfSense] Number of packages to be installed: 1 52 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100% 52 KiB 53.1kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing py311-speedtest-cli-2.1.3... [1/1] Extracting py311-speedtest-cli-2.1.3: 100% [25.03-RC][admin@5100.stevew.lan]/root: rehashAs I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.
-
I can try that later today. I’ll post results soon.
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
Yes there is a CLI client. In fact there are two:
[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest py311-speedtest-cli-2.1.3 Command line interface for testing internet bandwidth speedtest-go-1.7.9 CLI and Go API to test internet speed using speedtest.netYou can install and run those directly:
[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli Updating pfSense-core repository catalogue... pfSense-core repository is up to date. Updating pfSense repository catalogue... pfSense repository is up to date. All repositories are up to date. The following 1 package(s) will be affected (of 0 checked): New packages to be INSTALLED: py311-speedtest-cli: 2.1.3 [pfSense] Number of packages to be installed: 1 52 KiB to be downloaded. Proceed with this action? [y/N]: y [1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100% 52 KiB 53.1kB/s 00:01 Checking integrity... done (0 conflicting) [1/1] Installing py311-speedtest-cli-2.1.3... [1/1] Extracting py311-speedtest-cli-2.1.3: 100% [25.03-RC][admin@5100.stevew.lan]/root: rehashAs I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.
Sadly, it's completely unreliable running on the FW. Getting 450mbit up and 45 down on the 6100, since it's picking a completely different server than even my other FreeBSD box.
-
Tried one more test - Put a QNAP Multigig switch in between the PFSense box and the RG.
The 6100 maintained its speeds.
The HP Box also maintained its lower upload speeds.
Went into the Z2G9's BIOS (it's running the latest) and disabled PCI Express Power Management.
Upload speeds remained at 450-700Mbits, their typical range. Going back to the 6100 yielded the usual 3800-4500 up and down.
Almost at my wit's end. This is a very standard build using HP workstation components and Intel NICs. No idea why I'm seeing this.
Any ideas?
-Bear
-
You can specify which server it uses:
--list Display a list of speedtest.net servers sorted by distance --server SERVER Specify a server ID to test against. Can be supplied multiple times -
✓ Test Server: [68863] 0.00km San Antonio, TX by AT&T ✓ Latency: 9.472954ms Jitter: 1.390244ms Min: 8.694556ms Max: 13.568589ms ✓ Packet Loss Analyzer: Running in background (<= 30 Secs) ✓ Download: 1350.20 Mbps (Used: 1708.20MB) (Latency: 15ms Jitter: 10ms Min: 8ms Max: 44ms) ✓ Upload: 763.02 Mbps (Used: 930.43MB) (Latency: 11ms Jitter: 2ms Min: 8ms Max: 13ms) ✓ Packet Loss: N/AI'm getting really slow speeds from the 6100 vs my Mac or Windows system with 10GBE to the point where this up speed is aligned with the ~750mbit I get on those systems with the HP/X520.
In other words, the slow performance test locally on the 6100 would mask the issue as I get 3500+mbits up when running speedtest from a local system vs 760 from the 6100 itself.
-
Hmm, OK. Probably not useful data there then.
Do you see the same in an iperf test?
-
@stephenw10 Haven't run iPerf yet. I'll try that later, but I'm not sure what data we're after here; the performance is identical for low upload speeds using 2 different Intel NICs in the same x16 slot @ x8, and both WAN and LAN are on the same NIC.
Flow control is off, PCI power management is off, and with an intermediary multigig switch, results remain identical.
-
If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.
Also you can run iperf directly from pfSense with more useful results.
-
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.
Also you can run iperf directly from pfSense with more useful results.
IPerf isn't jibing with speedtest. Here's my recent speedtest result: https://www.speedtest.net/my-result/d/739237147
Here's a few iPERF 3 runs to supposedly local addresses:
Connecting to host 37.19.206.20, port 5201 [ 5] local 75.31.24.1 port 38531 connected to 37.19.206.20 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.2 MBytes 237 Mbits/sec 0 6.41 MBytes [ 5] 1.00-2.00 sec 46.9 MBytes 393 Mbits/sec 0 6.41 MBytes [ 5] 2.00-3.01 sec 47.6 MBytes 397 Mbits/sec 0 6.41 MBytes [ 5] 3.01-4.00 sec 46.9 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 4.00-5.00 sec 47.2 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 5.00-6.00 sec 47.2 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 6.00-7.03 sec 48.2 MBytes 395 Mbits/sec 0 6.41 MBytes [ 5] 7.03-8.00 sec 46.0 MBytes 396 Mbits/sec 0 6.41 MBytes [ 5] 8.00-9.00 sec 47.5 MBytes 398 Mbits/sec 0 6.41 MBytes [ 5] 9.00-10.00 sec 47.4 MBytes 397 Mbits/sec 0 6.41 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 453 MBytes 380 Mbits/sec 0 sender [ 5] 0.00-10.05 sec 453 MBytes 379 Mbits/sec receiverConnecting to host dal.speedtest.clouvider.net, port 5200 [ 5] local 75.31.24.1 port 48312 connected to 2.56.188.136 port 5200 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.01 sec 5.62 MBytes 46.9 Mbits/sec 8 92.8 KBytes [ 5] 1.01-2.00 sec 11.2 MBytes 94.8 Mbits/sec 0 162 KBytes [ 5] 2.00-3.00 sec 17.8 MBytes 149 Mbits/sec 0 231 KBytes [ 5] 3.00-4.02 sec 20.9 MBytes 172 Mbits/sec 3 220 KBytes [ 5] 4.02-5.00 sec 19.1 MBytes 163 Mbits/sec 4 179 KBytes [ 5] 5.00-6.01 sec 19.2 MBytes 160 Mbits/sec 0 248 KBytes [ 5] 6.01-7.00 sec 24.8 MBytes 210 Mbits/sec 0 316 KBytes [ 5] 7.00-8.01 sec 22.2 MBytes 186 Mbits/sec 81 114 KBytes [ 5] 8.01-9.01 sec 10.9 MBytes 91.1 Mbits/sec 46 34.7 KBytes [ 5] 9.01-10.00 sec 6.12 MBytes 51.5 Mbits/sec 0 104 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 158 MBytes 132 Mbits/sec 142 sender [ 5] 0.00-10.01 sec 157 MBytes 132 Mbits/sec receiverI can consistently get similar speeds to other providers as well. iperf3 isn't getting me close to a gigabit. Real world transfers are in the 2.5-3.5Gbit range usually.
-
Now, if I do the iperf3 from my Mac Studio with 10GBE going /through/ the 6100 Max...
Connecting to host speedtest.dal13.us.leaseweb.net, port 5201 [ 7] local 75.31.24.8 port 61252 connected to 209.58.159.68 port 5201 [ ID] Interval Transfer Bitrate [ 7] 0.00-1.01 sec 186 MBytes 1.55 Gbits/sec [ 7] 1.01-2.00 sec 247 MBytes 2.08 Gbits/sec [ 7] 2.00-3.01 sec 249 MBytes 2.08 Gbits/sec [ 7] 3.01-4.01 sec 247 MBytes 2.07 Gbits/sec [ 7] 4.01-5.01 sec 248 MBytes 2.08 Gbits/sec [ 7] 5.01-6.01 sec 247 MBytes 2.07 Gbits/sec [ 7] 6.01-7.01 sec 248 MBytes 2.08 Gbits/sec [ 7] 7.01-8.01 sec 220 MBytes 1.84 Gbits/sec [ 7] 8.01-9.00 sec 207 MBytes 1.74 Gbits/sec [ 7] 9.00-10.01 sec 216 MBytes 1.81 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 7] 0.00-10.01 sec 2.26 GBytes 1.94 Gbits/sec sender [ 7] 0.00-10.02 sec 2.26 GBytes 1.94 Gbits/sec receiverAnd do the same iperf3 test from the 6100 Max itself, just a moment later to the same iperf3 server...
Connecting to host speedtest.dal13.us.leaseweb.net, port 5201 [ 5] local 75.31.24.1 port 8528 connected to 209.58.159.68 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 11.4 MBytes 95.3 Mbits/sec 60 145 KBytes [ 5] 1.00-2.00 sec 6.88 MBytes 57.7 Mbits/sec 37 79.5 KBytes [ 5] 2.00-3.00 sec 11.9 MBytes 99.6 Mbits/sec 0 156 KBytes [ 5] 3.00-4.01 sec 10.5 MBytes 87.6 Mbits/sec 68 53.7 KBytes [ 5] 4.01-5.02 sec 7.50 MBytes 62.3 Mbits/sec 19 108 KBytes [ 5] 5.02-6.02 sec 10.6 MBytes 88.9 Mbits/sec 19 42.7 KBytes [ 5] 6.02-7.00 sec 7.88 MBytes 67.2 Mbits/sec 0 118 KBytes [ 5] 7.00-8.00 sec 15.6 MBytes 131 Mbits/sec 0 194 KBytes [ 5] 8.00-9.01 sec 23.8 MBytes 197 Mbits/sec 0 271 KBytes [ 5] 9.01-10.02 sec 31.8 MBytes 263 Mbits/sec 0 348 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.02 sec 138 MBytes 115 Mbits/sec 203 sender [ 5] 0.00-10.03 sec 137 MBytes 115 Mbits/sec receiver -
Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.
Can you test using the i5?
Was that with multiple parallel streams?
The default direction for iperf3 is from the client to the server, so upload. Try using
-Rto get a download result. -
@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:
Hmm, interesting. Yes, as I said, running iperf from pfSense itself is always going to give a worse result than thorough it just because of the processing required to run the test itself. That's particularly true of the 6100 where each CPU core is not that powerful.
Can you test using the i5?
Was that with multiple parallel streams?
The default direction for iperf3 is from the client to the server, so upload. Try using
-Rto get a download result.6100 Max With -R and -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.28 sec 3.21 GBytes 2.68 Gbits/sec 9325 sender [SUM] 0.00-10.00 sec 2.88 GBytes 2.47 Gbits/sec receiverMac Studio behind 6100 Max with -R and -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 -R [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 5.26 GBytes 4.52 Gbits/sec 11595 sender [SUM] 0.00-10.00 sec 5.12 GBytes 4.40 Gbits/sec receiver6100 Max with just -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.07 sec 3.10 GBytes 2.65 Gbits/sec 12761 sender [SUM] 0.00-10.09 sec 3.04 GBytes 2.59 Gbits/sec receiverMac Studio behind 6100 Max with just -P 120
# iperf3 -c speedtest.dal13.us.leaseweb.net -p 5201-5210 -P 120 [ ID] Interval Transfer Bitrate Retr [SUM] 0.00-10.01 sec 2.06 GBytes 1.77 Gbits/sec sender [SUM] 0.00-10.02 sec 2.05 GBytes 1.76 Gbits/sec receiverNot quite my SpeedTest Results, but enough that I can try the same on the i5 later tonight and append this thread with some of those numbers.
-
Probably hitting the CPU core limit on the 6100 there. But it's at least the same both ways and more than the 400Mps limit you hit previously. So try the i5 there and see if it still hits it.
-
I'll run the same tests tonight - I need to take my network offline to jump to the i5-based system, so it'll need to be after hours this evening, but I'll get some data here soon. Thanks for the help. :)
-Bear
