Slow upload speeds on HP Z2 G9 PFSense Box

Bear

Here's an ifconfig from the Netgate 6100 using the same Intel transceiver:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 90:ec:77:21:2c:9f
inet6 fe80::92ec:77ff:fe21:2c9f%ix1 prefixlen 64 scopeid 0x6
media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
status: active

Here's what I get when it's plugged into the X520:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 48:df:37:3f:28:f5
inet6 fe80::4adf:37ff:fe3f:28f5%ix1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The BG320 RG just shows a connection at 5Gbit. There aren't any errors or collisions reported. They both just happen to be using ix1 for WAN on both units, coincidentally.

It looks like the X520 is enabling flow control. I'm not sure the RG has flow control though. Wouldn't they both need to have it enabled for it to do anything? Would it be worth adding hw.ix.flow_control with a value of 0 to System Tunables?

Appreciate the help!

Bear

Just tried adding the system tunable, however when plugged in, was still showing the rxpause,txpause even after a reboot. Not sure where to go from here.

Also, traffic shaping is off.

EDIT:
Added hw.ix.flow_control=0 to /boot/loader.conf.local

The interface is coming up without rxpause,txpause now.

However, I am still seeing the same 450Mbit upload speeds, whereas I'm getting 3600-4500 on the 6100. Download speeds are pretty identical.
Thoughts?

stephenw10

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Hmm, I would usually expect the X520 to behave better....

If you check the Status > Interfaces page do you see errors or collisions?

If so you can dig further into the mac stats in the sysctl output for each NIC.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Yes, the WAN is linked at 5Gbit, whether it's plugged into the 6100 or the X520 in the HP, which is in an x8 slot.

Hmm, I would usually expect the X520 to behave better....

Me too. I had this result with the X710-DA2, so I figured I'd go with the tried and true X520. No dice.

If you check the Status > Interfaces page do you see errors or collisions?

There are no errors or collisions on the WAN side. Sadly, I can't keep the new firewall on long enough to see if there are any other things going on, longer-term. I've got a lot of traffic on this business line that I'm hosting and cutting me down to 100-400Mbits has some dire consequences.

If so you can dig further into the mac stats in the sysctl output for each NIC.

What specifically would you like me to look at next time I connect it up?

stephenw10

Are there errors on the LAN side? It could be a problem there.

Bear

@stephenw10 Not seeing any errors on the LAN side at all. Interrupts aren't too bad either.

I'm running a filtered bridge. Same config as on the 6100.

Getting the same results as when I used the 710 as well, completely different NIC.

stephenw10

How are you testing? Do you see any packet loss?

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

How are you testing? Do you see any packet loss?

Just using speedtest and fast.com from 10GBE Macs and Linux boxes with Intel NICs - Seeing zero packet loss and 8-10ms ping times.

stephenw10

Hmm, you might try running a test from pfSense directly. It won't be accurate but if you see the same big difference between up and downloads that at least narrows it down to the WAN side.

Can you see anything about the link state from the connected modem.

Bear

@stephenw10

Both WAN and LAN are on the same X520 NIC. Is there a native speedtest on pfsense?

stephenw10

Yes there is a CLI client. In fact there are two:

[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest
py311-speedtest-cli-2.1.3      Command line interface for testing internet bandwidth
speedtest-go-1.7.9             CLI and Go API to test internet speed using speedtest.net

You can install and run those directly:

[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py311-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

52 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100%   52 KiB  53.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py311-speedtest-cli-2.1.3...
[1/1] Extracting py311-speedtest-cli-2.1.3: 100%
[25.03-RC][admin@5100.stevew.lan]/root: rehash

As I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.

Bear

I can try that later today. I’ll post results soon.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Yes there is a CLI client. In fact there are two:

[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest
py311-speedtest-cli-2.1.3      Command line interface for testing internet bandwidth
speedtest-go-1.7.9             CLI and Go API to test internet speed using speedtest.net

You can install and run those directly:

[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py311-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

52 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100%   52 KiB  53.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py311-speedtest-cli-2.1.3...
[1/1] Extracting py311-speedtest-cli-2.1.3: 100%
[25.03-RC][admin@5100.stevew.lan]/root: rehash

As I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.

Sadly, it's completely unreliable running on the FW. Getting 450mbit up and 45 down on the 6100, since it's picking a completely different server than even my other FreeBSD box.

Bear

Tried one more test - Put a QNAP Multigig switch in between the PFSense box and the RG.

The 6100 maintained its speeds.

The HP Box also maintained its lower upload speeds.

Went into the Z2G9's BIOS (it's running the latest) and disabled PCI Express Power Management.

Upload speeds remained at 450-700Mbits, their typical range. Going back to the 6100 yielded the usual 3800-4500 up and down.

Almost at my wit's end. This is a very standard build using HP workstation components and Intel NICs. No idea why I'm seeing this.

Any ideas?

-Bear

stephenw10

You can specify which server it uses:

  --list                Display a list of speedtest.net servers sorted by distance
  --server SERVER       Specify a server ID to test against. Can be supplied multiple times

Bear

@stephenw10

✓ Test Server: [68863] 0.00km San Antonio, TX by AT&T
✓ Latency: 9.472954ms Jitter: 1.390244ms Min: 8.694556ms Max: 13.568589ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 1350.20 Mbps (Used: 1708.20MB) (Latency: 15ms Jitter: 10ms Min: 8ms Max: 44ms)
✓ Upload: 763.02 Mbps (Used: 930.43MB) (Latency: 11ms Jitter: 2ms Min: 8ms Max: 13ms)
✓ Packet Loss: N/A

I'm getting really slow speeds from the 6100 vs my Mac or Windows system with 10GBE to the point where this up speed is aligned with the ~750mbit I get on those systems with the HP/X520.

In other words, the slow performance test locally on the 6100 would mask the issue as I get 3500+mbits up when running speedtest from a local system vs 760 from the 6100 itself.

stephenw10

Hmm, OK. Probably not useful data there then.

Do you see the same in an iperf test?

Bear

@stephenw10 Haven't run iPerf yet. I'll try that later, but I'm not sure what data we're after here; the performance is identical for low upload speeds using 2 different Intel NICs in the same x16 slot @ x8, and both WAN and LAN are on the same NIC.

Flow control is off, PCI power management is off, and with an intermediary multigig switch, results remain identical.

stephenw10

If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.

Also you can run iperf directly from pfSense with more useful results.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

If you're able to see the same issue against a remote iperf server you can then test against a local iperf server to check if it's something specific to the WAN connection.

Also you can run iperf directly from pfSense with more useful results.

IPerf isn't jibing with speedtest. Here's my recent speedtest result: https://www.speedtest.net/my-result/d/739237147

Here's a few iPERF 3 runs to supposedly local addresses:

Connecting to host 37.19.206.20, port 5201
[  5] local 75.31.24.1 port 38531 connected to 37.19.206.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  28.2 MBytes   237 Mbits/sec    0   6.41 MBytes
[  5]   1.00-2.00   sec  46.9 MBytes   393 Mbits/sec    0   6.41 MBytes
[  5]   2.00-3.01   sec  47.6 MBytes   397 Mbits/sec    0   6.41 MBytes
[  5]   3.01-4.00   sec  46.9 MBytes   396 Mbits/sec    0   6.41 MBytes
[  5]   4.00-5.00   sec  47.2 MBytes   396 Mbits/sec    0   6.41 MBytes
[  5]   5.00-6.00   sec  47.2 MBytes   396 Mbits/sec    0   6.41 MBytes
[  5]   6.00-7.03   sec  48.2 MBytes   395 Mbits/sec    0   6.41 MBytes
[  5]   7.03-8.00   sec  46.0 MBytes   396 Mbits/sec    0   6.41 MBytes
[  5]   8.00-9.00   sec  47.5 MBytes   398 Mbits/sec    0   6.41 MBytes
[  5]   9.00-10.00  sec  47.4 MBytes   397 Mbits/sec    0   6.41 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   453 MBytes   380 Mbits/sec    0             sender
[  5]   0.00-10.05  sec   453 MBytes   379 Mbits/sec                  receiver

Connecting to host dal.speedtest.clouvider.net, port 5200
[  5] local 75.31.24.1 port 48312 connected to 2.56.188.136 port 5200
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  5.62 MBytes  46.9 Mbits/sec    8   92.8 KBytes
[  5]   1.01-2.00   sec  11.2 MBytes  94.8 Mbits/sec    0    162 KBytes
[  5]   2.00-3.00   sec  17.8 MBytes   149 Mbits/sec    0    231 KBytes
[  5]   3.00-4.02   sec  20.9 MBytes   172 Mbits/sec    3    220 KBytes
[  5]   4.02-5.00   sec  19.1 MBytes   163 Mbits/sec    4    179 KBytes
[  5]   5.00-6.01   sec  19.2 MBytes   160 Mbits/sec    0    248 KBytes
[  5]   6.01-7.00   sec  24.8 MBytes   210 Mbits/sec    0    316 KBytes
[  5]   7.00-8.01   sec  22.2 MBytes   186 Mbits/sec   81    114 KBytes
[  5]   8.01-9.01   sec  10.9 MBytes  91.1 Mbits/sec   46   34.7 KBytes
[  5]   9.01-10.00  sec  6.12 MBytes  51.5 Mbits/sec    0    104 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   158 MBytes   132 Mbits/sec  142             sender
[  5]   0.00-10.01  sec   157 MBytes   132 Mbits/sec                  receiver

I can consistently get similar speeds to other providers as well. iperf3 isn't getting me close to a gigabit. Real world transfers are in the 2.5-3.5Gbit range usually.