Slow upload speeds on HP Z2 G9 PFSense Box

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. What does it actually show as linked at in ifconfig?

One difference between those two setups is that the X553 NICs in the C3K chipset in the 6100 do not have the data lines to read the link type from a module. The X520 discrete NIC almost certainly does.

It shows at 10GBE on both, though does show a bit more info about the SFP transceiver on the X520. Again, I'm getting 4600mbits down on both. Just the up speed is bad on the HP with 2 different Intel NICs.

stephenw10

Sure feels like a flow control issue. If it shows as linked at 10G but is actually linked at 5G I wouldn't have much confidence in what it reports for flow control. Can you see the link status at the other side of the link?

Bear

@stephenw10

Here's an ifconfig from the Netgate 6100 using the same Intel transceiver:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 90:ec:77:21:2c:9f
inet6 fe80::92ec:77ff:fe21:2c9f%ix1 prefixlen 64 scopeid 0x6
media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
status: active

Here's what I get when it's plugged into the X520:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 48:df:37:3f:28:f5
inet6 fe80::4adf:37ff:fe3f:28f5%ix1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The BG320 RG just shows a connection at 5Gbit. There aren't any errors or collisions reported. They both just happen to be using ix1 for WAN on both units, coincidentally.

It looks like the X520 is enabling flow control. I'm not sure the RG has flow control though. Wouldn't they both need to have it enabled for it to do anything? Would it be worth adding hw.ix.flow_control with a value of 0 to System Tunables?

Appreciate the help!

Bear

Just tried adding the system tunable, however when plugged in, was still showing the rxpause,txpause even after a reboot. Not sure where to go from here.

Also, traffic shaping is off.

EDIT:
Added hw.ix.flow_control=0 to /boot/loader.conf.local

The interface is coming up without rxpause,txpause now.

However, I am still seeing the same 450Mbit upload speeds, whereas I'm getting 3600-4500 on the 6100. Download speeds are pretty identical.
Thoughts?

stephenw10

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Hmm, I would usually expect the X520 to behave better....

If you check the Status > Interfaces page do you see errors or collisions?

If so you can dig further into the mac stats in the sysctl output for each NIC.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Yes, the WAN is linked at 5Gbit, whether it's plugged into the 6100 or the X520 in the HP, which is in an x8 slot.

Hmm, I would usually expect the X520 to behave better....

Me too. I had this result with the X710-DA2, so I figured I'd go with the tried and true X520. No dice.

If you check the Status > Interfaces page do you see errors or collisions?

There are no errors or collisions on the WAN side. Sadly, I can't keep the new firewall on long enough to see if there are any other things going on, longer-term. I've got a lot of traffic on this business line that I'm hosting and cutting me down to 100-400Mbits has some dire consequences.

If so you can dig further into the mac stats in the sysctl output for each NIC.

What specifically would you like me to look at next time I connect it up?

stephenw10

Are there errors on the LAN side? It could be a problem there.

Bear

@stephenw10 Not seeing any errors on the LAN side at all. Interrupts aren't too bad either.

I'm running a filtered bridge. Same config as on the 6100.

Getting the same results as when I used the 710 as well, completely different NIC.

stephenw10

How are you testing? Do you see any packet loss?

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

How are you testing? Do you see any packet loss?

Just using speedtest and fast.com from 10GBE Macs and Linux boxes with Intel NICs - Seeing zero packet loss and 8-10ms ping times.

stephenw10

Hmm, you might try running a test from pfSense directly. It won't be accurate but if you see the same big difference between up and downloads that at least narrows it down to the WAN side.

Can you see anything about the link state from the connected modem.

Bear

@stephenw10

Both WAN and LAN are on the same X520 NIC. Is there a native speedtest on pfsense?

stephenw10

Yes there is a CLI client. In fact there are two:

[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest
py311-speedtest-cli-2.1.3      Command line interface for testing internet bandwidth
speedtest-go-1.7.9             CLI and Go API to test internet speed using speedtest.net

You can install and run those directly:

[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py311-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

52 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100%   52 KiB  53.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py311-speedtest-cli-2.1.3...
[1/1] Extracting py311-speedtest-cli-2.1.3: 100%
[25.03-RC][admin@5100.stevew.lan]/root: rehash

As I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.

Bear

I can try that later today. I’ll post results soon.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Yes there is a CLI client. In fact there are two:

[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest
py311-speedtest-cli-2.1.3      Command line interface for testing internet bandwidth
speedtest-go-1.7.9             CLI and Go API to test internet speed using speedtest.net

You can install and run those directly:

[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py311-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

52 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100%   52 KiB  53.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py311-speedtest-cli-2.1.3...
[1/1] Extracting py311-speedtest-cli-2.1.3: 100%
[25.03-RC][admin@5100.stevew.lan]/root: rehash

As I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.

Sadly, it's completely unreliable running on the FW. Getting 450mbit up and 45 down on the 6100, since it's picking a completely different server than even my other FreeBSD box.

Bear

Tried one more test - Put a QNAP Multigig switch in between the PFSense box and the RG.

The 6100 maintained its speeds.

The HP Box also maintained its lower upload speeds.

Went into the Z2G9's BIOS (it's running the latest) and disabled PCI Express Power Management.

Upload speeds remained at 450-700Mbits, their typical range. Going back to the 6100 yielded the usual 3800-4500 up and down.

Almost at my wit's end. This is a very standard build using HP workstation components and Intel NICs. No idea why I'm seeing this.

Any ideas?

-Bear

stephenw10

You can specify which server it uses:

  --list                Display a list of speedtest.net servers sorted by distance
  --server SERVER       Specify a server ID to test against. Can be supplied multiple times

Bear

@stephenw10

✓ Test Server: [68863] 0.00km San Antonio, TX by AT&T
✓ Latency: 9.472954ms Jitter: 1.390244ms Min: 8.694556ms Max: 13.568589ms
✓ Packet Loss Analyzer: Running in background (<= 30 Secs)
✓ Download: 1350.20 Mbps (Used: 1708.20MB) (Latency: 15ms Jitter: 10ms Min: 8ms Max: 44ms)
✓ Upload: 763.02 Mbps (Used: 930.43MB) (Latency: 11ms Jitter: 2ms Min: 8ms Max: 13ms)
✓ Packet Loss: N/A

I'm getting really slow speeds from the 6100 vs my Mac or Windows system with 10GBE to the point where this up speed is aligned with the ~750mbit I get on those systems with the HP/X520.

In other words, the slow performance test locally on the 6100 would mask the issue as I get 3500+mbits up when running speedtest from a local system vs 760 from the 6100 itself.

stephenw10

Hmm, OK. Probably not useful data there then.

Do you see the same in an iperf test?

Bear

@stephenw10 Haven't run iPerf yet. I'll try that later, but I'm not sure what data we're after here; the performance is identical for low upload speeds using 2 different Intel NICs in the same x16 slot @ x8, and both WAN and LAN are on the same NIC.

Flow control is off, PCI power management is off, and with an intermediary multigig switch, results remain identical.