Slow upload speeds on HP Z2 G9 PFSense Box

Bear

I've got a Netgate 6100 presently and everything is working decently with PFSense +.

I'm moving to a HP Z2 G9 workstation for additional overhead for running some add-ons as well as my ISP moving to 10GBE, needing a bit more power.

The HP is running the same version of PFSense as the 6100. The NIC for the WAN/LAN is an Intel X520 with a FS Intel transceiver to go from NBaseT to SFP+. Same as what I've been using in the Netgate.

However, my upload speeds seem limited to about 400Mbit while I hit 4200 on the Netgate. I've also tried this with an Intel X720 NIC and have seen the same issue with upload speeds. Everything is now using the ix driver, it shows the correct info on the SFP port, and download speeds are fine. It's only upload speeds that are suffering.

Anyone have any suggestions? I've tried turning off the hardware acceleration options, and that's done nothing. CPU is an i5-14500 and the system has 32GB of ECC DDR5 RAM. It has one other NIC, an Intel X550 in it that's bridged. The config was restored from the Netgate to the HP Box with interface mappings reassigned.

stephenw10

What is the NIC actually connected to?

Do you see flow-control active in either set up?

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

What is the NIC actually connected to?

An ATT BGW320 - Its WAN connection is an NbaseT @ 5Gbit

Do you see flow-control active in either set up?

No. Neither in the Netgate nor the HP Z2 G9's setups.

stephenw10

What's on the LAN side? A switch? The same one?

Is that the X550 on the WAN? The X520 NIC cannot link at base5 as far as I know.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

What's on the LAN side? A switch? The same one?

A Netgear 12 port Multigig. Second port from the X520 is connected to an SFP+ on it.

Is that the X550 on the WAN? The X520 NIC cannot link at base5 as far as I know.

The X520 is on the WAN - You are correct on speed support, however I'm using an FS Intel NBaseT to 10GBE transceiver module linked here - https://www.fs.com/products/178041.html - Again, works fine on the 6100.

stephenw10

Hmm, interesting. What does it actually show as linked at in ifconfig?

One difference between those two setups is that the X553 NICs in the C3K chipset in the 6100 do not have the data lines to read the link type from a module. The X520 discrete NIC almost certainly does.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. What does it actually show as linked at in ifconfig?

One difference between those two setups is that the X553 NICs in the C3K chipset in the 6100 do not have the data lines to read the link type from a module. The X520 discrete NIC almost certainly does.

It shows at 10GBE on both, though does show a bit more info about the SFP transceiver on the X520. Again, I'm getting 4600mbits down on both. Just the up speed is bad on the HP with 2 different Intel NICs.

stephenw10

Sure feels like a flow control issue. If it shows as linked at 10G but is actually linked at 5G I wouldn't have much confidence in what it reports for flow control. Can you see the link status at the other side of the link?

Bear

@stephenw10

Here's an ifconfig from the Netgate 6100 using the same Intel transceiver:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 90:ec:77:21:2c:9f
inet6 fe80::92ec:77ff:fe21:2c9f%ix1 prefixlen 64 scopeid 0x6
media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
status: active

Here's what I get when it's plugged into the X520:

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 48:df:37:3f:28:f5
inet6 fe80::4adf:37ff:fe3f:28f5%ix1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The BG320 RG just shows a connection at 5Gbit. There aren't any errors or collisions reported. They both just happen to be using ix1 for WAN on both units, coincidentally.

It looks like the X520 is enabling flow control. I'm not sure the RG has flow control though. Wouldn't they both need to have it enabled for it to do anything? Would it be worth adding hw.ix.flow_control with a value of 0 to System Tunables?

Appreciate the help!

Bear

Just tried adding the system tunable, however when plugged in, was still showing the rxpause,txpause even after a reboot. Not sure where to go from here.

Also, traffic shaping is off.

EDIT:
Added hw.ix.flow_control=0 to /boot/loader.conf.local

The interface is coming up without rxpause,txpause now.

However, I am still seeing the same 450Mbit upload speeds, whereas I'm getting 3600-4500 on the 6100. Download speeds are pretty identical.
Thoughts?

stephenw10

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Hmm, I would usually expect the X520 to behave better....

If you check the Status > Interfaces page do you see errors or collisions?

If so you can dig further into the mac stats in the sysctl output for each NIC.

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

Hmm, interesting. So in both cases that's the WAN? And it's actually linked at 5G?

Yes, the WAN is linked at 5Gbit, whether it's plugged into the 6100 or the X520 in the HP, which is in an x8 slot.

Hmm, I would usually expect the X520 to behave better....

Me too. I had this result with the X710-DA2, so I figured I'd go with the tried and true X520. No dice.

If you check the Status > Interfaces page do you see errors or collisions?

There are no errors or collisions on the WAN side. Sadly, I can't keep the new firewall on long enough to see if there are any other things going on, longer-term. I've got a lot of traffic on this business line that I'm hosting and cutting me down to 100-400Mbits has some dire consequences.

If so you can dig further into the mac stats in the sysctl output for each NIC.

What specifically would you like me to look at next time I connect it up?

stephenw10

Are there errors on the LAN side? It could be a problem there.

Bear

@stephenw10 Not seeing any errors on the LAN side at all. Interrupts aren't too bad either.

I'm running a filtered bridge. Same config as on the 6100.

Getting the same results as when I used the 710 as well, completely different NIC.

stephenw10

How are you testing? Do you see any packet loss?

Bear

@stephenw10 said in Slow upload speeds on HP Z2 G9 PFSense Box:

How are you testing? Do you see any packet loss?

Just using speedtest and fast.com from 10GBE Macs and Linux boxes with Intel NICs - Seeing zero packet loss and 8-10ms ping times.

stephenw10

Hmm, you might try running a test from pfSense directly. It won't be accurate but if you see the same big difference between up and downloads that at least narrows it down to the WAN side.

Can you see anything about the link state from the connected modem.

Bear

@stephenw10

Both WAN and LAN are on the same X520 NIC. Is there a native speedtest on pfsense?

stephenw10

Yes there is a CLI client. In fact there are two:

[25.03-RC][admin@5100.stevew.lan]/root: pkg search speedtest
py311-speedtest-cli-2.1.3      Command line interface for testing internet bandwidth
speedtest-go-1.7.9             CLI and Go API to test internet speed using speedtest.net

You can install and run those directly:

[25.03-RC][admin@5100.stevew.lan]/root: pkg install py311-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py311-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

52 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py311-speedtest-cli-2.1.3.pkg: 100%   52 KiB  53.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py311-speedtest-cli-2.1.3...
[1/1] Extracting py311-speedtest-cli-2.1.3: 100%
[25.03-RC][admin@5100.stevew.lan]/root: rehash

As I said it will not give an accurate value at bandwidths that high but you can use it to compare relative rates.

Bear

I can try that later today. I’ll post results soon.