82576-2T-X1 Speed issues

docklost

Hey Everyone

I recently purchased a 82576-2T-X1 off Amazon to use in my pfsense box. I am only getting 250mbps down when I should be getting close to a gigabit. I found that there were suggestions to Disable hardware TCP segmentation offload and Disable hardware large receive offload which were done by default. I also checked the mbuf but it was at the recommended size. The computer is a quad core intel cpu a 3.0ghz, plenty of ram and HD space. When I plug up directly up to the modem I get gigabit all day long.

Is there anything else I can check or change? I’m not sure what to do at this point.

stephenw10

You are using the ports on the card as WAN and LAN?

Check for errors in Status > Interfaces. Make sure both show linked at 1G.

Steve

RobH 0

@docklost I could not find this device on Amazon, can you provide a link?

stephenw10

@docklost said in 82576-2T-X1 Speed issues:

82576-2T-X1

It's a NIC using the 82576 chip with 2 base-T ports and a 1X PCIe interface. Pretty standard and should be well supported. Provided it's not fake of course.

Steve

docklost

@stephenw10 said in 82576-2T-X1 Speed issues:

You are using the ports on the card as WAN and LAN?

Check for errors in Status > Interfaces. Make sure both show linked at 1G.

Steve

Yes igb0 is wan igb1 is lab

For both I am seeing 1000base full duplex for the media. I am not seeing anything where it shows linked at 1G

docklost

@robh-0 said in 82576-2T-X1 Speed issues:

@docklost I could not find this device on Amazon, can you provide a link?

https://www.amazon.com/dp/B01LXTF48X?ref_=cm_sw_r_cp_ud_dp_VFH75B2A6834PNJDXRSH

stephenw10

Actually from 10GTek so it's probably genuine at least.

OK try some tests to/from pfSense directly so you're only using one NIC at a time.

You can install and run the speedtest-cli client in pfSense to test the WAN. Though results are always lower than testing through the box when it's working correctly.
You can use iperf3 to test the LAN side. There is a package for iperf.

[2.6.0-RELEASE][admin@m4600.stevew.lan]/root: pkg install py38-speedtest-cli
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	py38-speedtest-cli: 2.1.3 [pfSense]

Number of packages to be installed: 1

38 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching py38-speedtest-cli-2.1.3.pkg: 100%   38 KiB  39.0kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing py38-speedtest-cli-2.1.3...
[1/1] Extracting py38-speedtest-cli-2.1.3: 100%
[2.6.0-RELEASE][admin@m4600.stevew.lan]/root: rehash
[2.6.0-RELEASE][admin@m4600.stevew.lan]/root: speedtest
Retrieving speedtest.net configuration...
Testing from Plusnet (1.2.3.4)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Vorboss Limited (London) [10.00 km]: 13.237 ms
Testing download speed................................................................................
Download: 61.83 Mbit/s
Testing upload speed......................................................................................................
Upload: 13.49 Mbit/s

Steve

docklost

Below are my speedtest results.

Retrieving speedtest.net configuration...
Testing from Spectrum (1.2.3.4)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Comporium (Rock Hill, SC) [118.58 km]: 61.058 ms
Testing download speed................................................................................
Download: 268.52 Mbit/s
Testing upload speed......................................................................................................
Upload: 37.41 Mbit/s

LAN is fine internally.
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 92.5 MBytes 776 Mbits/sec
[ 5] 1.00-2.00 sec 94.8 MBytes 796 Mbits/sec
[ 5] 2.00-3.00 sec 94.5 MBytes 792 Mbits/sec
[ 5] 3.00-4.00 sec 94.0 MBytes 788 Mbits/sec
[ 5] 4.00-5.00 sec 90.0 MBytes 755 Mbits/sec
[ 5] 5.00-6.00 sec 93.8 MBytes 787 Mbits/sec
[ 5] 6.00-7.00 sec 90.7 MBytes 761 Mbits/sec
[ 5] 7.00-8.00 sec 94.4 MBytes 792 Mbits/sec
[ 5] 8.00-9.00 sec 94.7 MBytes 795 Mbits/sec
[ 5] 9.00-10.00 sec 94.5 MBytes 792 Mbits/sec

[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 934 MBytes 783 Mbits/sec sender
[ 5] 0.00-10.00 sec 934 MBytes 783 Mbits/sec receiver

stephenw10

Ok, so you see about the same result in speedtest on pfSense as you do from a LAN side client.

So the first thing I would do is swap the NIC assignments and see if the low speed moves to the LAN.

Steve

docklost

@stephenw10

Nearly the same results just flipped. The only thing I haven’t done yet is swap out cables. I’m going to try that at lunch.

stephenw10

Still slow WAN side?

Ok, so something in the interface/port config there. Check the output of ifconfig vvvm igb1. Assuming WAN is now igb1.

Steve

docklost

@stephenw10

ipconfig vvvm igb1 returned an error. (ifconfig: interface vvvm does not exist)
I assume you meant ipconfig -vvvm igb1?

igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: WAN
	options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
	capabilities=f53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,NETMAP,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 80:61:5f:08:2d:75
	inet6 ffff %igb1 prefixlen 64 scopeid 0x2
	inet 1.2.3.4 netmask 0xfffff000 broadcast 255.255.255.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	supported media:
		media autoselect
		media 1000baseT
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

stephenw10

Indeed I omitted the -.

Nothing significant there.

The next thing I would check for is flow control:

[2.6.0-RELEASE][admin@t70.stevew.lan]/root: sysctl dev.igb.1.fc
dev.igb.1.fc: 3

It's probably enabled like it is there. I would try disabling it:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#flow-control

Steve

docklost

@stephenw10

I changed it to zero. And ran another speed test and it increased it about 50Mbits/s to 330Mbit/s

stephenw10

Hmm, well any effect there is interesting.

Try running top -HaSP at the cli whilst you're testing. Make sure you're not hitting a limit on any CPU core.

docklost

@stephenw10

So I tested several times last night and only one core would spike to 15%. Generally it was below 10% usage.

stephenw10

And it was the expected interface loading?

Ideally I would hope to see at least two cores with some loading. If you tested with speedtest that uses multiple connections so it should use multiple queues. But if nothing was at 100% there should be headroom for more.

The next test I would try to do is iperf through the firewall. So that could be to a public iperf server of by using your own iperf server on the WAN side.

Steve

docklost

@stephenw10

So I did one on the firewall and one from my Mac

Firewall

iperf3 -c iperf3.velocityonline.net -p 5210
Connecting to host iperf3.velocityonline.net, port 5210
[  5] local 1.2.3.4 port 15533 connected to 216.162.135.203 port 5210
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.16 MBytes  43.3 Mbits/sec  250    188 KBytes       
[  5]   1.00-2.00   sec  4.32 MBytes  36.3 Mbits/sec    1    130 KBytes       
[  5]   2.00-3.00   sec  4.69 MBytes  39.3 Mbits/sec    0    175 KBytes       
[  5]   3.00-4.00   sec  4.21 MBytes  35.3 Mbits/sec    1    118 KBytes       
[  5]   4.00-5.00   sec  4.53 MBytes  38.0 Mbits/sec    0    165 KBytes       
[  5]   5.00-6.00   sec  4.13 MBytes  34.6 Mbits/sec    1    112 KBytes       
[  5]   6.00-7.00   sec  4.45 MBytes  37.3 Mbits/sec    0    160 KBytes       
[  5]   7.00-8.00   sec  4.68 MBytes  39.3 Mbits/sec    0    197 KBytes       
[  5]   8.00-9.00   sec  4.31 MBytes  36.2 Mbits/sec    1    133 KBytes       
[  5]   9.00-10.00  sec  4.72 MBytes  39.6 Mbits/sec    0    177 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  45.2 MBytes  37.9 Mbits/sec  254             sender
[  5]   0.00-10.11  sec  44.4 MBytes  36.9 Mbits/sec                  receiver

iperf Done.

Mac

iperf3 -c iperf3.velocityonline.net -p 5201
Connecting to host iperf3.velocityonline.net, port 5201
[  7] local 1.2.3.4 port 59478 connected to 216.162.135.203 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  4.27 MBytes  35.8 Mbits/sec                  
[  7]   1.00-2.00   sec  4.48 MBytes  37.6 Mbits/sec                  
[  7]   2.00-3.00   sec  4.71 MBytes  39.5 Mbits/sec                  
[  7]   3.00-4.00   sec  4.67 MBytes  39.1 Mbits/sec                  
[  7]   4.00-5.00   sec  4.61 MBytes  38.7 Mbits/sec                  
[  7]   5.00-6.00   sec  4.57 MBytes  38.4 Mbits/sec                  
[  7]   6.00-7.00   sec  4.45 MBytes  37.3 Mbits/sec                  
[  7]   7.00-8.00   sec  4.74 MBytes  39.7 Mbits/sec                  
[  7]   8.00-9.00   sec  4.43 MBytes  37.1 Mbits/sec                  
[  7]   9.00-10.00  sec  4.74 MBytes  39.8 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.00  sec  45.7 MBytes  38.3 Mbits/sec                  sender
[  7]   0.00-10.02  sec  45.4 MBytes  38.0 Mbits/sec                  receiver

stephenw10

Ah, so not a great test server I guess, both well below the limit you were seeing.

Are you able to put something on the WAN side to test against? Another pfSense instance is what I would probably use there.

docklost

@stephenw10

I want to be clear about my tests. The Firewall one was from the pfsense box to a public iperf server. The Mac went though my pfsense box to the same server. I want to make sure I did it the way you wanted me to do it.