Issues with download speed.

dbudd424

So I upgraded from 1gbps fiber to 5gbps today and have been trying to figure out why my download is always slower than my upload.

Windows 10 Machine with a 10gb SolarFlare Nic.

Windows 10 Machine with 2.5gb Nic

Pfsense Box: ( Home lab )
2.7.0-RELEASE Pfsense
i7-7700 CPU
32gb ram.
Intel X550-T2

Network:
icx-6610
Lan is plugged into a Mikrotik S+RJ10 6-speed RJ-45
2.5gb Windows machine is also plugged into a Mikrotik S+RJ10 6-speed RJ-45
Running a 40G QSFP+ to 4xSFP+ to 10gb Nic machines.

Switch showing the port is up and no errors for the port the lan is plugged into.

Services:
Running:
dhcp
dpinger
haproxy
miniupnpd
ntpd
sshd
syslogd
tailscale
telegraf
Unbound

Stopped:
iperf
PfBlockerNG ( Turned it off as I thought this was an issue)

netstat -i output: (

After a restart the Ierrs went away but starting to see them on Link#2 only atm.

I have unchecked all the hardware checksum options under advanced and network tested with them off and on and made no difference.

Iperf results for Lan:
Windows w/10gb nic hitting the lan on PFsense

Hitting my Unraid box with a 10gb card

CPU Activity:
Iperf hitting Lan

Speedtest.net

I did check with limiters under firewalls and I tested by creating a FQ_CODEL scheduler and assigning it to rules but it caused some other internet issues so I removed it out of rules.

Adding Extra Information:

Boot/loader.config:
kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
kern.ipc.nmbjumbo9="524288"
opensolaris_load="YES"
zfs_load="YES"
kern.geom.label.gptid.enable="0"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
opensolaris_load="YES"
zfs_load="YES"
autoboot_delay="3"
hw.hn.vf_transparent="0"
hw.hn.use_if_start="1"
net.link.ifqmaxlen="128"

I'm going to try changing out the ethernet cables and try swapping in a Intel X540-T2 that I have handy. Outside of that I'm not sure what else I can try.

Any suggestions or if anyone is seeing something I'm missing please let me know it would be greatly appreciated.

stephenw10

Can you see the full 5G down if you bypass the firewall entirely?

Steve

dbudd424

@stephenw10 so when they tested with a OWC Thunderbolt 3 10g Ethernet Adapter it showed full speed. If I bypass pfsense directly and go to the computer with the 10gb solarflare card using a Wiitek rj45 module I get around the same download speed but my upload speed matches the download instead of hitting 5gb when using the firewall. I'm going to toss a intel-540-t2 card into that box and re-test without the module.

stephenw10

Testing directly from the Windows client you see reduced speeds both up and down?

That does seem like a client issue then.

NOCling

Do you use HT?
If it is active go and disable it could lower latency and push throughput.

dbudd424

Wanted to follow back up. I have the issue fixed now.

After doing a bunch of testing and cable swapping. turned of HT and even did a nic card update. Tested outside of the firewall and it worked. So after I ended up just moving the Intel nic down to the next PCI slot and it fixed. the issue was a motherboard issue.

stephenw10

Nice. I can only dream of those speeds.