[SOLVED] Strange upload performance with Gigabit FTTH connection

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks well with that gig down you should have like 20 or 30 linux distros download and ready - have you tested it yet?

Well, the quest needs modification. Tested on Ubuntu LTS and got 964 megabits/sec down and 954 megabits/sec up. So, the issue is most definitely within the Windows 11 networking setup someplace.

I see @SteveITS has posted some possibilities while I was testing the Linux distro. Will check those out, although I previously have played around with the RSC settings.

At least I have eliminated all hardware possibilities. I really don't want to wipe and reload Windows, though. I've got too much other app configurations I hate to lose.

johnpoz

@bmeeks said in Strange upload performance with Gigabit FTTH connection:

Windows 11 networking setup someplace.

Oh yeah there you go - you found the problem ;) heheh

You got a spare disk around, some ssd you can just do a install of 10, and see if any issues with that..

edit: I browsed through that 2nd one - its mostly gibberish.. Bunch of settings that have shit to do with it.. Possible maybe the the large send offload, you could try disabling that maybe.. but disable RSS, that is nonsense..

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks said in Strange upload performance with Gigabit FTTH connection:

Windows 11 networking setup someplace.

Oh yeah there you go - you found the problem ;) heheh

You got a spare disk around, some ssd you can just do a install of 10, and see if any issues with that..

Well, I know it's not Windows 11 because my wife's 8950 machine with 11 works fine. Also have Windows 10 on an older HP desktop with Realtek motherboard NIC and that one works fine. So, it's something in the network stack of my machine. I installed it new in the middle of March last year. All my Windows installs are Windows Pro edition and are domain-joined.

I could just order another Intel NIC and install that. Or I may could totally remove the driver for the current Killer NIC and reinstall the software with JUST the driver.

johnpoz

@bmeeks valid point, but your wifes 11 is using a different nic maybe? And you reset the stack already you mentioned.. I take your wifes is default on the stack, no tweaking you did..

edit: for a test could you pull your wifes nic? Or you have another nic? Or maybe even shoot a usb nic? But what is strange is no issues locally on the nic... You found a odd one for sure.

edit2: I take it your not running any "security" sort of software - you mentioned you disabled all that nonsense that dell installs with that killer nic..

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks valid point, but your wifes 11 is using a different nic maybe? And you reset the stack already you mentioned.. I take your wifes is default on the stack, no tweaking you did..

No tweaking on hers. And hers still has the Killer Network Performance app installed.

In digging around old files in the hidden ProgramData directory I did see reference to Dell Optimizer back last March. That would have shipped on the machine back then I think. And according to the web, that app caused lots of issues. And one post even said it caused issues specifically with accessing external sites (not on the local LAN). I must have uninstalled the Optimizer app because there is no trace of it on the machine now that I see (but have not scanned the registry yet). But perhaps it left some hooks in place ??

Bob.Dig

@johnpoz I think the vmware installation is altering Windows.
Maybe hyper-v is an option for the future.

johnpoz

@bob-dig I think he uninstalled that..

Bob.Dig

@johnpoz No, I mean that moment you install vmware it will change your Windows installation (forever).

I know that third party firewall can have an impact too, but I guess it would be already mentioned, if @bmeeks would use one.

johnpoz

@bob-dig hmmm I have not heard of that.. While I have had vmware on this machine at one point - again I don't have the upload internet to see if I am having the same problem.

But perhaps it left some hooks in place ??

I always wipe and do a clean install of my machines - just not a fan of the bloatware they always install.. While I have been happy with the dell 8940.. It was a clean install..

bmeeks

I have uninstalled VMware Workstation, but @Bob-Dig has a valid point. It may have left something in place.

I have tried turning the LRO and TSO features off and on, but it had no effect. Same with hardware checksumming -- no change in behavior. Currently, I have all those back at the defaults (and they are matching the comparable settings on my wife's working PC that has a slightly newer version of the same NIC).

I'm leaning towards something being altered deep down in the networking stack, and VMware is a prime suspect. Time to go hunting ... lots of apps do a very crappy job of fully uninstalling themselves.

johnpoz

@bmeeks but you would think when you did a reset via netsh that shit would of been removed?

Now that you have vmware removed - maybe try another reset of the stack?

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bob-dig hmmm I have not heard of that.. While I have had vmware on this machine at one point - again I don't have the upload internet to see if I am having the same problem.

But perhaps it left some hooks in place ??

I always wipe and do a clean install of my machines - just not a fan of the bloatware they always install.. While I have been happy with the dell 8940.. It was a clean install..

I do routinely remove all the bloatware, starting with the McAfee stuff and then all the Dell stuff. But I don't do a complete wipe and install.

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks but you would think when you did a reset via netsh that shit would of been removed?

Now that you have vmware removed - maybe try another reset of the stack?

I think I did do the reset after VMware removal, but I will repeat just to be sure.

bmeeks

Well, I think I'm beaten. Just sitting here in the corner with my tail between my legs sulking ... . I have been unable to identify a cause for this weird external network throttling, and so therefore have not fixed it either. Speeds up and down to internal testing hosts run at full line rate, but speeds to external sites download at full Gigabit line rate, but upload only between 150 and 200 megabits/sec (usually more like 150ish).

What I've Tried
Note these steps are not necessarily in the order they were performed, they are simply in the order "remembered" by my now weary mind ... .

Tried removing VMware Workstation -- no change in behavior.
Reset the Windows network stack (twice) -- no change in behavior.
Totally uninstalled the Intel Killer E2600 Ethernet device (from Device Manager), rebooted, and let the default Microsoft driver install -- no change in behavior (although the Microsoft driver appeared a tad slower downloading).
Installed the newest Intel driver for the NIC (driver only, not the Killer Performance Center bloatware).
Uninstalled every possible piece of Dell bloatware -- no change in behavior.
Ran the Windows DISM tool to scan and repair the Windows 11 image -- no change in behavior (and the tool found nothing wrong).
Booted an Ubuntu USB live image on the machine and got the expected 950+ megabits/sec up and down speeds testing from Ubuntu to the external speed test server, so that rules out local hardware and cabling since it works fine on a Linux distro using the same NIC, cabling, and switch port.

There are only two things left to do. One is to replace the NIC with a completely different model. Likely will try that. That's a pretty easy next step with no real downside (except spending a little cash).

The other option is to wipe the PC and reload Windows 11, but I am loathe to do that as the PC has a lot of my daily work software installed and configured on it. While I could reinstall, it would be a real pain to reconfigure all that -- especially if I did all that and it still did not fix the upload speed problem!

SteveITS

@bmeeks if you had a spare drive you could take your good/live drive out and install on that.

bmeeks

@steveits said in Strange upload performance with Gigabit FTTH connection:

@bmeeks if you had a spare drive you could take your good/live drive out and install on that.

Yeah, that is an option. This newer machine boots from an SSD. Easy to grab an different Intel PCIe NIC and test with that, too.

Gertjan

@bmeeks said in Strange upload performance with Gigabit FTTH connection:

However, my Dell XPS 8940 always gets 950+ megabits/sec down, but never exceeds 200 megabits/sec up!

That's my office PC !

I was reading this thread, an thought : let's test mine.
I had a VDSL 23 Mbits down / 2 Mbits up for the last decade or so, so I wasn't doing 'speedtest' a lot.
Since last month : 1 Gbit up & down. Pure fiber fun.

I remote logged into my Office PC, the 8940, opened a browser, 'speedtest' and .... emptied my mouth full with coffee on my screen :

Ok, upfront : I don't know what happened at that moment on my network.
I know that, using that PC, running Macrium backup, it spits out the full 1 Gbit to my local Syno NAS.
Like :

So, the killer NIC can spit out a close to 1Gbit/sec.

I might want to try the build in Wifi NIC ....
I have that killer stuff still installed, as it permits me to scan the the wifi networks around me. But it's will join the "Candy Bar" apps soon.

From the PC I'm using right now, also a Dell, an older OptiPlex 5080 running W10Pro :

So, well, ISP promises one thing, reality is another.

Btw : I have a 4100.

I'll investigate on my side.

johnpoz

@gertjan said in Strange upload performance with Gigabit FTTH connection:

opened a browser, 'speedtest'

Try the client speedtest..

Windows has a fatter client for speedtest.net in the app store.

bmeeks

@Gertjan:
I downloaded and installed this free open source speed test app: https://openspeedtest.com/selfhosted-speedtest. It can be installed either on a conventional web server such as nginx, or you can download a standalone app that can run on any Windows client. I chose the standalone package, and that's the one in the link. If you want the web server version, just back up to their home page. I installed my LAN speed tester on a 2018 vintage HP desktop PC with a Realtek motherboard NIC.

This local test server will let you test speeds within your LAN just like iperf3. I prefer this one over iperf3 for experimenting with this problem because of the super friendly GUI. My ISP is using the same open source speed tester and hosting it locally inside their network (on a CGNAT subnet, so not available from outside). I get 980 megabits/sec up and down to my LAN speed tester. I get about the same 980 down (varies a little from test to test as expected) but only 150-180 up to my ISP's server from my Dell 8940. That's what blows my mind. The Killer E2600 NIC seems capable of saturating the Gigabit local link, but chokes going external. How does it know ?? .

Doing packet captures, I see a lot of TCP problems when going to the external server. I see zero when testing to the LAN server. So, my ISP does have some issues. But weird that my Dell 8950 (with a Killer E3100G 2.5 Gig NIC) can hit 900-950 megabits/sec up to the ISP's server. Also my Windows virtual machines hosted on ESXi can get right at Gigabit link saturation speed up and down to the ISP's server. These other machines do sporadically fail to get top speeds, but even their "low point" upload speed far outpaces what the Dell 8940 with the Kill E2600 NIC gets.

Gertjan

@bmeeks

I went the old fashioned way : let's check if there is some one else with 'upload' issues.
Same PC - or same series.
Found : https://www.dell.com/community/XPS-Desktops/XPS-8940-internet-speed-capped-at-1-Gbps/td-p/8278282

Before :

Then I shut those two option (in the Killer Intelligence Center) to 'vide' (empty ? Off ?) :

Re test - didn't reboot the PC :

and things get better :

I can modulate real time the upload speed by switching this witch from On to off and back :

On => the needle falls down to a couple of Mbits/sec
Off => It spikes to 750+ Mbits/sec in less then a second.

I guess this calls for the kill of the killer.