[SOLVED] Strange upload performance with Gigabit FTTH connection

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks valid point, but your wifes 11 is using a different nic maybe? And you reset the stack already you mentioned.. I take your wifes is default on the stack, no tweaking you did..

No tweaking on hers. And hers still has the Killer Network Performance app installed.

In digging around old files in the hidden ProgramData directory I did see reference to Dell Optimizer back last March. That would have shipped on the machine back then I think. And according to the web, that app caused lots of issues. And one post even said it caused issues specifically with accessing external sites (not on the local LAN). I must have uninstalled the Optimizer app because there is no trace of it on the machine now that I see (but have not scanned the registry yet). But perhaps it left some hooks in place ??

Bob.Dig

@johnpoz I think the vmware installation is altering Windows.
Maybe hyper-v is an option for the future.

johnpoz

@bob-dig I think he uninstalled that..

Bob.Dig

@johnpoz No, I mean that moment you install vmware it will change your Windows installation (forever).

I know that third party firewall can have an impact too, but I guess it would be already mentioned, if @bmeeks would use one.

johnpoz

@bob-dig hmmm I have not heard of that.. While I have had vmware on this machine at one point - again I don't have the upload internet to see if I am having the same problem.

But perhaps it left some hooks in place ??

I always wipe and do a clean install of my machines - just not a fan of the bloatware they always install.. While I have been happy with the dell 8940.. It was a clean install..

bmeeks

I have uninstalled VMware Workstation, but @Bob-Dig has a valid point. It may have left something in place.

I have tried turning the LRO and TSO features off and on, but it had no effect. Same with hardware checksumming -- no change in behavior. Currently, I have all those back at the defaults (and they are matching the comparable settings on my wife's working PC that has a slightly newer version of the same NIC).

I'm leaning towards something being altered deep down in the networking stack, and VMware is a prime suspect. Time to go hunting ... lots of apps do a very crappy job of fully uninstalling themselves.

johnpoz

@bmeeks but you would think when you did a reset via netsh that shit would of been removed?

Now that you have vmware removed - maybe try another reset of the stack?

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bob-dig hmmm I have not heard of that.. While I have had vmware on this machine at one point - again I don't have the upload internet to see if I am having the same problem.

But perhaps it left some hooks in place ??

I always wipe and do a clean install of my machines - just not a fan of the bloatware they always install.. While I have been happy with the dell 8940.. It was a clean install..

I do routinely remove all the bloatware, starting with the McAfee stuff and then all the Dell stuff. But I don't do a complete wipe and install.

bmeeks

@johnpoz said in Strange upload performance with Gigabit FTTH connection:

@bmeeks but you would think when you did a reset via netsh that shit would of been removed?

Now that you have vmware removed - maybe try another reset of the stack?

I think I did do the reset after VMware removal, but I will repeat just to be sure.

bmeeks

Well, I think I'm beaten. Just sitting here in the corner with my tail between my legs sulking ... . I have been unable to identify a cause for this weird external network throttling, and so therefore have not fixed it either. Speeds up and down to internal testing hosts run at full line rate, but speeds to external sites download at full Gigabit line rate, but upload only between 150 and 200 megabits/sec (usually more like 150ish).

What I've Tried
Note these steps are not necessarily in the order they were performed, they are simply in the order "remembered" by my now weary mind ... .

Tried removing VMware Workstation -- no change in behavior.
Reset the Windows network stack (twice) -- no change in behavior.
Totally uninstalled the Intel Killer E2600 Ethernet device (from Device Manager), rebooted, and let the default Microsoft driver install -- no change in behavior (although the Microsoft driver appeared a tad slower downloading).
Installed the newest Intel driver for the NIC (driver only, not the Killer Performance Center bloatware).
Uninstalled every possible piece of Dell bloatware -- no change in behavior.
Ran the Windows DISM tool to scan and repair the Windows 11 image -- no change in behavior (and the tool found nothing wrong).
Booted an Ubuntu USB live image on the machine and got the expected 950+ megabits/sec up and down speeds testing from Ubuntu to the external speed test server, so that rules out local hardware and cabling since it works fine on a Linux distro using the same NIC, cabling, and switch port.

There are only two things left to do. One is to replace the NIC with a completely different model. Likely will try that. That's a pretty easy next step with no real downside (except spending a little cash).

The other option is to wipe the PC and reload Windows 11, but I am loathe to do that as the PC has a lot of my daily work software installed and configured on it. While I could reinstall, it would be a real pain to reconfigure all that -- especially if I did all that and it still did not fix the upload speed problem!

SteveITS

@bmeeks if you had a spare drive you could take your good/live drive out and install on that.

bmeeks

@steveits said in Strange upload performance with Gigabit FTTH connection:

@bmeeks if you had a spare drive you could take your good/live drive out and install on that.

Yeah, that is an option. This newer machine boots from an SSD. Easy to grab an different Intel PCIe NIC and test with that, too.

Gertjan

@bmeeks said in Strange upload performance with Gigabit FTTH connection:

However, my Dell XPS 8940 always gets 950+ megabits/sec down, but never exceeds 200 megabits/sec up!

That's my office PC !

I was reading this thread, an thought : let's test mine.
I had a VDSL 23 Mbits down / 2 Mbits up for the last decade or so, so I wasn't doing 'speedtest' a lot.
Since last month : 1 Gbit up & down. Pure fiber fun.

I remote logged into my Office PC, the 8940, opened a browser, 'speedtest' and .... emptied my mouth full with coffee on my screen :

Ok, upfront : I don't know what happened at that moment on my network.
I know that, using that PC, running Macrium backup, it spits out the full 1 Gbit to my local Syno NAS.
Like :

So, the killer NIC can spit out a close to 1Gbit/sec.

I might want to try the build in Wifi NIC ....
I have that killer stuff still installed, as it permits me to scan the the wifi networks around me. But it's will join the "Candy Bar" apps soon.

From the PC I'm using right now, also a Dell, an older OptiPlex 5080 running W10Pro :

So, well, ISP promises one thing, reality is another.

Btw : I have a 4100.

I'll investigate on my side.

johnpoz

@gertjan said in Strange upload performance with Gigabit FTTH connection:

opened a browser, 'speedtest'

Try the client speedtest..

Windows has a fatter client for speedtest.net in the app store.

bmeeks

@Gertjan:
I downloaded and installed this free open source speed test app: https://openspeedtest.com/selfhosted-speedtest. It can be installed either on a conventional web server such as nginx, or you can download a standalone app that can run on any Windows client. I chose the standalone package, and that's the one in the link. If you want the web server version, just back up to their home page. I installed my LAN speed tester on a 2018 vintage HP desktop PC with a Realtek motherboard NIC.

This local test server will let you test speeds within your LAN just like iperf3. I prefer this one over iperf3 for experimenting with this problem because of the super friendly GUI. My ISP is using the same open source speed tester and hosting it locally inside their network (on a CGNAT subnet, so not available from outside). I get 980 megabits/sec up and down to my LAN speed tester. I get about the same 980 down (varies a little from test to test as expected) but only 150-180 up to my ISP's server from my Dell 8940. That's what blows my mind. The Killer E2600 NIC seems capable of saturating the Gigabit local link, but chokes going external. How does it know ?? .

Doing packet captures, I see a lot of TCP problems when going to the external server. I see zero when testing to the LAN server. So, my ISP does have some issues. But weird that my Dell 8950 (with a Killer E3100G 2.5 Gig NIC) can hit 900-950 megabits/sec up to the ISP's server. Also my Windows virtual machines hosted on ESXi can get right at Gigabit link saturation speed up and down to the ISP's server. These other machines do sporadically fail to get top speeds, but even their "low point" upload speed far outpaces what the Dell 8940 with the Kill E2600 NIC gets.

Gertjan

@bmeeks

I went the old fashioned way : let's check if there is some one else with 'upload' issues.
Same PC - or same series.
Found : https://www.dell.com/community/XPS-Desktops/XPS-8940-internet-speed-capped-at-1-Gbps/td-p/8278282

Before :

Then I shut those two option (in the Killer Intelligence Center) to 'vide' (empty ? Off ?) :

Re test - didn't reboot the PC :

and things get better :

I can modulate real time the upload speed by switching this witch from On to off and back :

On => the needle falls down to a couple of Mbits/sec
Off => It spikes to 750+ Mbits/sec in less then a second.

I guess this calls for the kill of the killer.

bmeeks

@gertjan:
Yes, the Killer Control Center (now called Killer Intelligence Center in the newest software) is a bad thing. But I have completely uninstalled it from my PC.

I did not test it prior to removing, but I suspect, if installed, the software would impact local LAN uploads (like to my test server) as well as WAN uploads.

stephenw10

Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

bmeeks

@stephenw10 said in Strange upload performance with Gigabit FTTH connection:

Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

That is certainly my suspicion, but damned if I can find it. I've searched everywhere. Even went so far as to reinstall the bloatware, turn everything off in the app, and then tested. No change. Removed it all again -- still no change.

A new Intel i225 NIC arrives today or tomorrow. Will disable the Killer E2600 NIC and install the Intel and see what that does.

bmeeks

Wanted to come back and update this thread. I ordered a generic Intel i225v NIC from Amazon and installed in my Dell XPS 8940. I plugged my LAN cable into the new NIC and instantly my upload problem is solved. Get the same full external link upload speed on the 8940 as I was getting on the newer XPS 8950 (which comes with the Intel Killer E3100G, by the way).

I still don't understand why the Killer E2600 NIC in the Dell XPS 8940 would saturate the link when uploading on the LAN and fail to breach 200 megabits/sec or so upload when sending to external sites on the WAN. As I mentioned at the start of this thread, the XPS 8950 with the E3100G NIC could saturate the same external link every time. So could a 2018 HP desktop with a Realtek NIC and several of my ESXi virtual machines.

So, something was fishy with the E2600 motherboard NIC. I have not been able to identify what it could be. And nothing in my mind can explain the disparity between LAN upload speeds and WAN upload speeds with that NIC. The only possibility that remotely makes sense is the original Killer software did something relative to the E2600 driver such that it is throttling traffic to external (non-LAN) IP networks.

But replacing the Killer E2600 with a generic i225v fixed the problem. I will leave the motherboard NIC enabled but not plugged up. It may be handy to have the extra port and put it in a separate static IP subnet for quickly connecting up something for testing or setup (like an extra Netgate appliance, for example). It does not seem to cause any issues having the onboard Killer E2600 enabled while using the Intel i225v in a PCIe slot.