[SOLVED] Strange upload performance with Gigabit FTTH connection

bmeeks

@steveits said in Strange upload performance with Gigabit FTTH connection:

@bmeeks if you had a spare drive you could take your good/live drive out and install on that.

Yeah, that is an option. This newer machine boots from an SSD. Easy to grab an different Intel PCIe NIC and test with that, too.

Gertjan

@bmeeks said in Strange upload performance with Gigabit FTTH connection:

However, my Dell XPS 8940 always gets 950+ megabits/sec down, but never exceeds 200 megabits/sec up!

That's my office PC !

I was reading this thread, an thought : let's test mine.
I had a VDSL 23 Mbits down / 2 Mbits up for the last decade or so, so I wasn't doing 'speedtest' a lot.
Since last month : 1 Gbit up & down. Pure fiber fun.

I remote logged into my Office PC, the 8940, opened a browser, 'speedtest' and .... emptied my mouth full with coffee on my screen :

Ok, upfront : I don't know what happened at that moment on my network.
I know that, using that PC, running Macrium backup, it spits out the full 1 Gbit to my local Syno NAS.
Like :

So, the killer NIC can spit out a close to 1Gbit/sec.

I might want to try the build in Wifi NIC ....
I have that killer stuff still installed, as it permits me to scan the the wifi networks around me. But it's will join the "Candy Bar" apps soon.

From the PC I'm using right now, also a Dell, an older OptiPlex 5080 running W10Pro :

So, well, ISP promises one thing, reality is another.

Btw : I have a 4100.

I'll investigate on my side.

johnpoz

@gertjan said in Strange upload performance with Gigabit FTTH connection:

opened a browser, 'speedtest'

Try the client speedtest..

Windows has a fatter client for speedtest.net in the app store.

bmeeks

@Gertjan:
I downloaded and installed this free open source speed test app: https://openspeedtest.com/selfhosted-speedtest. It can be installed either on a conventional web server such as nginx, or you can download a standalone app that can run on any Windows client. I chose the standalone package, and that's the one in the link. If you want the web server version, just back up to their home page. I installed my LAN speed tester on a 2018 vintage HP desktop PC with a Realtek motherboard NIC.

This local test server will let you test speeds within your LAN just like iperf3. I prefer this one over iperf3 for experimenting with this problem because of the super friendly GUI. My ISP is using the same open source speed tester and hosting it locally inside their network (on a CGNAT subnet, so not available from outside). I get 980 megabits/sec up and down to my LAN speed tester. I get about the same 980 down (varies a little from test to test as expected) but only 150-180 up to my ISP's server from my Dell 8940. That's what blows my mind. The Killer E2600 NIC seems capable of saturating the Gigabit local link, but chokes going external. How does it know ?? .

Doing packet captures, I see a lot of TCP problems when going to the external server. I see zero when testing to the LAN server. So, my ISP does have some issues. But weird that my Dell 8950 (with a Killer E3100G 2.5 Gig NIC) can hit 900-950 megabits/sec up to the ISP's server. Also my Windows virtual machines hosted on ESXi can get right at Gigabit link saturation speed up and down to the ISP's server. These other machines do sporadically fail to get top speeds, but even their "low point" upload speed far outpaces what the Dell 8940 with the Kill E2600 NIC gets.

Gertjan

@bmeeks

I went the old fashioned way : let's check if there is some one else with 'upload' issues.
Same PC - or same series.
Found : https://www.dell.com/community/XPS-Desktops/XPS-8940-internet-speed-capped-at-1-Gbps/td-p/8278282

Before :

Then I shut those two option (in the Killer Intelligence Center) to 'vide' (empty ? Off ?) :

Re test - didn't reboot the PC :

and things get better :

I can modulate real time the upload speed by switching this witch from On to off and back :

On => the needle falls down to a couple of Mbits/sec
Off => It spikes to 750+ Mbits/sec in less then a second.

I guess this calls for the kill of the killer.

bmeeks

@gertjan:
Yes, the Killer Control Center (now called Killer Intelligence Center in the newest software) is a bad thing. But I have completely uninstalled it from my PC.

I did not test it prior to removing, but I suspect, if installed, the software would impact local LAN uploads (like to my test server) as well as WAN uploads.

stephenw10

Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

bmeeks

@stephenw10 said in Strange upload performance with Gigabit FTTH connection:

Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

That is certainly my suspicion, but damned if I can find it. I've searched everywhere. Even went so far as to reinstall the bloatware, turn everything off in the app, and then tested. No change. Removed it all again -- still no change.

A new Intel i225 NIC arrives today or tomorrow. Will disable the Killer E2600 NIC and install the Intel and see what that does.

bmeeks

Wanted to come back and update this thread. I ordered a generic Intel i225v NIC from Amazon and installed in my Dell XPS 8940. I plugged my LAN cable into the new NIC and instantly my upload problem is solved. Get the same full external link upload speed on the 8940 as I was getting on the newer XPS 8950 (which comes with the Intel Killer E3100G, by the way).

I still don't understand why the Killer E2600 NIC in the Dell XPS 8940 would saturate the link when uploading on the LAN and fail to breach 200 megabits/sec or so upload when sending to external sites on the WAN. As I mentioned at the start of this thread, the XPS 8950 with the E3100G NIC could saturate the same external link every time. So could a 2018 HP desktop with a Realtek NIC and several of my ESXi virtual machines.

So, something was fishy with the E2600 motherboard NIC. I have not been able to identify what it could be. And nothing in my mind can explain the disparity between LAN upload speeds and WAN upload speeds with that NIC. The only possibility that remotely makes sense is the original Killer software did something relative to the E2600 driver such that it is throttling traffic to external (non-LAN) IP networks.

But replacing the Killer E2600 with a generic i225v fixed the problem. I will leave the motherboard NIC enabled but not plugged up. It may be handy to have the extra port and put it in a separate static IP subnet for quickly connecting up something for testing or setup (like an extra Netgate appliance, for example). It does not seem to cause any issues having the onboard Killer E2600 enabled while using the Intel i225v in a PCIe slot.

johnpoz

@bmeeks said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

E2600 driver such that it is throttling traffic to external (non-LAN) IP networks.

I would agree - but... When I test to different lan across pfsense, which isn't local - but is rfc1918.. I do not see such an issue...

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  receiver

Its a few mbps slower - but its being routed, and actually doing outbound nat.. Because my nas is multihomed and the IP I use on a different vlan it talks back on the lan IP and is asymmetrical..

Working from home tmrw - so might change the IP to something non rfc to see if see anything - but I don't believe I have any of that killer stuff installed. But upon looking, I think I must of installed some driver or something that installed some crapware as well - because I see these killer services.

The good thing, other than you got it sorted finally. Is if I ever get gig internet and see such problems I know I can just install a different nic in my 8940 to get around the problem - hehe..

edit: weird, I just stopped all those services - and retested.. And my iperf was crappy compared to with them running

stopped and disabled those killer services and get this

$ iperf3.exe -c 192.168.2.42
Connecting to host 192.168.2.42, port 5201
[  5] local 192.168.9.100 port 35419 connected to 192.168.2.42 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  75.5 MBytes   633 Mbits/sec
[  5]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
[  5]   2.00-3.00   sec   111 MBytes   930 Mbits/sec
[  5]   3.00-4.00   sec   111 MBytes   928 Mbits/sec
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec
[  5]   5.00-6.00   sec  78.2 MBytes   657 Mbits/sec
[  5]   6.00-7.00   sec   111 MBytes   929 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   935 Mbits/sec
[  5]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
[  5]   9.00-10.00  sec   111 MBytes   931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.02 GBytes   877 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.02 GBytes   876 Mbits/sec                  receiver

Maybe that killer nic needs that shit?

SteveITS

@johnpoz said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

if I ever get gig internet and see such problems

"first world problem"

johnpoz

@steveits ehhehe true ;) hehehe

edit: NeverMind - those services are not needed.. I just retested and not seeing any slowdowns - seems some users kicked off streaming, and so I think pfsense took a bit of hit since it was busy handling the outbound traffic for my plex users...

I will do some more testing when not doing any serving up of stuff..

bmeeks

@johnpoz:
It did not make sense to me either. I honestly can't figure out what it may be. All I do know is that with the replacement Intel card both internal and external upload at the same general speed now.

Of course the external tests do vary slightly, but nothing like they did before. For example, in a series of tests over say a 15-minute period, I might see the upload bounce between 750 and 960. That can easily be load in my ISP's network during primetime. I'm in the Eastern Time Zone and was testing between 6:00 PM and 7:00 PM (kids home from school and streaming). Also saw the download bounce in sync. There is also another buddy of mine in the neighborhood testing for the ISP. He has some automated tests running, and I may be periodically colliding with one of his tests. We are on the same XGS-PON system but different OLT ports.

But there also may be some growing pains on the ISP's network as this is both new in the ground and new headend hardware for them to become familiar with. The tech did tell me they have a vendor engineer remoting in tomorrow to check the settings on all the OLT hardware. But any issues in their network I don't think were impacting what I was seeing on the Dell 8940. I say that because the Dell 8950 worked fine while the 8940 was stumbling. You would expect ISP network issues to have impacted the 8940 and 8950 equally.

johnpoz

@bmeeks well keep us updated.. Very interesting problem to be sure - and I would agree, if isp issue - would of thought your 8950 would of seen the same sort of issues.

bmeeks

@johnpoz said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

@bmeeks well keep us updated.. Very interesting problem to be sure - and I would agree, if isp issue - would of thought your 8950 would of seen the same sort of issues.

Will do. The downside of a small town locally-owned ISP is they can't justify the expense of a large IPv4 space for public IPs, so they resort to CGNAT. But the big upside is you can talk to actual humans you know on a first-name basis when you have problems or questions. That makes for a better relationship than you get with a big company where any support issue first takes you to some remote call center with a minimum wage person reading from a script .

For example, I still have my cable ISP for a bit until I am fully satisfied with the new fiber service and any startup kinks all worked out. But with the cable ISP, I can't even call the local office. The "local" phone number is published, but it automatically routes you to their call center. And now, since COVID, all their reps actually work from home. So you get routed to the call center and then to the cell phones of their Help Desk folks! Last time I called- when I upgraded to their Gigabit service- the lady that activated the upgrade was at her house on her cell phone! I know this because I could tell via the voice quality, and she kept telling me she was having issues with her PC remoting back into the ISP's systems.

stephenw10

Nice!
That is odd. Gotta be some remaining setting somewhere. Maybe some hardware off-loading being enabled.
Have you tested the old NIC since installing the new one? We have seen NICs get stuck in some odd config until a full power cycle is done. It seems possible you might have not done that even booting Linux. But you would have to pull the plug to install a NIC.

bmeeks

@stephenw10 said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

Nice!
That is odd. Gotta be some remaining setting somewhere. Maybe some hardware off-loading being enabled.
Have you tested the old NIC since installing the new one? We have seen NICs get stuck in some odd config until a full power cycle is done. It seems possible you might have not done that even booting Linux. But you would have to pull the plug to install a NIC.

No, a complete pull-the-plug power down was not done with the motherboard NIC still in use. I did power-off when doing the Linux live-image test because that's the only way to get the Dell to boot from USB. But as you say, a power-off is not the same as power off and pull the plug. I did pull the plug when adding the new NIC.

Might give that theory a shot today.

stephenw10

I once spent weeks trying to diagnose a PC with failing USB ports that behaved like that. Even reinstalling Windows only sometimes brought them back. I didn't realise I was only fully power cycling it on some installs. Incredibly frustrating!

bmeeks

I did a quick comparison using first the Killer E2600 NIC and then the i225v Intel generic. This was after I had fully powered down and unplugged power to the PC yesterday when installing the new Intel NIC.

Killer E2600 Results

Intel i225v Results

The speed results will naturally vary a few 10s of megabits/sec randomly per test due to load on the ISP network. These tests are to their internal non-public speed test server.

One variable in the above test I have not accounted for is that the wireless NIC on the Dell motherboard is also a Killer chip. And that driver uses some of the same Killer network services. It had apparently turned them back on at the previous reboot as I found them running prior to running the Killer test today. So, it's possible the service did something to the Killer E2600 NIC when I powered it back up yesterday, and that impacts the speed test results today. Really a strange puzzle.

I don't want to totally cripple this PC for in-depth testing as it's my main machine for daily work. Stephen's theory has merit, though. It is possible the Windows driver does something to the NIC hardware at bootup that adversely impacts the test results. But evidence from the Linux boot test indicates that whatever is done is not persistent as it did not survive into the Linux environment. There, the NIC performed just fine.

stephenw10

Ah, right I'd forgotten it ran correctly in Linux.
Hmm, bizarre indeed. Only so much time you can spend diagnosing stuff...