Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    [SOLVED] Strange upload performance with Gigabit FTTH connection

    Hardware
    6
    55
    360
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeks
      bmeeks @SteveITS last edited by

      @steveits said in Strange upload performance with Gigabit FTTH connection:

      @bmeeks if you had a spare drive you could take your good/live drive out and install on that.

      Yeah, that is an option. This newer machine boots from an SSD. Easy to grab an different Intel PCIe NIC and test with that, too.

      1 Reply Last reply Reply Quote 0
      • Gertjan
        Gertjan @bmeeks last edited by Gertjan

        @bmeeks said in Strange upload performance with Gigabit FTTH connection:

        However, my Dell XPS 8940 always gets 950+ megabits/sec down, but never exceeds 200 megabits/sec up!

        That's my office PC !

        I was reading this thread, an thought : let's test mine.
        I had a VDSL 23 Mbits down / 2 Mbits up for the last decade or so, so I wasn't doing 'speedtest' a lot.
        Since last month : 1 Gbit up & down. Pure fiber fun.

        I remote logged into my Office PC, the 8940, opened a browser, 'speedtest' and .... emptied my mouth full with coffee on my screen :

        610f6bbf-5c93-4a3c-8153-29b269e0592f-image.png

        Ok, upfront : I don't know what happened at that moment on my network.
        I know that, using that PC, running Macrium backup, it spits out the full 1 Gbit to my local Syno NAS.
        Like :

        0c060174-9b66-4ecf-8e06-bf981a6f2e7e-image.png

        So, the killer NIC can spit out a close to 1Gbit/sec.

        I might want to try the build in Wifi NIC ....
        I have that killer stuff still installed, as it permits me to scan the the wifi networks around me. But it's will join the "Candy Bar" apps soon.

        From the PC I'm using right now, also a Dell, an older OptiPlex 5080 running W10Pro :

        9c5862ef-322c-4c16-a795-41725b1a16db-image.png

        So, well, ISP promises one thing, reality is another.

        Btw : I have a 4100.

        I'll investigate on my side.

        No "help me" PM's please. Use the forum.

        johnpoz bmeeks 2 Replies Last reply Reply Quote 0
        • johnpoz
          johnpoz LAYER 8 Global Moderator @Gertjan last edited by

          @gertjan said in Strange upload performance with Gigabit FTTH connection:

          opened a browser, 'speedtest'

          Try the client speedtest..

          Windows has a fatter client for speedtest.net in the app store.

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 23.01 | Lab VMs CE 2.6, 2.7

          1 Reply Last reply Reply Quote 0
          • bmeeks
            bmeeks @Gertjan last edited by

            @Gertjan:
            I downloaded and installed this free open source speed test app: https://openspeedtest.com/selfhosted-speedtest. It can be installed either on a conventional web server such as nginx, or you can download a standalone app that can run on any Windows client. I chose the standalone package, and that's the one in the link. If you want the web server version, just back up to their home page. I installed my LAN speed tester on a 2018 vintage HP desktop PC with a Realtek motherboard NIC.

            This local test server will let you test speeds within your LAN just like iperf3. I prefer this one over iperf3 for experimenting with this problem because of the super friendly GUI. My ISP is using the same open source speed tester and hosting it locally inside their network (on a CGNAT subnet, so not available from outside). I get 980 megabits/sec up and down to my LAN speed tester. I get about the same 980 down (varies a little from test to test as expected) but only 150-180 up to my ISP's server from my Dell 8940. That's what blows my mind. The Killer E2600 NIC seems capable of saturating the Gigabit local link, but chokes going external. How does it know ?? 😁.

            Doing packet captures, I see a lot of TCP problems when going to the external server. I see zero when testing to the LAN server. So, my ISP does have some issues. But weird that my Dell 8950 (with a Killer E3100G 2.5 Gig NIC) can hit 900-950 megabits/sec up to the ISP's server. Also my Windows virtual machines hosted on ESXi can get right at Gigabit link saturation speed up and down to the ISP's server. These other machines do sporadically fail to get top speeds, but even their "low point" upload speed far outpaces what the Dell 8940 with the Kill E2600 NIC gets.

            1 Reply Last reply Reply Quote 0
            • Gertjan
              Gertjan last edited by Gertjan

              @bmeeks

              I went the old fashioned way : let's check if there is some one else with 'upload' issues.
              Same PC - or same series.
              Found : https://www.dell.com/community/XPS-Desktops/XPS-8940-internet-speed-capped-at-1-Gbps/td-p/8278282

              Before :

              caccd858-5fb9-4f01-bcc6-1f2478cafe7e-image.png

              Then I shut those two option (in the Killer Intelligence Center) to 'vide' (empty ? Off ?) :

              5bc2e00e-3cf1-4502-afc2-7dd85d084135-image.png

              Re test - didn't reboot the PC :

              482820c5-c2cf-4fda-9e5b-35af22e9bf3f-image.png

              and things get better :

              I can modulate real time the upload speed by switching this witch from On to off and back :

              11df4dff-f617-46b7-b39c-16cd027dbeca-image.png

              On => the needle falls down to a couple of Mbits/sec
              Off => It spikes to 750+ Mbits/sec in less then a second.

              I guess this calls for the kill of the killer.

              No "help me" PM's please. Use the forum.

              bmeeks 1 Reply Last reply Reply Quote 0
              • bmeeks
                bmeeks @Gertjan last edited by

                @gertjan:
                Yes, the Killer Control Center (now called Killer Intelligence Center in the newest software) is a bad thing. But I have completely uninstalled it from my PC.

                I did not test it prior to removing, but I suspect, if installed, the software would impact local LAN uploads (like to my test server) as well as WAN uploads.

                1 Reply Last reply Reply Quote 0
                • stephenw10
                  stephenw10 Netgate Administrator last edited by

                  Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

                  bmeeks 1 Reply Last reply Reply Quote 0
                  • bmeeks
                    bmeeks @stephenw10 last edited by bmeeks

                    @stephenw10 said in Strange upload performance with Gigabit FTTH connection:

                    Did it disable that setting when it uninstalled though? I could easily believe it's still set somewhere.

                    That is certainly my suspicion, but damned if I can find it. I've searched everywhere. Even went so far as to reinstall the bloatware, turn everything off in the app, and then tested. No change. Removed it all again -- still no change.

                    A new Intel i225 NIC arrives today or tomorrow. Will disable the Killer E2600 NIC and install the Intel and see what that does.

                    1 Reply Last reply Reply Quote 0
                    • bmeeks
                      bmeeks last edited by bmeeks

                      Wanted to come back and update this thread. I ordered a generic Intel i225v NIC from Amazon and installed in my Dell XPS 8940. I plugged my LAN cable into the new NIC and instantly my upload problem is solved. Get the same full external link upload speed on the 8940 as I was getting on the newer XPS 8950 (which comes with the Intel Killer E3100G, by the way).

                      I still don't understand why the Killer E2600 NIC in the Dell XPS 8940 would saturate the link when uploading on the LAN and fail to breach 200 megabits/sec or so upload when sending to external sites on the WAN. As I mentioned at the start of this thread, the XPS 8950 with the E3100G NIC could saturate the same external link every time. So could a 2018 HP desktop with a Realtek NIC and several of my ESXi virtual machines.

                      So, something was fishy with the E2600 motherboard NIC. I have not been able to identify what it could be. And nothing in my mind can explain the disparity between LAN upload speeds and WAN upload speeds with that NIC. The only possibility that remotely makes sense is the original Killer software did something relative to the E2600 driver such that it is throttling traffic to external (non-LAN) IP networks.

                      But replacing the Killer E2600 with a generic i225v fixed the problem. I will leave the motherboard NIC enabled but not plugged up. It may be handy to have the extra port and put it in a separate static IP subnet for quickly connecting up something for testing or setup (like an extra Netgate appliance, for example). It does not seem to cause any issues having the onboard Killer E2600 enabled while using the Intel i225v in a PCIe slot.

                      johnpoz 1 Reply Last reply Reply Quote 2
                      • johnpoz
                        johnpoz LAYER 8 Global Moderator @bmeeks last edited by johnpoz

                        @bmeeks said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

                        E2600 driver such that it is throttling traffic to external (non-LAN) IP networks.

                        I would agree - but... When I test to different lan across pfsense, which isn't local - but is rfc1918.. I do not see such an issue...

                        [ ID] Interval           Transfer     Bitrate
                        [  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec                  sender
                        [  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec                  receiver
                        

                        Its a few mbps slower - but its being routed, and actually doing outbound nat.. Because my nas is multihomed and the IP I use on a different vlan it talks back on the lan IP and is asymmetrical..

                        Working from home tmrw - so might change the IP to something non rfc to see if see anything - but I don't believe I have any of that killer stuff installed. But upon looking, I think I must of installed some driver or something that installed some crapware as well - because I see these killer services.

                        crap.jpg

                        The good thing, other than you got it sorted finally. Is if I ever get gig internet and see such problems I know I can just install a different nic in my 8940 to get around the problem - hehe..

                        edit: weird, I just stopped all those services - and retested.. And my iperf was crappy compared to with them running

                        stopped and disabled those killer services and get this

                        $ iperf3.exe -c 192.168.2.42
                        Connecting to host 192.168.2.42, port 5201
                        [  5] local 192.168.9.100 port 35419 connected to 192.168.2.42 port 5201
                        [ ID] Interval           Transfer     Bitrate
                        [  5]   0.00-1.00   sec  75.5 MBytes   633 Mbits/sec
                        [  5]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
                        [  5]   2.00-3.00   sec   111 MBytes   930 Mbits/sec
                        [  5]   3.00-4.00   sec   111 MBytes   928 Mbits/sec
                        [  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec
                        [  5]   5.00-6.00   sec  78.2 MBytes   657 Mbits/sec
                        [  5]   6.00-7.00   sec   111 MBytes   929 Mbits/sec
                        [  5]   7.00-8.00   sec   112 MBytes   935 Mbits/sec
                        [  5]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
                        [  5]   9.00-10.00  sec   111 MBytes   931 Mbits/sec
                        - - - - - - - - - - - - - - - - - - - - - - - - -
                        [ ID] Interval           Transfer     Bitrate
                        [  5]   0.00-10.00  sec  1.02 GBytes   877 Mbits/sec                  sender
                        [  5]   0.00-10.00  sec  1.02 GBytes   876 Mbits/sec                  receiver
                        

                        Maybe that killer nic needs that shit?

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                        S bmeeks 2 Replies Last reply Reply Quote 0
                        • S
                          SteveITS @johnpoz last edited by

                          @johnpoz said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

                          if I ever get gig internet and see such problems

                          "first world problem"

                          Steve

                          Only install packages for your version, or risk breaking it. If yours is older, select it in System/Update/Update Settings.
                          When upgrading, let it finish; do not reboot early. Allow 10-15 minutes, or more depending on packages and device speed.

                          johnpoz 1 Reply Last reply Reply Quote 0
                          • johnpoz
                            johnpoz LAYER 8 Global Moderator @SteveITS last edited by johnpoz

                            @steveits ehhehe true ;) hehehe

                            edit: NeverMind - those services are not needed.. I just retested and not seeing any slowdowns - seems some users kicked off streaming, and so I think pfsense took a bit of hit since it was busy handling the outbound traffic for my plex users...

                            I will do some more testing when not doing any serving up of stuff..

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                            1 Reply Last reply Reply Quote 0
                            • bmeeks
                              bmeeks @johnpoz last edited by bmeeks

                              @johnpoz:
                              It did not make sense to me either. I honestly can't figure out what it may be. All I do know is that with the replacement Intel card both internal and external upload at the same general speed now.

                              Of course the external tests do vary slightly, but nothing like they did before. For example, in a series of tests over say a 15-minute period, I might see the upload bounce between 750 and 960. That can easily be load in my ISP's network during primetime. I'm in the Eastern Time Zone and was testing between 6:00 PM and 7:00 PM (kids home from school and streaming). Also saw the download bounce in sync. There is also another buddy of mine in the neighborhood testing for the ISP. He has some automated tests running, and I may be periodically colliding with one of his tests. We are on the same XGS-PON system but different OLT ports.

                              But there also may be some growing pains on the ISP's network as this is both new in the ground and new headend hardware for them to become familiar with. The tech did tell me they have a vendor engineer remoting in tomorrow to check the settings on all the OLT hardware. But any issues in their network I don't think were impacting what I was seeing on the Dell 8940. I say that because the Dell 8950 worked fine while the 8940 was stumbling. You would expect ISP network issues to have impacted the 8940 and 8950 equally.

                              johnpoz 1 Reply Last reply Reply Quote 0
                              • johnpoz
                                johnpoz LAYER 8 Global Moderator @bmeeks last edited by

                                @bmeeks well keep us updated.. Very interesting problem to be sure - and I would agree, if isp issue - would of thought your 8950 would of seen the same sort of issues.

                                An intelligent man is sometimes forced to be drunk to spend time with his fools
                                If you get confused: Listen to the Music Play
                                Please don't Chat/PM me for help, unless mod related
                                SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                bmeeks 1 Reply Last reply Reply Quote 0
                                • bmeeks
                                  bmeeks @johnpoz last edited by bmeeks

                                  @johnpoz said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

                                  @bmeeks well keep us updated.. Very interesting problem to be sure - and I would agree, if isp issue - would of thought your 8950 would of seen the same sort of issues.

                                  Will do. The downside of a small town locally-owned ISP is they can't justify the expense of a large IPv4 space for public IPs, so they resort to CGNAT. But the big upside is you can talk to actual humans you know on a first-name basis when you have problems or questions. That makes for a better relationship than you get with a big company where any support issue first takes you to some remote call center with a minimum wage person reading from a script 🙂.

                                  For example, I still have my cable ISP for a bit until I am fully satisfied with the new fiber service and any startup kinks all worked out. But with the cable ISP, I can't even call the local office. The "local" phone number is published, but it automatically routes you to their call center. And now, since COVID, all their reps actually work from home. So you get routed to the call center and then to the cell phones of their Help Desk folks! Last time I called- when I upgraded to their Gigabit service- the lady that activated the upgrade was at her house on her cell phone! I know this because I could tell via the voice quality, and she kept telling me she was having issues with her PC remoting back into the ISP's systems.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10
                                    stephenw10 Netgate Administrator last edited by

                                    Nice!
                                    That is odd. Gotta be some remaining setting somewhere. Maybe some hardware off-loading being enabled.
                                    Have you tested the old NIC since installing the new one? We have seen NICs get stuck in some odd config until a full power cycle is done. It seems possible you might have not done that even booting Linux. But you would have to pull the plug to install a NIC.

                                    bmeeks 1 Reply Last reply Reply Quote 0
                                    • bmeeks
                                      bmeeks @stephenw10 last edited by bmeeks

                                      @stephenw10 said in [SOLVED] Strange upload performance with Gigabit FTTH connection:

                                      Nice!
                                      That is odd. Gotta be some remaining setting somewhere. Maybe some hardware off-loading being enabled.
                                      Have you tested the old NIC since installing the new one? We have seen NICs get stuck in some odd config until a full power cycle is done. It seems possible you might have not done that even booting Linux. But you would have to pull the plug to install a NIC.

                                      No, a complete pull-the-plug power down was not done with the motherboard NIC still in use. I did power-off when doing the Linux live-image test because that's the only way to get the Dell to boot from USB. But as you say, a power-off is not the same as power off and pull the plug. I did pull the plug when adding the new NIC.

                                      Might give that theory a shot today.

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10
                                        stephenw10 Netgate Administrator last edited by

                                        I once spent weeks trying to diagnose a PC with failing USB ports that behaved like that. Even reinstalling Windows only sometimes brought them back. I didn't realise I was only fully power cycling it on some installs. Incredibly frustrating!

                                        1 Reply Last reply Reply Quote 0
                                        • bmeeks
                                          bmeeks last edited by bmeeks

                                          I did a quick comparison using first the Killer E2600 NIC and then the i225v Intel generic. This was after I had fully powered down and unplugged power to the PC yesterday when installing the new Intel NIC.

                                          Killer E2600 Results
                                          SpeedTest_Killer_E2600.png

                                          Intel i225v Results
                                          SpeedTest_Intel_i225v.png

                                          The speed results will naturally vary a few 10s of megabits/sec randomly per test due to load on the ISP network. These tests are to their internal non-public speed test server.

                                          One variable in the above test I have not accounted for is that the wireless NIC on the Dell motherboard is also a Killer chip. And that driver uses some of the same Killer network services. It had apparently turned them back on at the previous reboot as I found them running prior to running the Killer test today. So, it's possible the service did something to the Killer E2600 NIC when I powered it back up yesterday, and that impacts the speed test results today. Really a strange puzzle.

                                          I don't want to totally cripple this PC for in-depth testing as it's my main machine for daily work. Stephen's theory has merit, though. It is possible the Windows driver does something to the NIC hardware at bootup that adversely impacts the test results. But evidence from the Linux boot test indicates that whatever is done is not persistent as it did not survive into the Linux environment. There, the NIC performed just fine.

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10
                                            stephenw10 Netgate Administrator last edited by

                                            Ah, right I'd forgotten it ran correctly in Linux.
                                            Hmm, bizarre indeed. Only so much time you can spend diagnosing stuff...

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post