SG-1000 throughput slow down


  • Netgate Administrator

    Hmm, nothing much happening there besides the NIC interrupt load which is what you would expect. Definitely not CPU limited then. What sort of throughput were you seeing when that was shown?

    Steve



  • Still had it on the screen

    File Size: 50 MB, Time: 118.0096 s, Speed: 3.39 Mbps
    


  • Don't know if this would help, but I found a file with in my ISP network that gave me near top speed for testing.

    I now get only ~3% idle cpu and a ~27% reduction in throughput instead of a ~76% reduction when testing

    http://speedcheck.cdn.on.net/100meg.test direct to cable modem

    File Size: 100 MB, Time: 10.6448 s, Speed: 75.15 Mbps
    File Size: 100 MB, Time: 10.4513 s, Speed: 76.55 Mbps
    File Size: 100 MB, Time: 10.3895 s, Speed: 77.00 Mbps
    

    http://speedcheck.cdn.on.net/100meg.test throught SG-1000

    File Size: 100 MB, Time: 14.1694 s, Speed: 56.46 Mbps
    File Size: 100 MB, Time: 14.3546 s, Speed: 55.73 Mbps
    File Size: 100 MB, Time: 14.5417 s, Speed: 55.01 Mbps
    
    134 processes: 2 running, 110 sleeping, 22 waiting
    CPU: 11.9% user,  0.0% nice, 11.9% system, 73.5% interrupt,  2.7% idle
    Mem: 26M Active, 80M Inact, 109M Wired, 25M Buf, 268M Free
    Swap:
    
    PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
    11 root -92 - 0K 176K WAIT 843:37 71.11%
    10 root 155 ki31 0K 8K RUN 110.0H 3.66%
    53297 root -74 0 9612K 4460K bpf 0:00 1.68% /usr/local/bandwidthd/bandwidthd
    53284 root -74 0 9612K 4460K bpf 0:00 1.68% /usr/local/bandwidthd/bandwidthd
    53748 root -74 0 9612K 4460K bpf 0:00 1.67% /usr/local/bandwidthd/bandwidthd
    54507 root -74 0 9612K 4460K bpf 0:00 1.67% /usr/local/bandwidthd/bandwidthd
    54421 root -74 0 9612K 4460K bpf 0:00 1.66% /usr/local/bandwidthd/bandwidthd
    54133 root -74 0 9612K 4460K bpf 0:00 1.66% /usr/local/bandwidthd/bandwidthd
    53617 root -74 0 9612K 4460K bpf 0:00 1.65% /usr/local/bandwidthd/bandwidthd
    54145 root -74 0 9612K 4460K bpf 0:00 1.62% /usr/local/bandwidthd/bandwidthd
    9275 unbound 4 0 21764K 12948K kqread 0:02 1.19% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
    72982 root 43 0 7312K 3172K RUN 0:00 1.12% top -aSH

    how much of throughput reduction would be expected when putting the SG-1000 inline?


  • Netgate Administrator

    I have tested the SG-1000 to at lest 125Mbps so I would not expect any reduction on a 100Mb line. At least not limited by CPU.

    Steve



  • Thank you @stephenw10, do you have any suggestions on what I can try next to figure out why I'm seeing a reduction in throughput?

    Do I need to capture traffic logs? I should be able to do that on the laptop.


  • Netgate Administrator

    Yes if you're able to capture test traffic we should be able to see any TCP weirdness for example.

    Steve



  • Sorry for the slow update, work got in the way. anyways got some capture data and anonymised it, They can be found on this onedrive share pcapng files.

    I could see anything that jumped out but then again I'm not really sure what I should be looking for.


  • Netgate Administrator

    Great. Ok those captures are ~55Mbps via the SG-1000 and ~75Mbps direct? Rather than one of the very low numbers you saw previously like 2Mbps?

    Steve


  • LAYER 8 Global Moderator

    Take a look at your window size.. Looks be huge different in size between when your running through sg1000 an when not... So yeah that would DRASTICALLY affect your overall download speed..

    0_1539014148425_winsize.png

    Also seeing lots of retrans in your sg1000 sniff..

    Where exactly was this sniff taken, at the client or on the sg1000?


  • Netgate Administrator

    Hmm, this is interesting. One of us appears to have a borked wireshark and it's probably me. I'm showing that same window scaling issue but it's in the other direction. But that appears to agree with the actual packet data.... 🙃

    The direction I'm interested in, Remote server to private IP, seems ungraphable. Hence probably me!


  • LAYER 8 Global Moderator

    Well the one sniff has some issues - doesn't look like it caught the syn?

    Clearly something going on there.. Would like to see cleaner sniffs and tests - and where exactly is the sniff being done at?



  • Hi,
    The sniff was done on the laptop as I needed to bypass the SG-1000 for one of the test.

    I can do another sniff from the SG-1000 if that would help? You also said you would like to see a clean capture? What would you like me to do here.

    All I did on this sniff was to filter for the remote address in src and dst then ran it through a anonymiser for posting.

    And yes I found a down load source that gives me a good speed on my ISP but as you see it still has a reduction through the SG-1000.

    Thanks for having a look.


  • LAYER 8 Global Moderator

    Clean in the fact the sg1000 sniff is missing the start of the conversation.. I show it seeming to miss the start of conversation.

    0_1539026121427_missinginfo.png

    So not sure if you caught the tail end of one speed test and then started another, etc..

    But from sg1000 sniff having way smaller window size then yeah the speed of that speed test is going to be slower..



  • Thanks @johnpoz,
    When I get chance I'll redo the sg-1000 sniff, do you have any ideas on why the windows size would be different?

    The only thing I can think of is, I was lazy and ran my sg-1000 sniff over my home network i.e laptop->switch->switch->sg-1000->cable modem, I'll try the sniff directly connected to the sg-1000 next time to see if that helps.

    For a bit of info:
    The two switches ara a WNDR3709 running OpenWRT for AP and a GS108. The laptop was on a wired connection laptop->GS108->WNDR3709->SG-1000->Cable Modem.

    stuck at work for the next 10-12hrs so I can't do any test for a bit.



  • Ok uploaded 2 more sniffs (got real slow connection tonight) same link pcapng files

    this time I connected to the SG-1000 direct and then the cable modem direct, but may have to do the test once ISP network gets back up to speed.

    anyways have a look and see if you see anything strange.

    Thanks.


  • LAYER 8 Global Moderator

    Your still not catching the start in the direct sniff.. But again your window resize are no where close to each other... So yeah no shit your download is going to be way slower!!

    Your window size when your direct is huge compared to when your on the sg1000 per those sniffs

    0_1539169143508_winsize.png

    You running a proxy or something on pfsense? If I had to guess to why your seeing the smaller windowsize have to guess its due to your dupes or fast retrans..

    Not allowing the windowsize to grow... But that is just a GUESS!!! If you want to benchmark the sg1000 and what it can do you really need to make sure environment is the same in both your tests. So take the isp and the internet out of the equation and put a server on your wan, and then your client on the lan -- do your speedtesting, iperf, download off a http, etc. But can promise you this if your window size is that small compared to when it grows large then yeah your going to see a real speed difference..



  • Ok run a test with ipef3 running on a RPI on the wan side, ran the test 3 time in each setup and the sniffs are only filtered on the RPI address 10.0.0.2 dis and src, so hopefully it should have got all the info.

    Test 1:
    PC->GS108->SG-1000->RPI: wireshark 20181012_Filter_SG-1000.pcapng

    Test 2:
    PC->RPI: wireshark 20181012_Filter_Direct.pcapng

    Both file can be found at the same link pcapng files

    Thanks.


  • LAYER 8 Global Moderator

    you for such tests you don't need to capture the payload right... That way your sniff is not so LARGE..

    What was the speed difference? HUGE amount of out of order packets? Which doesn't seem possible?? And still seeing what wireshark says are retrans and dupes, etc.. The direct see no window size updates they start out and stay there.. While the other tests shows lots of adjustments and higher than the 6000 was seeing before.

    But something OFF in that sniff through the sg1000, I don't have a sg1000 but will try and test this weekend on how they should look going through firewall..0_1539339415893_outoforder.png



  • Thanks @johnpoz,
    I'm kind of new to wireshark/sniffing and only have done the basics, how should I set it up to run the test. what are you looking for and what can I drop.

    Thanks for your help.


  • LAYER 8 Global Moderator

    in the interface options section just change the snaplen to something only a few bytes vs the default of the whole thing.. We really just need to see the headers we don't need all the data to troubleshoot what is going on.