iperf3 speeds confusing - fast = bad internet slow = good internet?


  • Yesterday I spent a lot of time going through my network settings and attempting to optimize certain things. I was running iperf3 on many of my VM's to see if I had any bottlenecks on my 10Gbe capable system (ESXi host).

    pfsense being virtualized with two 10GBe NIC's passed through always seemed to deliver the speeds I needed for internet (1500/1000 fiber) - however I was surprised to see a fairly low iperf3 score of around 2.8-3Gbit/s

    After some light reading I turned on all the offloading under Advanced > Networking and wow - iperf3 jumped right up to 13-15 Gbit/s!! I was blown away at that point and let it all be.

    Everything seemed to be working fine - however today I seen a lot of failed attempts of friends and family to be able to initialize a connection to my plex server. this was surprisedin as on my LAN Plex was working fine, speed tests were fine, etc.

    So I attempted it myself from my phone on LTE - sure enough it took forever to 'buffer' the first few seconds on low bit rate and high bitrate/original quality just never loaded.

    I started to play with the 3 settings one by one until I found the combination that works.

    All 3 offloads off > works
    TSO + LRO = off and Hardware checksum = on > works
    LRO + Hardware checksum = off and TSO = on > works

    Of course my iperf3 speeds are back down to ~2.9Gbit/s and any other combination doesn't work.

    I also found during this test that turning LRO is where I see the massive iperf3 increase back to 13-15 Gbit/s.

    So, what am I missing here? How can worse iperf3 throughput numbers (which also has far more retries) not give me issues with throughput on the WAN side (coming in only?) vs much higher iperf3 numbers with far fewer retries essentially breaking the internet for at least plex/large throughputs.

    My NIC's are 2x onboard 82599EB (x520?) SFP+ to a 10Gb 16XG Edgeswitch
    8GB ram and 4v CPU to the Pfsense box.
    Disabling Suricata and NG made no difference at all.

    iperf results:

    Only LRO Offloading enabled:
    
    Connecting to host 10.10.0.254, port 5201
    [  4] local 10.10.1.10 port 46730 connected to 10.10.0.254 port 5201
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec  2.08 GBytes  17.9 Gbits/sec    0   1011 KBytes
    [  4]   1.00-2.00   sec  1.43 GBytes  12.3 Gbits/sec  995   1.28 MBytes
    [  4]   2.00-3.00   sec  1.43 GBytes  12.3 Gbits/sec  1132   1.00 MBytes
    [  4]   3.00-4.00   sec  1.64 GBytes  14.1 Gbits/sec    0   1.13 MBytes
    [  4]   4.00-5.00   sec  1.47 GBytes  12.6 Gbits/sec  616    574 KBytes
    [  4]   5.00-6.00   sec  1.49 GBytes  12.8 Gbits/sec    0   1.15 MBytes
    [  4]   6.00-7.00   sec  1.85 GBytes  15.9 Gbits/sec    0   1.27 MBytes
    [  4]   7.00-8.00   sec  1.24 GBytes  10.7 Gbits/sec    0   1.31 MBytes
    [  4]   8.00-9.00   sec  1.48 GBytes  12.7 Gbits/sec  843   1.20 MBytes
    [  4]   9.00-10.00  sec  1.42 GBytes  12.2 Gbits/sec    0   1.20 MBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec  3586             sender
    [  4]   0.00-10.00  sec  15.5 GBytes  13.3 Gbits/sec                  receiver
    iperf Done.
    
    Offloading disabled to have a properworking WAN:
    
    Connecting to host 10.10.0.254, port 5201
    [  4] local 10.10.1.10 port 47114 connected to 10.10.0.254 port 5201
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec   249 MBytes  2.09 Gbits/sec  1072    407 KBytes
    [  4]   1.00-2.00   sec   355 MBytes  2.97 Gbits/sec  2626    506 KBytes
    [  4]   2.00-3.00   sec   351 MBytes  2.94 Gbits/sec  2562    437 KBytes
    [  4]   3.00-4.00   sec   362 MBytes  3.04 Gbits/sec  2005    416 KBytes
    [  4]   4.00-5.00   sec   350 MBytes  2.93 Gbits/sec  2492    455 KBytes
    [  4]   5.00-6.00   sec   360 MBytes  3.02 Gbits/sec  2616    298 KBytes
    [  4]   6.00-7.00   sec   355 MBytes  2.98 Gbits/sec  2112    420 KBytes
    [  4]   7.00-8.00   sec   358 MBytes  3.00 Gbits/sec  2374    413 KBytes
    [  4]   8.00-9.00   sec   361 MBytes  3.03 Gbits/sec  2521    380 KBytes
    [  4]   9.00-10.00  sec   369 MBytes  3.10 Gbits/sec  1419    366 KBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec  21799             sender
    [  4]   0.00-10.00  sec  3.39 GBytes  2.91 Gbits/sec                  receiver
    
    

  • @notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:

    Yesterday I spent a lot of time going through my network settings and attempting to optimize certain things.

    Hi,

    So, I haven’t found a better guide to what you set out to achieve:
    https://calomel.org/freebsd_network_tuning.html
    https://calomel.org/network_performance.html

    😉

    +++edit:

    we mention this a lot:
    TSO / LRO is not good for you if you are a router (pfSense), if you are an endpoint = YES


  • @daddygo thanks for the links...

    However I suppose I don't see the relationship to my iperf3 speeds in this case.

    BTW, the ESXi box is very capable in terms of CPU horsepower IMO:

    2x Xeon E5-2695 v2 @ 2.40GHz

    Don't get me wrong, my current config is obviously capable of saturating my internet connection (at least downstream) with speed tests consistently around 1500-1600 Mbit/s. But ingress isn't always so stable (which I know is not so easy).

    I just did a quick iperf3 test with the offloads disabled (current working config).

    Interestingly enough the slow iperf3 speeds are only TO pfsense (pfsense being the ipef3 server). Going the other way (CentOs box being the iperf3 server) I'm seeing very good speeds as I would expect.

    I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where I'm seeing only ~2.8Gbit/s

    See results (from the CentOS box):

    Connecting to host 10.10.0.254, port 5201
    [  4] local 10.10.1.32 port 40738 connected to 10.10.0.254 port 5201
    [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
    [  4]   0.00-1.00   sec   292 MBytes  2.45 Gbits/sec   34    509 KBytes
    [  4]   1.00-2.00   sec   345 MBytes  2.89 Gbits/sec  215    431 KBytes
    [  4]   2.00-3.00   sec   356 MBytes  2.99 Gbits/sec   78    361 KBytes
    [  4]   3.00-4.00   sec   332 MBytes  2.79 Gbits/sec   57    318 KBytes
    [  4]   4.00-5.00   sec   321 MBytes  2.68 Gbits/sec   69    393 KBytes
    [  4]   5.00-6.00   sec   322 MBytes  2.71 Gbits/sec  134    501 KBytes
    [  4]   6.00-7.00   sec   325 MBytes  2.73 Gbits/sec   42    498 KBytes
    [  4]   7.00-8.00   sec   325 MBytes  2.73 Gbits/sec  113    452 KBytes
    [  4]   8.00-9.00   sec   334 MBytes  2.80 Gbits/sec   82    458 KBytes
    [  4]   9.00-10.00  sec   335 MBytes  2.81 Gbits/sec   26    540 KBytes
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec  850             sender
    [  4]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec                  receiver
    
    iperf Done.
    [root@dockersrv02 ~]# iperf3 -s
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    Accepted connection from 10.10.0.254, port 57649
    [  5] local 10.10.1.32 port 5201 connected to 10.10.0.254 port 64408
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-1.00   sec  1.04 GBytes  8.92 Gbits/sec
    [  5]   1.00-2.00   sec  1.56 GBytes  13.4 Gbits/sec
    [  5]   2.00-3.00   sec  1.66 GBytes  14.2 Gbits/sec
    [  5]   3.00-4.00   sec  1.65 GBytes  14.2 Gbits/sec
    [  5]   4.00-5.00   sec  1.66 GBytes  14.2 Gbits/sec
    [  5]   5.00-6.00   sec  1.67 GBytes  14.3 Gbits/sec
    [  5]   6.00-7.00   sec  1.64 GBytes  14.1 Gbits/sec
    [  5]   7.00-8.00   sec  1.66 GBytes  14.2 Gbits/sec
    [  5]   8.00-9.00   sec  1.58 GBytes  13.6 Gbits/sec
    [  5]   9.00-10.00  sec  1.64 GBytes  14.1 Gbits/sec
    [  5]  10.00-10.20  sec   333 MBytes  14.3 Gbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bandwidth
    [  5]   0.00-10.20  sec  0.00 Bytes  0.00 bits/sec                  sender
    [  5]   0.00-10.20  sec  16.1 GBytes  13.5 Gbits/sec                  receiver
    -----------------------------------------------------------
    Server listening on 5201
    -----------------------------------------------------------
    ^Ciperf3: interrupt - the server has terminated
    [root@dockersrv02 ~]#
    
    

  • @notorious_vr said in iperf3 speeds confusing - fast = bad internet slow = good internet?:

    I'm not quite sure where I can make the improvements reuiqred in the incoming connections to pfsense where

    We talked a lot here about this on the forum... 😉
    Always the configuration first, then the measurements (iPerf) can come.

    Yes this is different under Linux, as you can read in the description in the link.

    I wouldn’t use hypervisor instead of bare metal for an important device like router + FW, but it’s a matter of taste, you don’t have redundancy.. (I hope you know that)

    so i suggest you set your parameters first: loader.conf.local

    such as:

    • flowcontrol disable (dev.ixyz.0.fc=0)
    • EEE disable (dev.ixyz.0.eee_disabled: value=1)
      (these are explanatory values)

    something like that, but implement it on your own hardware set:

    0604c2c9-ae29-45fc-bf63-b4f33a378b26-image.png

    +++edit:
    test,.... can still be these:

    ifconfig -vma
    dmesg | grep -i msi
    vmstat -i
    sysctl -a | grep num_queues