High latency reported by gateway monitoring after upgrade



  • I notice one strange thing, the gateway monitoring reports very high latencies (between 200 to 500ms). The Ping tool from diags reports the same erratic values:

    PING XX.XX.XX.193 (XX.XX.XX.193) from XX.XX.XX.211: 56 data bytes
    64 bytes from XX.XX.XX.193: icmp_seq=0 ttl=255 time=249.198 ms
    64 bytes from XX.XX.XX.193: icmp_seq=1 ttl=255 time=7.642 ms
    64 bytes from XX.XX.XX.193: icmp_seq=2 ttl=255 time=7.163 ms
    64 bytes from XX.XX.XX.193: icmp_seq=3 ttl=255 time=41.792 ms
    64 bytes from XX.XX.XX.193: icmp_seq=4 ttl=255 time=8.605 ms
    64 bytes from XX.XX.XX.193: icmp_seq=5 ttl=255 time=264.649 ms
    64 bytes from XX.XX.XX.193: icmp_seq=6 ttl=255 time=7.349 ms
    64 bytes from XX.XX.XX.193: icmp_seq=7 ttl=255 time=145.968 ms
    64 bytes from XX.XX.XX.193: icmp_seq=8 ttl=255 time=295.444 ms
    64 bytes from XX.XX.XX.193: icmp_seq=9 ttl=255 time=6.980 ms

    –- XX.XX.XX.193 ping statistics ---
    10 packets transmitted, 10 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 6.980/103.479/295.444/116.485 ms

    But if I ping the same gateway from a Windows PC on the LAN I get this:

    Pinging XX.XX.XX.193 with 32 bytes of data:
    Reply from XX.XX.XX.193: bytes=32 time=32ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=9ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=8ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=6ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=7ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=8ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=7ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=7ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=8ms TTL=254
    Reply from XX.XX.XX.193: bytes=32 time=8ms TTL=254

    Ping statistics for XX.XX.XX.193:
        Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 6ms, Maximum = 32ms, Average = 10ms

    In 2.2.6 the gateway monitoring was reporting normal values up till this morning when I upgraded to 2.3 . I tried several speedtest sites from different PC on the LAN and they all normal latencies.



  • Split this to its own topic because it's unrelated to where it was posted and this is something we've had other threads on before and likely will have more now.

    What you're likely seeing with that symptom is being rate limited by the IP you're pinging. dpinger pings more frequently than apinger, twice a second instead of once, by default. Edit the gateway and change your probe interval to 1000, and it probably stabilizes. Or don't ping your gateway, that's not a worthwhile indication of your Internet connectivity and routers often rate limit pings to their own IPs, use 8.8.8.8 or something else as a monitor IP.



  • I'm seeing something similar after upgrading to 2.3
    Please see the screenshots from the monitoring tab for WAN quality

    ![Screen Shot 2016-04-13 at 2.00.49 PM.png](/public/imported_attachments/1/Screen Shot 2016-04-13 at 2.00.49 PM.png)
    ![Screen Shot 2016-04-13 at 2.00.49 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-04-13 at 2.00.49 PM.png_thumb)
    ![Screen Shot 2016-04-12 at 5.36.47 PM.png](/public/imported_attachments/1/Screen Shot 2016-04-12 at 5.36.47 PM.png)
    ![Screen Shot 2016-04-12 at 5.36.47 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2016-04-12 at 5.36.47 PM.png_thumb)



  • @cmb:

    Split this to its own topic because it's unrelated to where it was posted and this is something we've had other threads on before and likely will have more now.

    What you're likely seeing with that symptom is being rate limited by the IP you're pinging. dpinger pings more frequently than apinger, twice a second instead of once, by default. Edit the gateway and change your probe interval to 1000, and it probably stabilizes. Or don't ping your gateway, that's not a worthwhile indication of your Internet connectivity and routers often rate limit pings to their own IPs, use 8.8.8.8 or something else as a monitor IP.

    Thanks! I tried using 8.8.8.8 as monitoring IP and changed the probe interval to 1000 and 2000, making sure the alert interval was greater than the probe interval (as suggested on the settings page), but the results were the same. The latency reported was all over the place from 40ms to 500+ms and even showing the gateway offline from time to time. All this while the connections is actually working with no noticeable latency for clients on the LAN.



  • 40ms to 500+ms

    And you sure your internet connection or Googles DNS server isnt getting slammed?

    Id traceroute to that server and pick a closer hop to you that responds to pings.

    :)

    edit- reason I mention is that Im literally 9 hops from Googles 8.8.8.8 which I believe is across Puget Sound from me in Seattle.

    Ping statistics for 8.8.8.8:
        Packets: Sent = 122, Received = 122, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 5ms, Maximum = 49ms, Average = 10ms

    Not to brag but I have no jitter on my internet connection until I get past my ISP's stuff.



  • High on my list of "nice to have"  :)

    @chpalmer:

    Not to brag but I have no jitter on my internet connection until I get past my ISP's stuff.



  • @MaxPF:

    Thanks! I tried using 8.8.8.8 as monitoring IP and changed the probe interval to 1000 and 2000, making sure the alert interval was greater than the probe interval (as suggested on the settings page), but the results were the same. The latency reported was all over the place from 40ms to 500+ms and even showing the gateway offline from time to time. All this while the connections is actually working with no noticeable latency for clients on the LAN.

    In that case it's more likely it's timing related on the system. Are you running in Hyper-V by chance? If you enable Hyper-V's time sync, and enable NTP inside the VM, you'll end up with timing issues that cause system clock accuracy issues which results in what you're describing.



  • @chaos777b:

    I'm seeing something similar after upgrading to 2.3
    Please see the screenshots from the monitoring tab for WAN quality

    Yours looks pretty reasonable, though might be hitting rate-limiting like I mentioned earlier. Is your monitor IP your gateway IP? Might want to try increasing the interval. Also what does pinging the same IP from Diag>Ping look like?



  • No VMs, this is an Alix 2D13 box. I don't think it's a problem with my connection (residential cable 60/10) or Google getting slammed. This is a ping and trace route from a Windows 10 PC on my LAN:

    C:\Users\Max>ping /n 10 8.8.8.8

    Pinging 8.8.8.8 with 32 bytes of data:
    Reply from 8.8.8.8: bytes=32 time=17ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=17ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=15ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=18ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=15ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=16ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=15ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=15ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=17ms TTL=57
    Reply from 8.8.8.8: bytes=32 time=15ms TTL=57

    Ping statistics for 8.8.8.8:
        Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 15ms, Maximum = 18ms, Average = 16ms

    C:\Users\Max>tracert 8.8.8.8

    Tracing route to google-public-dns-a.google.com [8.8.8.8]
    over a maximum of 30 hops:

    1    15 ms    8 ms    <1 ms  alix.xxxxxxx.ca [192.168.1.1]
      2    13 ms    15 ms    9 ms  10.106.198.129
      3    48 ms    19 ms    16 ms  67.231.220.33
      4    20 ms    62 ms    22 ms  van58-9-231-73.dynamic.rogerstelecom.net [209.148.231.73]
      5    71 ms    54 ms    46 ms  van58-9-229-225.dynamic.rogerstelecom.net [209.148.229.225]
      6    57 ms    17 ms    34 ms  209.85.255.197
      7    80 ms    36 ms    65 ms  209.85.244.81
      8    16 ms    16 ms    39 ms  google-public-dns-a.google.com [8.8.8.8]

    Trace complete.

    This is from the Alix box (used the diag tools) just few seconds apart from previous test:

    PING 8.8.8.8 (8.8.8.8 ) from 24.xx.xx.xx: 56 data bytes
    64 bytes from 8.8.8.8: icmp_seq=0 ttl=58 time=417.078 ms
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=15.511 ms
    64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=15.816 ms
    64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=730.556 ms
    64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=15.938 ms
    64 bytes from 8.8.8.8: icmp_seq=5 ttl=58 time=16.812 ms
    64 bytes from 8.8.8.8: icmp_seq=6 ttl=58 time=734.701 ms
    64 bytes from 8.8.8.8: icmp_seq=7 ttl=58 time=731.509 ms
    64 bytes from 8.8.8.8: icmp_seq=8 ttl=58 time=730.188 ms
    64 bytes from 8.8.8.8: icmp_seq=9 ttl=58 time=725.335 ms

    –- 8.8.8.8 ping statistics ---
    10 packets transmitted, 10 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 15.511/413.344/734.701/336.799 ms

    Traceroute (using ICMP):

    1  10.106.198.129  847.450 ms  814.365 ms  195.907 ms
    2  67.231.220.33  1021.659 ms  19.124 ms  244.071 ms
    3  209.148.231.73  749.957 ms  184.235 ms  823.869 ms
    4  209.148.229.225  24.710 ms  576.026 ms  191.700 ms
    5  209.85.255.197  210.232 ms  999.869 ms  227.566 ms
    6  209.85.244.81  450.445 ms  17.743 ms  319.402 ms
    7  8.8.8.8  14.587 ms  161.061 ms  823.111 ms

    There is no noticeable latency from any of the clients on the LAN. Speed test, ping tests are all normal, even playing online games is smooth. The problem seems to be when testing from the Alix box. In 2.2.6 apinger was reporting reporting the correct values.