Insane latency, Roadrunner shows 237GB traffic in 1 hour [>500Mb/s]



  • OK, I'm tired of beating my head against the wall, so here I am to ask, and thanks, this forum really helped me out once before.

    My latency went through the roof about 10 days ago, averaging 800+ms.  Can't find anything to cause this, no traffic shaping, no squid, nothing complicated, my cpu is generally 1-4%, memory 20%, no proxy, no vpn, same IP address [even after a reset cablemodem].  Latency averaged 10-20, then jumped to the 800 level, just now, off pfsense:

    (pts/1) root% ping netgear.com
    PING netgear.com (206.16.44.90): 56 data bytes
    64 bytes from 206.16.44.90: icmp_seq=0 ttl=239 time=1689.948 ms
    64 bytes from 206.16.44.90: icmp_seq=1 ttl=239 time=1690.413 ms
    64 bytes from 206.16.44.90: icmp_seq=2 ttl=239 time=1686.832 ms
    64 bytes from 206.16.44.90: icmp_seq=3 ttl=239 time=1693.035 ms
    64 bytes from 206.16.44.90: icmp_seq=4 ttl=239 time=1706.723 ms
    64 bytes from 206.16.44.90: icmp_seq=5 ttl=239 time=1722.520 ms
    64 bytes from 206.16.44.90: icmp_seq=6 ttl=239 time=1721.005 ms
    64 bytes from 206.16.44.90: icmp_seq=7 ttl=239 time=1733.508 ms
    64 bytes from 206.16.44.90: icmp_seq=8 ttl=239 time=1737.384 ms
    64 bytes from 206.16.44.90: icmp_seq=10 ttl=239 time=1416.852 ms
    64 bytes from 206.16.44.90: icmp_seq=11 ttl=239 time=1144.971 ms
    64 bytes from 206.16.44.90: icmp_seq=12 ttl=239 time=773.192 ms
    64 bytes from 206.16.44.90: icmp_seq=13 ttl=239 time=912.830 ms
    64 bytes from 206.16.44.90: icmp_seq=14 ttl=239 time=935.013 ms
    64 bytes from 206.16.44.90: icmp_seq=15 ttl=239 time=934.783 ms
    ^C
    –- netgear.com ping statistics ---

    17 packets transmitted, 15 packets received, 11.8% packet loss
    round-trip min/avg/max/stddev = 773.192/1433.267/1737.384/362.798 ms

    That^^ is ridiculous, I realized I needed help from you folk.  So, perusing my roadrunner stats on their site, I see they show me with 670GB for Feb, 0 March, 0 April, 760 GB May, 239 GB for June til now.  The two months with zero traffic I was online as always.  Further noodling  and I see that for one hour on may 15, my traffic was 237GB!  This is roughly half-gigabit speed, but I have only 20Mb/s connection.  Does anyone have a idea what I'm seeing?  Is timewarner maybe throttling me in some way?

    And, the squirelliness continues, some traceroutes,
    a windows cmndline:

    C:\Windows\system32>TRACERT.EXE netgear.com

    Tracing route to netgear.com [206.16.44.90]
    over a maximum of 30 hops:

    1    <1 ms    <1 ms    <1 ms  pfsense [10.0.0.1]
      2  1612 ms  1623 ms  1367 ms  10.239.73.1
      3  453 ms  477 ms  493 ms  tge7-1.austtxm-er02.texas.rr.com [66.68.1.125]
      4  611 ms  623 ms  668 ms  tge0-10-0-11.austtxrdcsc-cr02.texas.rr.com [24.175.41.22]
      5  762 ms  767 ms  767 ms  agg22.hstntxl3-cr01.texas.rr.com [24.175.41.48]
      6  673 ms  663 ms  687 ms  ae-2-0.cr0.hou30.tbone.rr.com [66.109.6.108]
      7  799 ms  803 ms  845 ms  107.14.17.141
      8  879 ms  888 ms  896 ms  ip65-47-204-109.z204-47-65.customer.algx.net [65.47.204.109]
      9  681 ms  672 ms  681 ms  192.205.36.101
    10  807 ms  838 ms  835 ms  cr1.dlstx.ip.att.net [12.123.18.74]
    11  927 ms  922 ms  935 ms  cr1.phmaz.ip.att.net [12.122.28.182]
    12  1015 ms  1019 ms  1021 ms  12.123.158.5
    13  1098 ms  1120 ms  1126 ms  12-122-254-218.attens.net [12.122.254.218]
    14  1174 ms  1197 ms  1236 ms  mdf002c7613r0002-gig-12-1.phx1.attens.net [63.241.130.202]
    15  1272 ms  1264 ms  1276 ms  206.16.44.90

    Then, same computer, winmtr:

    Host # loss sent rcvd best avg worst last
    10.0.0.1 1 0 % 85 85 0 0 4 0
    10.239.73.1 2 100 % 44 0 ∞ 0 0 0
    66.68.1.125 3 0 % 67 67 76 292 369 276
    24.175.41.22 4 4 % 65 62 161 318 361 342
    24.175.41.48 5 4 % 64 61 193 284 362 271
    66.109.6.108 6 1 % 65 64 144 257 362 216
    107.14.17.141 7 0 % 65 65 145 282 386 261
    65.47.204.109 8 1 % 64 63 182 141 376 0
    192.205.36.101 9 1 % 64 63 126 272 361 238
    12.123.18.74 10 100 % 48 0 ∞ 0 0 0
    12.122.28.182 11 100 % 48 0 ∞ 0 0 0

    • 12 100 % 0 0 ∞ 0 0 0
      12.122.254.218 13 100 % 51 0 ∞ 0 0 0
      63.241.130.202 14 100 % 43 0 ∞ 0 0 0
      206.16.44.90 15 0 % 62 62 143 340 396 324

    And, ???? nmap, same PC–almost looks normal, WTF

    Starting Nmap 6.25 ( http://nmap.org ) at 2013-06-10 04:15 Central Daylight Time
    Nmap scan report for netgear.com (206.16.44.90)
    Host is up (0.67s latency).
    TRACEROUTE (using proto 1/icmp)

    HOP RTT      ADDRESS

    1  0.00 ms  pfsense (10.0.0.1)
    2  0.00 ms  10.239.73.1
    3  0.00 ms  tge7-1.ausbtx5402h.texas.rr.com (66.68.1.125)
    4  15.00 ms tge0-10-0-11.ausutxir02r.texas.rr.com (24.175.41.22)
    5  15.00 ms agg22.hstntxl3-cr01.texas.rr.com (24.175.41.48)
    6  15.00 ms ae-2-0.cr0.hou30.tbone.rr.com (66.109.6.108)
    7  47.00 ms 107.14.17.141
    8  15.00 ms ip65-47-204-109.z204-47-65.customer.algx.net (65.47.204.109)
    9  15.00 ms 192.205.36.101
    10  47.00 ms cr1.dlstx.ip.att.net (12.123.18.74)
    11  32.00 ms cr1.phmaz.ip.att.net (12.122.28.182)
    12  15.00 ms 12.123.158.5
    13  15.00 ms 12-122-254-218.attens.net (12.122.254.218)
    14  16.00 ms mdf002c7613r0002-gig-12-1.phx1.attens.net (63.241.130.202)
    15  16.00 ms 206.16.44.90
    Nmap done: 1 IP address (1 host up) scanned in 14.16 seconds

    The 10.239.79.1 is the NIC at the LAN interface.  A few infos:

    Canonical Hostname pfsense
    Listening IP 10.0.0.1
    Kernel Version FreeBSD 8.1-RELEASE-p6 #0 Mon Dec 12 18:15
    Distro Name FreeBSD
    Uptime 154 days 13 hours 35 minutes
    Current Users 3
    Load Averages 0.20 0.14 0.10
    3.1%

    (pts/1) root% ifconfig
    fwe0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            options=8 <vlan_mtu>ether 02:11:d8:3b:81:71
            ch 1 dma -1
    fwip0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
            lladdr 0.11.d8.0.1.3b.81.71.a.2.ff.fe.0.0.0.0
    re0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:14:d1:15:45:33
            inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
            inet6 fe80::214:d1ff:fe15:4533%re0 prefixlen 64 scopeid 0x3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    re1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:21:2f:2f:a5:92
            inet 10.0.5.1 netmask 0xffffff00 broadcast 10.0.5.255
            inet6 fe80::221:2fff:fe2f:a592%re1 prefixlen 64 scopeid 0x4
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    nfe0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=80008 <vlan_mtu,linkstate>ether 00:1a:92:df:2a:14
            inet6 fe80::21a:92ff:fedf:2a14%nfe0 prefixlen 64 scopeid 0x5
            inet 173.174.94.52 netmask 0xffffe000 broadcast 255.255.255.255
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    pflog0: flags=100 <promisc>metric 0 mtu 33664
    pfsync0: flags=0<> metric 0 mtu 1460
            syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
    enc0: flags=0<> metric 0 mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
            nd6 options=3 <performnud,accept_rtadv>ovpnc1: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
            options=80000 <linkstate>re0_vlan1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=3 <rxcsum,txcsum>ether 00:14:d1:15:45:33
            inet6 fe80::211:d800:13b:8171%re0_vlan1 prefixlen 64 scopeid 0xb
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
            vlan: 1 parent interface: re0
    ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::211:d800:13b:8171%ovpns2 prefixlen 64 scopeid 0xc
            inet 10.0.3.1 –> 10.0.3.2 netmask 0xffffffff
            nd6 options=3 <performnud,accept_rtadv>Opened by PID 41888
    tun3: flags=8010 <pointopoint,multicast>metric 0 mtu 1500
            options=80000 <linkstate>I'm stumped, any help would be greatly appreciated.</linkstate></pointopoint,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,simplex,multicast></linkstate></pointopoint,multicast></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></promisc></full-duplex></performnud,accept_rtadv></vlan_mtu,linkstate></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,promisc,simplex,multicast></broadcast,simplex,multicast></vlan_mtu></broadcast,simplex,multicast>


  • Netgate Administrator

    I'd have to guess an upstream problem given the stats on the Roadrunner site. I mean why no data at all for 2 months and 500Mbps for an hour? Something definitely wrong at their end IMHO.

    Steve



  • You said that you have a 20 mb/s connection than that speed is impossible and it might be that the problem was intern, but if your server is co-located and you got a 20mbit connection, be sure to look out since receiving high spikes might cost your legs because you will be rated for the spike after more than one day is past.

    to me it feels like this:

    1. DDOS(if the time is nearly precise one hour than the problem is most likely a ddos attack, perhaps a bot test?)
    2. intern conflict
    3. Kernel Panic?!?



  • Well, i feel a tad sheepish, and assish, but it was my reinstall of windows8 that was causing the problem.  I thought of that, but, 1-couldn't think why that would cause a tracert done by pfsense itself and all the other connected systems to go bad, and 2-could see no indicators of windows being set up differently, or anything that I could see that suggested some kind of 'footprint' of windows getting it's grubby fingers into the mix.  I need to talk to timewarner about the erroneous traffic reporting, but such calls tend to make me want to kill myself, but ya gotta do …

    Thanks for the suggestions guys, have a good one