Latency spikes while transferring data
-
Hello all, I have been struggling with this all day and can't seem to find a solution. When I run a speedtest on comcast's site, the latency skyrockets on both the download and upload tests, with upload being the worst. I've tried enabling/disabling TSO, LRO, Checksum Offload, MSI-X, and fastforwarding, with no difference. Enabling polling basically kills the interface (no data can be sent).
System has 2x Intel 82573L interfaces.
Edit: I've also tried commenting out extra loader.conf stuff and extra Tunables, with no difference.
I'll include ping results, loader.conf, and Tunables below. Any suggestions are greatly appreciated! :)
Ping during speedtest:
/root(4): ping bing.com PING bing.com (204.79.197.200): 56 data bytes 64 bytes from 204.79.197.200: icmp_seq=0 ttl=120 time=13.405 ms 64 bytes from 204.79.197.200: icmp_seq=1 ttl=120 time=13.757 ms 64 bytes from 204.79.197.200: icmp_seq=2 ttl=120 time=13.666 ms 64 bytes from 204.79.197.200: icmp_seq=3 ttl=120 time=12.597 ms 64 bytes from 204.79.197.200: icmp_seq=4 ttl=120 time=13.712 ms 64 bytes from 204.79.197.200: icmp_seq=5 ttl=120 time=13.684 ms 64 bytes from 204.79.197.200: icmp_seq=6 ttl=120 time=14.116 ms 64 bytes from 204.79.197.200: icmp_seq=7 ttl=120 time=12.549 ms 64 bytes from 204.79.197.200: icmp_seq=8 ttl=120 time=13.600 ms 64 bytes from 204.79.197.200: icmp_seq=9 ttl=120 time=18.159 ms 64 bytes from 204.79.197.200: icmp_seq=10 ttl=120 time=59.694 ms 64 bytes from 204.79.197.200: icmp_seq=11 ttl=120 time=91.635 ms 64 bytes from 204.79.197.200: icmp_seq=12 ttl=120 time=104.155 ms 64 bytes from 204.79.197.200: icmp_seq=13 ttl=120 time=107.565 ms 64 bytes from 204.79.197.200: icmp_seq=14 ttl=120 time=126.674 ms 64 bytes from 204.79.197.200: icmp_seq=15 ttl=120 time=153.777 ms 64 bytes from 204.79.197.200: icmp_seq=16 ttl=120 time=142.239 ms 64 bytes from 204.79.197.200: icmp_seq=17 ttl=120 time=159.227 ms 64 bytes from 204.79.197.200: icmp_seq=18 ttl=120 time=178.841 ms 64 bytes from 204.79.197.200: icmp_seq=19 ttl=120 time=175.886 ms 64 bytes from 204.79.197.200: icmp_seq=20 ttl=120 time=16.405 ms 64 bytes from 204.79.197.200: icmp_seq=21 ttl=120 time=16.096 ms 64 bytes from 204.79.197.200: icmp_seq=22 ttl=120 time=13.026 ms 64 bytes from 204.79.197.200: icmp_seq=23 ttl=120 time=13.708 ms 64 bytes from 204.79.197.200: icmp_seq=24 ttl=120 time=22.384 ms 64 bytes from 204.79.197.200: icmp_seq=25 ttl=120 time=14.192 ms 64 bytes from 204.79.197.200: icmp_seq=26 ttl=120 time=9764.597 ms 64 bytes from 204.79.197.200: icmp_seq=27 ttl=120 time=8770.282 ms 64 bytes from 204.79.197.200: icmp_seq=28 ttl=120 time=7776.026 ms 64 bytes from 204.79.197.200: icmp_seq=29 ttl=120 time=6775.911 ms 64 bytes from 204.79.197.200: icmp_seq=30 ttl=120 time=5781.350 ms 64 bytes from 204.79.197.200: icmp_seq=31 ttl=120 time=4780.921 ms 64 bytes from 204.79.197.200: icmp_seq=32 ttl=120 time=3779.889 ms 64 bytes from 204.79.197.200: icmp_seq=33 ttl=120 time=2784.440 ms 64 bytes from 204.79.197.200: icmp_seq=34 ttl=120 time=1784.390 ms 64 bytes from 204.79.197.200: icmp_seq=35 ttl=120 time=784.341 ms 64 bytes from 204.79.197.200: icmp_seq=36 ttl=120 time=15.181 ms 64 bytes from 204.79.197.200: icmp_seq=37 ttl=120 time=13.373 msloader.conf:
autoboot_delay="3" vm.kmem_size="435544320" vm.kmem_size_max="535544320" kern.ipc.nmbclusters="70000" debug.acpi.disabled="thermal" #kern.hz="2000" #kern.timecounter.hardware="TSC" #kern.timecounter.smp_tsc="1" #kern.timecounter.smp_tsc_adjust="1" ahci_load="YES" # AHCI aio_load="YES" # Async IO system calls autoboot_delay="3" # reduce boot menu delay from 10 to 3 seconds. cc_htcp_load="YES" # H-TCP Congestion Control for more aggressive increase in speed on higher # latency, high bandwidth networks. amdtemp_load="YES" # amd K8, K10, K11 thermal sensors #hw.em.enable_aim="1" # enable Intel's Adaptive Interrupt Moderation to reduce load for igb(4) (default 1) #hw.em.max_interrupt_rate="19200" # maximum number of interrupts per second generated by single igb(4) (default 8000) hw.em.num_queues="1" # number of queues supported on the hardware NIC (default 0),(Intel i350 = 8 queues) # For saturated network set to zero(0) which will auto tune and the driver will create # as many queues as CPU cores up to a max of eight(8). For lightly loaded networks set to # one(1) to reduce interrupts, lower latency and increase efficiency. (vmstat -i) hw.em.enable_msix="1" # enable MSI-X interrupts for PCI-E devices so nic polling is not needed anymore (default 1) hw.em.eee_setting="0" # Enable Energy Efficient Ethernet dev.em.0.eee_control="0" dev.em.1.eee_control="0" hw.em.txd="4096" # igb under load will not drop packets by using transmit and receive descriptor hw.em.rxd="4096" # rings in main memory which point to packet buffers. the igb driver transfers packet # data to and from main memory independent of the CPU, using the descriptor rings as # lists of packet transmit and receive requests to carry out. Increase each if # your machine or network is saturated or if you have plenty of ram. (default 1024) hw.em.rx_process_limit="-1" # maximum number of received packets to process at a time, The default is # too low for most firewalls. recommend "-1" which means unlimited (default 100) #hw.em.rx_abs_int_delay="33" # Default receive interrupt delay limit in usecs #hw.em.tx_abs_int_delay="66" # Default transmit interrupt delay limit in usecs #hw.em.rx_int_delay="33" # Default receive interrupt delay in usecs #hw.em.tx_int_delay="66" # Default transmit interrupt delay in usecs net.isr.bindthreads="1" # bind a network thread to a real single CPU core _IF_ you are using a single network # queue (hw.igb.num_queues="1") and network processing is using less then 90% of the # single CPU core. For high bandwidth systems settting bindthreads to "0" will spread # the network processing load over multiple cpus allowing the system to handle more # throughput. The default is faster for most systems with multiple queues. (default 0) net.link.ifqmaxlen="55" # network interface output queue length in number of packets. we recommend the # number of packets the interface can transmit in 50 milliseconds. it is more efficient to # send packets out of a queue then to re-send them from an application, especially from # high latency wireless devices. if your upload speed is 25 megabit then set to # around "107". Do NOT set too high as to avoid excessive buffer bloat. (default 50) # calculate: bandwidth divided by 8 bits times 1000 divided by the MTU times 0.05 seconds # ( ( (25/8) * 1000 ) / 1.448 ) * 0.05 = 107.90 packets in 50 milliseconds. net.inet.tcp.tcbhashsize="1024" kern.ipc.somaxconn="2048" # On some systems HPET is almost 2 times faster than default ACPI-fast # Useful on systems with lots of clock_gettime / gettimeofday calls # See http://old.nabble.com/ACPI-fast-default-timecounter,-but-HPET-83--faster-td23248172.html # After revision 222222 HPET became default: http://svnweb.freebsd.org/base?view=revision&revision=222222 #kern.timecounter.hardware="HPET" # Tweaks hardware #coretemp_load="YES" #intel legal.intel_wpi.license_ack="1" legal.intel_ipw.license_ack="1" boot_multicons="YES" boot_serial="YES" comconsole_speed="9600" console="comconsole,vidconsole" hw.usb.no_pf="1"Tunables:
vfs.forcesync default (0) debug.pfftpproxy 1 vfs.read_max default (32) net.inet.ip.portrange.first default (1024) net.inet.tcp.blackhole default (2) net.inet.udp.blackhole default (1) net.inet.ip.random_id default (1) net.inet.tcp.drop_synfin default (1) net.inet.ip.redirect default (1) net.inet6.ip6.redirect default (1) net.inet6.ip6.use_tempaddr default (0) net.inet6.ip6.prefer_tempaddr default (0) net.inet.tcp.syncookies 0 net.inet.tcp.recvspace 65535 net.inet.tcp.sendspace 65535 net.inet.ip.fastforwarding 0 net.inet.tcp.delayed_ack D default (0) net.inet.udp.maxdgram default (57344) net.link.bridge.pfil_onlyip default (0) net.link.bridge.pfil_member default (1) net.link.bridge.pfil_bridge default (0) net.link.tap.user_open default (1) kern.randompid default (347) net.inet.ip.intr_queue_maxlen default (1000) hw.syscons.kbd_reboot default (0) net.inet.tcp.inflight.enable 0 net.inet.tcp.log_debug default (0) net.inet.icmp.icmplim default (0) net.inet.tcp.tso default (1) net.inet.udp.checksum default (1) kern.ipc.maxsockbuf default (4262144) net.inet.tcp.mssdflt 1460 net.inet.tcp.msl 9000 net.inet.ip.redirect 0 net.inet.raw.maxdgram 9216 net.inet.raw.recvspace 9216 kern.ipc.somaxconn 1024 dev.em.0.fc 0 dev.em.1.fc 0 net.inet.ip.check_interface 1 net.inet.ip.process_options 0 net.inet.icmp.drop_redirect 1 net.inet.tcp.drop_synfin 1 net.inet.tcp.fast_finwait2_recycle 1 net.inet.tcp.icmp_may_rst 0 net.inet.tcp.path_mtu_discovery 0 net.inet.tcp.nolocaltimewait 1 net.inet.tcp.rfc3042 0 net.inet6.icmp6.nodeinfo 0 -
A speedtest wil purposely flood the device with traffic to see how fast the link will go. Your "ping" packets will always get on the end of a long queue at the router when trying to go out (upstream) and so it will be "big number" of milliseconds before they even escape the router.
To get around that you need to play with the Traffic Shaper and let it prioritize ACK packets, ping packets, little stuff. Then those packets can "jump the queue" and get out ahead of all those waiting speed test packets.
Start reading about traffic shaping… -
I think the question here is why there's a latency spike while transferring data and not how you will shape the bandwidth. Even you shape the data still a latency spike will occur as this affects on how big the data will be transfer to another station.
-
I think the question here is why there's a latency spike while transferring data
Uhm… because
A speedtest wil purposely flood the device with traffic
?
-
Thanks for the info! I confirmed the same spike on my Ubiquiti edge lite, I guess I never looked at latency while speed testing before, only reason I did this time was adding a lot of the tweaks seen on the first post.
Would traffic shaping be a benefit or a hinder regarding online gaming (xbox 360, ps3, pc, ps4 soon) since I have heard shaping can introduce some latency. I also have a few other devices (not mine) that are sometimes in use at the same time while gaming, which generally consists of either youtube, netflix, or bittorrent, though the link doesn't get saturated very often.
The router currently has 1GB DDR2 RAM and a single core AMD Athlon64 2000+ 1GHz.
If shaping would help, is there a guide that explains how everything works/what it means?
Also, regarding the timecounter, which should I be using, ACPI-safe or TSC?
Thanks again! :)
Edit: One more question, does anyone know why the interfaces stop working when polling is enabled?
-
Latency that high is because of "buffer bloat". TCP uses dropped packets to determine when a link is congested. Because buffers are too large on many residential connections, the latency goes sky high because you have a long line of packets wasting to be processed. You can control the buffer bloat on your sending side by rate limiting the upload speed of your router/firewall to be 80%-90% of your actual upload speed.
Unfortunately, you can't control buffer bloat on your receiving end. It's up to your ISP to properly size those buffers.
-
Thanks for the buffer bloat explanation, figures comcast would have a huge buffer. :|
I did try the traffic shaping wizard, and that worked great to almost eliminate the upload latency spike while saturated, though download still hits about 200ms, which isn't all that bad I guess. I'll continue to research and tweak it.
The only thing left I can't figure out is why polling causes the network interfaces to stop working, I would think Intel Pro NICs would support that feature. Is there something in the kernel that isn't compiled in by default which is required for polling?