Traced down why connections are being refused under heavy load but how to fix?



  • With a FRESH install of embedded, just basic nat configuration and running torrents, I start to get more and more connections refused as if I ran out of states. My state count is 4059/100000 so that's not a problem. When I run tcpdump using,

    tcpdump -i ath0 (tcp[13] & 16 != 0) and (dst port http)

    I notice that when my connections get rejected (incomplete web page loads), i see the following:

    07:42:11.523303 IP tge.local.2535 > via.local.http: P 6028:6569(541) ack 2288 win 32749 <nop,nop,timestamp 1992614="" 2695163955="">07:42:11.719403 IP tge.local.2535 > via.local.http: . ack 2491 win 32724 <nop,nop,timestamp 1992616="" 2695164526="">07:42:11.784422 IP tge.local.2533 > via.local.http: P 8067:8607(540) ack 3548 win 32828 <nop,nop,timestamp 1992616="" 2695164055="">07:42:11.920195 IP tge.local.2535 > via.local.http: . ack 2496 win 32723 <nop,nop,timestamp 1992618="" 2695164659="">07:42:12.021325 IP tge.local.2533 > via.local.http: . ack 3750 win 32802 <nop,nop,timestamp 1992619="" 2695164785="">07:42:12.032331 IP tge.local.2535 > via.local.http: P 6569:7111(542) ack 2496 win 32723 <nop,nop,timestamp 1992619="" 2695164659="">07:42:12.221788 IP tge.local.2535 > via.local.http: . ack 2698 win 32698 <nop,nop,timestamp 1992621="" 2695165033="">07:42:12.222041 IP tge.local.2533 > via.local.http: . ack 3755 win 32802 <nop,nop,timestamp 1992621="" 2695164961="">07:42:12.323328 IP tge.local.2531 > via.local.http: . ack 2488 win 32729 <nop,nop,timestamp 1992622="" 2695165126="">07:42:12.329750 IP tge.local.2533 > via.local.http: P 8607:9121(514) ack 3755 win 32802 <nop,nop,timestamp 1992622="" 2695164961="">07:42:12.423189 IP tge.local.2535 > via.local.http: . ack 2703 win 32698 <nop,nop,timestamp 1992623="" 2695165161="">07:42:12.524200 IP tge.local.2531 > via.local.http: . ack 2493 win 32728 <nop,nop,timestamp 1992624="" 2695165263="">07:42:12.727590 IP tge.local.2535 > via.local.http: P 7111:7652(541) ack 2703 win 32698 <nop,nop,timestamp 1992626="" 2695165161="">07:42:12.878081 IP tge.local.2531 > via.local.http: P 6566:7263(697) ack 2493 win 32728 <nop,nop,timestamp 1992627="" 2695165263="">07:42:13.027431 IP tge.local.2535 > via.local.http: . ack 2906 win 32672 <nop,nop,timestamp 1992629="" 2695165728="">07:42:13.037392 IP tge.local.2535 > via.local.http: P 7652:8192(540) ack 2911 win 32672 <nop,nop,timestamp 1992629="" 2695165967="">07:42:13.128015 IP tge.local.2531 > via.local.http: . ack 2659 win 32708 <nop,nop,timestamp 1992630="" 2695165870="">07:42:13.228606 IP tge.local.2535 > via.local.http: . ack 3113 win 32850 <nop,nop,timestamp 1992631="" 2695166038="">I get a lot of nop,nop packets and I don't really know why it is doing that. Any help from the experts?

    Thanks.</nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp>



  • sysctl net.inet.tcp.sack.enable=0 try this command. Does it fix your issues?



  • I think it does help a little with the problem but sites are still not loading up completely. Are small packets like css info and other things like that getting dropped?

    I see the following:



  • This is driving me crazy. I'm surprised no one else has seen this…

    
    22:42:08.262828 IP tge.local.4333 > dav.bay0.hotmail.com.http: . ack 245002870 win 32850 <nop,nop,timestamp 0="" 2532586="">
    22:42:08.263325 IP tge.local.4333 > dav.bay0.hotmail.com.http: P 0:945(945) ack 1 win 32850 <nop,nop,timestamp 0="" 2532587="">
    22:42:08.291585 IP tge.local.4333 > dav.bay0.hotmail.com.http: . ack 531 win 32783 <nop,nop,timestamp 2532587="" 25602801="">
    22:42:08.291808 IP tge.local.4333 > dav.bay0.hotmail.com.http: F 945:945(0) ack 531 win 32783 <nop,nop,timestamp 2532587="" 25602801="">
    22:42:08.544856 IP tge.local.4334 > dav.bay0.hotmail.com.http: . ack 758041664 win 32850 <nop,nop,timestamp 0="" 2532589="">
    22:42:08.545159 IP tge.local.4334 > dav.bay0.hotmail.com.http: P 0:888(888) ack 1 win 32850 <nop,nop,timestamp 0="" 2532589="">
    22:42:08.545378 IP tge.local.4334 > dav.bay0.hotmail.com.http: P 888:1245(357) ack 1 win 32850 <nop,nop,timestamp 0="" 2532589="">
    22:42:08.599983 IP tge.local.4334 > dav.bay0.hotmail.com.http: . ack 688 win 32764 <nop,nop,timestamp 2532590="" 30286348="">
    22:42:08.600747 IP tge.local.4334 > dav.bay0.hotmail.com.http: F 1245:1245(0) ack 688 win 32764 <nop,nop,timestamp 2532590="" 30286348="">
    22:42:08.735461 IP tge.local.4335 > oe.bay116.hotmail.com.http: . ack 1840551811 win 32850 <nop,nop,timestamp 2532591="" 1831009190="">
    22:42:08.735834 IP tge.local.4335 > oe.bay116.hotmail.com.http: P 0:609(609) ack 1 win 32850 <nop,nop,timestamp 2532591="" 1831009190="">
    22:42:08.753764 IP tge.local.4335 > oe.bay116.hotmail.com.http: . ack 526 win 32784 <nop,nop,timestamp 2532591="" 1831009208="">
    22:42:08.755008 IP tge.local.4335 > oe.bay116.hotmail.com.http: F 609:609(0) ack 526 win 32784 <nop,nop,timestamp 2532591="" 1831009208="">
    22:42:08.892910 IP tge.local.4336 > oe.bay116.hotmail.com.http: . ack 2682583880 win 32850 <nop,nop,timestamp 2532593="" 1831009346="">
    22:42:08.893262 IP tge.local.4336 > oe.bay116.hotmail.com.http: P 0:609(609) ack 1 win 32850 <nop,nop,timestamp 2532593="" 1831009346="">
    22:42:08.915704 IP tge.local.4336 > oe.bay116.hotmail.com.http: . ack 525 win 32784 <nop,nop,timestamp 2532593="" 1831009368="">
    22:42:08.916222 IP tge.local.4336 > oe.bay116.hotmail.com.http: F 609:609(0) ack 525 win 32784 <nop,nop,timestamp 2532593="" 1831009368="">
    22:42:08.999397 IP tge.local.4337 > oe.bay116.hotmail.com.http: . ack 3847159758 win 32850 <nop,nop,timestamp 2532594="" 1831009454="">
    22:42:08.999724 IP tge.local.4337 > oe.bay116.hotmail.com.http: P 0:921(921) ack 1 win 32850 <nop,nop,timestamp 2532594="" 1831009454="">
    22:42:09.065854 IP tge.local.4337 > oe.bay116.hotmail.com.http: . ack 817 win 32748 <nop,nop,timestamp 2532594="" 1831009520="">
    22:42:09.066898 IP tge.local.4337 > oe.bay116.hotmail.com.http: F 921:921(0) ack 817 win 32748 <nop,nop,timestamp 2532594="" 1831009520="">
    22:42:09.122647 IP tge.local.4338 > 65.54.165.135.http: . ack 3734449601 win 32850 <nop,nop,timestamp 0="" 2532595="">
    22:42:09.123039 IP tge.local.4338 > 65.54.165.135.http: P 0:739(739) ack 1 win 32850 <nop,nop,timestamp 0="" 2532595="">
    22:42:09.144404 IP tge.local.4338 > 65.54.165.135.http: . ack 399 win 32800 <nop,nop,timestamp 2532595="" 18288140="">
    22:42:09.145203 IP tge.local.4338 > 65.54.165.135.http: F 739:739(0) ack 399 win 32800 <nop,nop,timestamp 2532595="" 18288140="">
    22:42:09.256942 IP tge.local.4339 > 65.54.165.135.http: . ack 4047933809 win 32850 <nop,nop,timestamp 0="" 2532596="">
    22:42:09.257542 IP tge.local.4339 > 65.54.165.135.http: P 0:739(739) ack 1 win 32850 <nop,nop,timestamp 0="" 2532596="">
    22:42:09.280202 IP tge.local.4339 > 65.54.165.135.http: . ack 399 win 32800 <nop,nop,timestamp 2532597="" 18403105="">
    22:42:09.280819 IP tge.local.4339 > 65.54.165.135.http: F 739:739(0) ack 399 win 32800 <nop,nop,timestamp 2532597="" 18403105="">
    22:42:09.390902 IP tge.local.4340 > 65.54.165.135.http: . ack 744914266 win 32850 <nop,nop,timestamp 0="" 2532598="">
    22:42:09.391220 IP tge.local.4340 > 65.54.165.135.http: P 0:686(686) ack 1 win 32850 <nop,nop,timestamp 0="" 2532598="">
    22:42:09.391486 IP tge.local.4340 > 65.54.165.135.http: P 686:1043(357) ack 1 win 32850 <nop,nop,timestamp 0="" 2532598="">
    22:42:09.429431 IP tge.local.4340 > 65.54.165.135.http: . ack 648 win 32769 <nop,nop,timestamp 2532598="" 18288302="">
    22:42:09.429641 IP tge.local.4340 > 65.54.165.135.http: . ack 649 win 32769 <nop,nop,timestamp 2532598="" 18288302="">
    22:42:09.430063 IP tge.local.4340 > 65.54.165.135.http: F 1043:1043(0) ack 649 win 32769 <nop,nop,timestamp 2532598="" 18288302="">
    22:42:10.722846 IP tge.local.4341 > oe.bay116.hotmail.com.http: . ack 3305645792 win 32850 <nop,nop,timestamp 2532611="" 1831011176="">
    22:42:10.723298 IP tge.local.4341 > oe.bay116.hotmail.com.http: P 0:970(970) ack 1 win 32850 <nop,nop,timestamp 2532611="" 1831011176="">
    22:42:10.723486 IP tge.local.4341 > oe.bay116.hotmail.com.http: P 970:1327(357) ack 1 win 32850 <nop,nop,timestamp 2532611="" 1831011176="">
    22:42:10.964700 IP tge.local.4341 > oe.bay116.hotmail.com.http: . ack 2253 win 32850 <nop,nop,timestamp 2532613="" 1831011416="">
    22:42:10.964912 IP tge.local.4341 > oe.bay116.hotmail.com.http: . ack 3636 win 32677 <nop,nop,timestamp 2532613="" 1831011416="">
    22:42:10.965083 IP tge.local.4341 > oe.bay116.hotmail.com.http: . ack 3637 win 32677 <nop,nop,timestamp 2532613="" 1831011416="">
    22:42:10.966292 IP tge.local.4341 > oe.bay116.hotmail.com.http: F 1327:1327(0) ack 3637 win 32677 <nop,nop,timestamp 2532613="" 183101141<br=""></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp></nop,nop,timestamp>
    


  • Sorry, disabling the sack did not help. Can someone please look at my status.php and see if there is something wrong with all the traffic coming into my box?

    http://www.pastebin.ca/984481

    Many thanks!



  • Only had a quick look but why do you have so many arp errors?

    arpresolve: can't allocate route for 24.130.144.1

    arplookup 24.130.144.1 failed: host is not on local network

    arpresolve: can't allocate route for 24.130.144.1

    arplookup 24.130.144.1 failed: host is not on local network

    arpresolve: can't allocate route for 24.130.144.1

    arplookup 24.130.144.1 failed: host is not on local network

    arpresolve: can't allocate route for 24.130.144.1

    arplookup 24.130.144.1 failed: host is not on local network

    arpresolve: can't allocate route for 24.130.144.1

    vr0: link state changed to DOWN

    vr0: link state changed to UP

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1

    arpresolve: can't allocate route for 76.126.216.1
    ....



  • I have dual wan and sometimes the cable modem changes IP once every few months and my lan rule to route all hosts on the same subnet is slightly incorrect. Fixed now. When I don't torrent, everything is fine and dandy. Anyone see any glaring statistics when i'm torrenting?



  • I have the same problem.

    I have two 18mbit(down) 2.5mbit(up) links and it seems to happen when i am seeding torrents.

    I will leave ubuntu seeding when i am at work today and see if it happens.

    James



  • System>advanced: Bump up the maximum firewallstates value. You also can monitor states either at status>system or by viewing the corresponding states rrd graph (status>rrdgraphs). Maybe you are running just out of states.



  • Thanks for your suggestion but that is the first thing I did:

    LIMITS:

    states    hard limit 100000

    src-nodes  hard limit  10000

    frags      hard limit  5000

    100,000 states is enough when I reach around 4-5K only.



  • You guys mentioned the word "torrents".

    My question : are you sure that some one isn't filtering upstream ??
    Some kind of QOS system used by your ISP ?



  • I can't confirm 100% but on my wrt54g modded with tomato firmware, I can run for a long time and my connection doesn't slow down.

    Also how I know it is not the ISP problem and it is the pfsense box because getting internally to 192.168.1.1 is a big problem too. Connections are reset, refresh 15 times and 1 time the page will load half.

    Sigh



  • @GoldServe:

    I can't confirm 100% but on my wrt54g modded with tomato firmware, …

    Just forget about my question. If another router works, then no ISP troubles.



  • No ideas still?



  • I can't believe I was so dumb. It is NOT PFSENSE and it was my windows xp sp2 causing it. I thought I had cracked the tcpip.sys to allow more than 10 syn connections but I guess I didn't. It was XP causing the problems and when I boosted the number of connections to 50, i'm all fine (crosses fingers)

    THANKS!



  • I never thought it was a good idea from microsoft to set such a low limit  :P


Locked