Realtek NIC + PPPoe + Gig, Cpu usage seems odd?



  • I am aware of all the issues with Realtek Nics and PPPoe and Gig speeds I have already applied the net.isr.dispatch="deferred" fix and running the compile 1.95 driver.

    That gave me another 30% increase in speed. But I was curious is anyone can explain the output below from Netstat -Q.

    From what i am reading it seems that it "favors" one Cpu(Cpu1) and never touches Cpu0 for ip??

    Configuration:
    Setting                        Current        Limit
    Thread count                         4            4
    Default queue limit                256        10240
    Dispatch policy               deferred          n/a
    Threads bound to CPUs         disabled          n/a
    
    Protocols:
    Name   Proto QLimit Policy Dispatch Flags
    ip         1   1000   flow  default   ---
    igmp       2    256 source  default   ---
    rtsock     3   1024 source  default   ---
    arp        4    256 source  default   ---
    ether      5    256 source   direct   ---
    ip6        6    256   flow  default   ---
    
    Workstreams:
    WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
       0   0   ip         0     0        0        0        0        0        0
       0   0   igmp       0     0        0        0        0        0        0
       0   0   rtsock     0     2        0        0        0      416      416
       0   0   arp        0     0        0        0        0        0        0
       0   0   ether      0     0  2283875        0        0        0  2283875
       0   0   ip6        0     0        0        0        0        0        0
       1   1   ip         0  1000        0        0     7907  3403871  3403871
       1   1   igmp       0     0        0        0        0        0        0
       1   1   rtsock     0     0        0        0        0        0        0
       1   1   arp        0     1        0        0        0       46       46
       1   1   ether      0     0  2922613        0        0        0  2922613
       1   1   ip6        0     1        0        0        0        6        6
       2   2   ip         0     3        0        0        0     9386     9386
       2   2   igmp       0     0        0        0        0        0        0
       2   2   rtsock     0     0        0        0        0        0        0
       2   2   arp        0     0        0        0        0        0        0
       2   2   ether      0     0     2865        0        0        0     2865
       2   2   ip6        0     0        0        0        0        0        0
       3   3   ip         0     2        0        0        0     1305     1305
       3   3   igmp       0     0        0        0        0        0        0
       3   3   rtsock     0     0        0        0        0        0        0
       3   3   arp        0     1        0        0        0       16       16
       3   3   ether      0     0     2902        0        0        0     2902
       3   3   ip6        0     1        0        0        0        2        2
    
    

    Ran a few more tests

    net.isr.maxthreads="3"
    net.isr.direct="0"
    net.isr.direct_force="0"
    

    Has netted me at least 2 cpu's working now, forcing it to 3 (4 total) caused it to use it more evenly? Setting it to 4 showed no gains.

    Configuration:
    Setting                        Current        Limit
    Thread count                         3            3
    Default queue limit                256        10240
    Dispatch policy               deferred          n/a
    Threads bound to CPUs         disabled          n/a
    
    Protocols:
    Name   Proto QLimit Policy Dispatch Flags
    ip         1   1000   flow  default   ---
    igmp       2    256 source  default   ---
    rtsock     3   1024 source  default   ---
    arp        4    256 source  default   ---
    ether      5    256 source   direct   ---
    ip6        6    256   flow  default   ---
    
    Workstreams:
    WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
       0   0   ip         0   901        0        0        0  2859408  2859408
       0   0   igmp       0     0        0        0        0        0        0
       0   0   rtsock     0     2        0        0        0      546      546
       0   0   arp        0     0        0        0        0        0        0
       0   0   ether      0     0  3307416        0        0        0  3307416
       0   0   ip6        0     1        0        0        0       15       15
       1   1   ip         0  1000        0        0     4381  2817582  2817582
       1   1   igmp       0     0        0        0        0        0        0
       1   1   rtsock     0     0        0        0        0        0        0
       1   1   arp        0     1        0        0        0     1230     1230
       1   1   ether      0     0  5272237        0        0        0  5272237
       1   1   ip6        0     1        0        0        0        7        7
       2   2   ip         0     0        0        0        0        0        0
       2   2   igmp       0     0        0        0        0        0        0
       2   2   rtsock     0     0        0        0        0        0        0
       2   2   arp        0     0        0        0        0        0        0
       2   2   ether      0     0     5874        0        0        0     5874
       2   2   ip6        0     0        0        0        0        0        0
    

    Still lots of Q drops. but will see if i can fix those.

    Running at ~ 800/750 Mbps now.


  • Netgate Administrator

    That seems expected for PPPoE. Only one CPU core is used.

    Unless you're indicating something deeper?

    Steve



  • @stephenw10 said in Realtek NIC + PPPoe + Gig, Cpu usage seems odd?:

    That seems expected for PPPoE. Only one CPU core is used.

    Unless you're indicating something deeper?

    Steve

    Well by making the tweak I was able to to get CPU0 working along with CPU1, which netted me another 10-15% worth of performance. PPPoe only using 1 core is very limiting in order to get the full speeds of my Gig line. Odd that I was able to get it to run with 2 cores by specifying

    net.isr.maxthreads="3"
    net.isr.direct="0"
    net.isr.direct_force="0"
    

    But i cannot seem to get it to use three. Also any ideas on what i can do for the QDrops?


  • Netgate Administrator

    Mmm, I'd have to read into it to answer that in any depth but as I understand it PPPoE is fundamentally single threaded and changing that is non-trivial. No amount if changing settings is going to change that.
    I would say what you;re seeing there is the limit of what the single core can do combined with the Realtek driver with one core in servicing the queues in each direction.
    You are seeing queue drops because it can't service the queues fast enough. I would have to guess that's the upload queue.

    What CPU is that?

    Steve



  • @stephenw10

    it's a Intel(R) Celeron(R) CPU N3150 @ 1.60GHz. I see activity on both CPU's (Qdrops, Queued and Handled) when running a download/upload test but only see Len packets on the ip service of CPU0 when downloading and CPU1 when uploading. Is there a ressourrce i can look up to go into more details as to what all the headers are? Len, Wmark.. and so on.

    Sample

    Workstreams:
    WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
       0   0   ip         0  1000        0        0    15609 193073039 193073039
       0   0   igmp       0     0        0        0        0        0        0
       0   0   rtsock     0     3        0        0        0     4725     4725
       0   0   arp        0     0        0        0        0        0        0
       0   0   ether      0     0 189534247        0        0        0 189534247
       0   0   ip6        0     2        0        0        0     8324     8324
       1   1   ip       674  1000        0        0    57090 120420852 120419444
       1   1   igmp       0     0        0        0        0        0        0
       1   1   rtsock     0     0        0        0        0        0        0
       1   1   arp        0     2        0        0        0    44889    44889
       1   1   ether      0     0 310688209        0        0        0 310688209
       1   1   ip6        0     2        0        0        0    12014    12014
       2   2   ip         0     0        0        0        0        0        0
       2   2   igmp       0     0        0        0        0        0        0
       2   2   rtsock     0     0        0        0        0        0        0
       2   2   arp        0     0        0        0        0        0        0
       2   2   ether      0     0  1016577        0        0        0  1016577
       2   2   ip6        0     0        0        0        0        0        0
    
    

  • Netgate Administrator

    I don't know of anything specific for that display but there are a number of FreeBSD man pages related to net isr.

    I think this may be confusing matters though, PPPoE is not IP. I suspect the deferred setting there allows netgraph to work better. Though that's a guess.

    The basic issue there though is that your CPU is not fast enough so the queues are filling and dropping packets once they are full. I have assumed the Len column to be the queue length at that instant and WMark to be the high water mark. In your output the queue that hits a watermark of 1000 has dropped packets as expected.

    To compare here's output from my box I was testing PPPoE throughput on:

    [2.4.5-DEVELOPMENT][admin@xtm800.stevew.lan]/root: netstat -Q
    Configuration:
    Setting                        Current        Limit
    Thread count                         4            4
    Default queue limit                256        10240
    Dispatch policy               deferred          n/a
    Threads bound to CPUs         disabled          n/a
    
    Protocols:
    Name   Proto QLimit Policy Dispatch Flags
    ip         1   1000   flow  default   ---
    igmp       2    256 source  default   ---
    rtsock     3   1024 source  default   ---
    arp        4    256 source  default   ---
    ether      5    256 source   direct   ---
    ip6        6    256   flow  default   ---
    
    Workstreams:
    WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
       0   0   ip         0    54   477871        0        0 12536027 13013897
       0   0   igmp       0     0        0        0        0        0        0
       0   0   rtsock     0     2        0        0        0    36549    36549
       0   0   arp        0     0      342        0        0        0      342
       0   0   ether      0     0   752091        0        0        0   752091
       0   0   ip6        0     0     2011        0        0        0     2011
       1   1   ip         0    11   536285        0        0   210576   746861
       1   1   igmp       0     0        0        0        0        0        0
       1   1   rtsock     0     0        0        0        0        0        0
       1   1   arp        0     1        0        0        0     2384     2384
       1   1   ether      0     0   511938        0        0        0   511938
       1   1   ip6        0     2        0        0        0    50151    50151
       2   2   ip         0     1  7995646        0        0      184  7995830
       2   2   igmp       0     0        0        0        0        0        0
       2   2   rtsock     0     0        0        0        0        0        0
       2   2   arp        0     0        0        0        0        0        0
       2   2   ether      0     0 19436994        0        0        0 19436994
       2   2   ip6        0     0        0        0        0        0        0
       3   3   ip         0     1  1948445        0        0      160  1948605
       3   3   igmp       0     0        0        0        0        0        0
       3   3   rtsock     0     0        0        0        0        0        0
       3   3   arp        0     0        0        0        0        0        0
       3   3   ether      0     0  1948571        0        0        0  1948571
       3   3   ip6        0     0        0        0        0        0        0
    

    The queues hardly get full at all and as a result no packets are dropped.

    Do you have powerd enabled? Is the CPU able to use 'burst' mode?

    Steve



  • @stephenw10

    Thanks for the Powerd clue, i guess the cpu was not "bursting" to 2.08Ghz without it on, not sure yet but it has gained me another 100Mbps , closing in on 900 now..


Log in to reply